feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 08:58:35 +02:00 · 2026-04-08 08:58:35 +02:00 · baa2d0220b
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
--- a/skills/ms-ai-governance/references/monitoring-observability/response-quality-metrics-rag.md
+++ b/skills/ms-ai-governance/references/monitoring-observability/response-quality-metrics-rag.md
@ -0,0 +1,622 @@
+# Response Quality Metrics and Evaluation for RAG Systems
+
+**Last updated:** 2026-02
+**Status:** GA
+**Category:** Monitoring & Observability
+
+---
+
+## Introduksjon
+
+Response quality metrics er kritisk for å evaluere effektiviteten av RAG-systemer (Retrieval-Augmented Generation). Mens infrastruktur-metrics (tokens, latency, throughput) forteller deg om systemet *kjører*, forteller kvalitetsmetrikker deg om systemet produserer *nyttige og korrekte svar*.
+
+I motsetning til tradisjonelle applikasjoner, hvor output er deterministisk, genererer LLM-er ikke-deterministische responser. Samme prompt kan gi forskjellige resultater hver gang. Dette krever et systematisk rammeverk for å måle kvalitet på tvers av dimensjoner som groundedness (er svaret basert på context?), relevance (adresserer svaret spørsmålet?), completeness (dekker svaret alle aspekter?), og coherence (flyter svaret logisk?).
+
+Azure AI Foundry og Azure AI Evaluation SDK tilbyr AI-assisterte evaluatorer som bruker GPT-modeller som "dommere" til å score responser, samt NLP-baserte metrics (BLEU, ROUGE, METEOR) for tekstlikhet. Sammen gir disse et helhetlig bilde av RAG-systemets evne til å levere korrekt, relevant og komplett informasjon fra grunnlagsdata.
+
+## Kjernekomponenter
+
+### RAG-spesifikke evaluatorer (AI-assistert)
+
+| Evaluator | Formål | Input | Output | Skala |
+|-----------|--------|-------|--------|-------|
+| **Groundedness** | Måler om response er konsistent med retrieved context (precision-aspekt) | Query (valgfri), Context, Response | Pass/Fail + score | 1-5 Likert |
+| **Groundedness Pro** | Streng consistency-sjekk med Azure AI Content Safety | Query, Context, Response | True/False + reason | Boolean |
+| **Relevance** | Måler hvor relevant response er til query | Query, Response | Pass/Fail + score | 1-5 Likert |
+| **Response Completeness** | Måler om response dekker all kritisk info fra ground truth (recall-aspekt) | Response, Ground truth | Pass/Fail + score | 1-5 Likert |
+| **Retrieval** | Måler tekstlig kvalitet av retrieved context chunks (uten ground truth) | Query, Context | Pass/Fail + score | 1-5 Likert |
+| **Coherence** | Måler logisk konsistens og flyt | Query, Response | Pass/Fail + score | 1-5 Likert |
+| **Fluency** | Måler naturlig språkkvalitet og lesbarhet | Response | Pass/Fail + score | 1-5 Likert |
+
+### Process-evaluering (retrieval-kvalitet)
+
+| Evaluator | Formål | Krever ground truth? | Metrics |
+|-----------|--------|---------------------|---------|
+| **Document Retrieval** | Måler hvor godt RAG henter korrekte dokumenter fra document store | Ja (query relevance labels) | Fidelity, NDCG, XDCG, Max Relevance, Holes |
+
+**Document Retrieval metrics forklart:**
+
+- **Fidelity**: Antall gode dokumenter returnert / totalt antall kjente gode dokumenter
+- **NDCG** (Normalized Discounted Cumulative Gain): Hvor godt ranking matcher ideell rekkefølge (alle relevante øverst)
+- **XDCG** (eXpected DCG): Kvalitet på top-k dokumenter uavhengig av andre dokumenter
+- **Max Relevance N**: Maksimal relevans i top-k chunks
+- **Holes**: Antall dokumenter med manglende query relevance judgments (sanity check)
+
+### NLP-baserte similarity metrics
+
+| Metric | Type | Formål | Input |
+|--------|------|--------|-------|
+| **F1 Score** | Token overlap | Harmonisk gjennomsnitt av precision og recall | Response, Ground truth |
+| **BLEU** | N-gram overlap | Maskinoversettelseskvalitet (opprinnelig for translation) | Response, Ground truth |
+| **GLEU** | Sentence-level variant | Google-BLEU for setningsnivå | Response, Ground truth |
+| **ROUGE** | N-gram recall | Overlap fokusert på recall (sammendrag-evaluering) | Response, Ground truth |
+| **METEOR** | Semantic overlap | Inkluderer stemming, synonymer, parafrasering | Response, Ground truth |
+
+### LLM-judge modellsupport
+
+Azure AI evaluatorer støtter både reasoning models (o-series) og non-reasoning models (GPT-4o, GPT-4.1):
+
+```python
+from azure.ai.evaluation import GroundednessEvaluator
+
+# Med reasoning model (o-series)
+groundedness = GroundednessEvaluator(
+    model_config=model_config,
+    is_reasoning_model=True,  # Aktiver reasoning mode
+    threshold=3
+)
+
+# Med standard GPT-4o
+groundedness = GroundednessEvaluator(
+    model_config=model_config,
+    threshold=3
+)
+```
+
+**Anbefaling:** Bruk reasoning models (e.g., `gpt-4.1-mini`) for kompleks evaluering som krever dypere resonnering — balanse mellom reasoning performance, cost og efficiency.
+
+## Arkitekturmønstre
+
+### Mønster 1: Multi-dimensional evaluation pipeline
+
+**Bruksområde:** Pre-production testing av RAG-system før deploy.
+
+**Arkitektur:**
+```
+[Test Dataset]
+    ↓
+[RAG Application] → Generates: Response, Context
+    ↓
+[Parallel Evaluation]
+    ├─ Groundedness (Context ↔ Response)
+    ├─ Relevance (Query ↔ Response)
+    ├─ Coherence (Response flow)
+    ├─ Retrieval (Query ↔ Context quality)
+    └─ Response Completeness (Response ↔ Ground truth)
+    ↓
+[Aggregated Metrics Dashboard]
+```
+
+**Fordeler:**
+- Holistisk kvalitetsbilde på tvers av dimensjoner
+- Parallell evaluering gir rask feedback
+- Aggregerte resultater identifiserer mønstre
+
+**Ulemper:**
+- Krever GPT-modell som judge (kostnad per evaluering)
+- Latency: 5-10 sekunder per query avhengig av antall evaluatorer
+- Ikke-deterministisk: samme prompt kan gi ulike scores
+
+**Implementering:**
+```python
+from azure.ai.evaluation import evaluate, GroundednessEvaluator, RelevanceEvaluator
+
+result = evaluate(
+    data="test_data.jsonl",
+    evaluators={
+        "groundedness": GroundednessEvaluator(model_config),
+        "relevance": RelevanceEvaluator(model_config)
+    },
+    evaluator_config={
+        "groundedness": {
+            "column_mapping": {
+                "query": "${data.query}",
+                "context": "${data.context}",
+                "response": "${data.response}"
+            }
+        }
+    },
+    azure_ai_project=azure_ai_project,
+    output_path="./eval_results.json"
+)
+```
+
+### Mønster 2: Parameter sweep med Document Retrieval
+
+**Bruksområde:** Optimalisere retrieval-parametere (top-k, chunk size, search algorithm).
+
+**Arkitektur:**
+```
+[Test Queries + Ground Truth Labels]
+    ↓
+[Generate Retrieval Results] → Variants:
+    ├─ Vector search, top-3, 500-token chunks
+    ├─ Hybrid search, top-5, 500-token chunks
+    ├─ Vector search, top-3, 1000-token chunks
+    └─ Semantic search, top-10, 500-token chunks
+    ↓
+[Document Retrieval Evaluator] → Per variant:
+    ├─ NDCG@k
+    ├─ Fidelity
+    ├─ XDCG
+    └─ Max Relevance
+    ↓
+[Compare Metrics Across Variants] → Select best configuration
+```
+
+**Fordeler:**
+- Systematisk optimalisering av retrieval
+- Datadrevet beslutning om search-parametere
+- Identifiserer trade-offs (e.g., høy NDCG vs. lavere latency)
+
+**Ulemper:**
+- Krever manuelt merkede ground truth labels (query relevance judgments)
+- Tidkrevende å generere labels for mange queries
+- Metrics reflekterer kun retrieval, ikke end-to-end kvalitet
+
+**Implementering:**
+```python
+from azure.ai.evaluation import DocumentRetrievalEvaluator
+
+retrieval_ground_truth = [
+    {"document_id": "1", "query_relevance_label": 4},
+    {"document_id": "2", "query_relevance_label": 2}
+]
+
+retrieved_documents = [
+    {"document_id": "2", "relevance_score": 45.1},
+    {"document_id": "3", "relevance_score": 29.2}
+]
+
+evaluator = DocumentRetrievalEvaluator(
+    ground_truth_label_min=0,
+    ground_truth_label_max=4,
+    ndcg_threshold=0.5
+)
+
+result = evaluator(
+    retrieval_ground_truth=retrieval_ground_truth,
+    retrieved_documents=retrieved_documents
+)
+```
+
+### Mønster 3: Continuous evaluation i production
+
+**Bruksområde:** Overvåke response quality over tid i production RAG-system.
+
+**Arkitektur:**
+```
+[Production Traffic]
+    ↓
+[Sample 5-10% of queries] → Log: Query, Context, Response
+    ↓
+[Scheduled Batch Evaluation] (daglig/ukentlig)
+    ├─ Groundedness trend
+    ├─ Relevance trend
+    └─ Coherence trend
+    ↓
+[Metrics Dashboard + Alerts]
+    ├─ Track: avg score over time, % pass/fail
+    └─ Alert: if avg score drops below threshold
+```
+
+**Fordeler:**
+- Oppdager kvalitetsdegradering over tid (f.eks., nye data i corpus)
+- Identifiserer edge cases fra production traffic
+- Lav overhead (kun sample av traffic)
+
+**Ulemper:**
+- Delayed feedback (batch-kjøring, ikke real-time)
+- Sampling kan misse sjeldne failure cases
+- Cost: GPT-judge for hver evaluering i batch
+
+**Implementering:**
+```python
+# Azure Monitor dashboard med KQL query
+AzureDiagnostics
+| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
+| where OperationName == "RAGEvaluation"
+| extend groundedness_score = toint(parse_json(Properties).groundedness)
+| summarize avg_groundedness = avg(groundedness_score) by bin(TimeGenerated, 1d)
+| render timechart
+```
+
+## Beslutningsveiledning
+
+### Kombinasjoner av metrics
+
+RAG-evaluering krever **flere metrics sammen** for å forstå hvor problemet ligger:
+
+| Symptom | Metrics | Mulig årsak | Løsning |
+|---------|---------|-------------|---------|
+| **Høy groundedness (0.9), lav correctness (0.4)** | Groundedness + Correctness | LLM bruker context men trekker feil konklusjoner | Juster prompt, sjekk source data for feil info |
+| **Høy utilization (0.9), lav completeness (0.3)** | Utilization + Completeness | Retrieval henter riktig men inkomplett info | Øk top-k, juster chunking for større context |
+| **Høy groundedness (0.9), høy utilization (0.9), lav similarity (0.3)** | Groundedness + Utilization + Similarity | System bruker riktig data men parafraser dårlig | Juster prompt for bedre parafrasering |
+| **Lav relevance** | Relevance | Response adresserer ikke query | Sjekk om relevant context ble retrieved; juster embedding model eller prompt |
+
+### Velg riktig evaluator for use case
+
+| Scenario | Anbefalt evaluator | Hvorfor |
+|----------|-------------------|---------|
+| Har IKKE ground truth, vil unngå hallucinations | **Groundedness** | Måler om response holder seg til context |
+| Har ground truth, vil sikre komplett svar | **Response Completeness** | Måler recall (ikke misse kritisk info) |
+| Vil ha strengeste groundedness-sjekk | **Groundedness Pro** | Azure AI Content Safety — binary True/False, strengere enn LLM-judge |
+| Vil optimalisere retrieval-parametere | **Document Retrieval** | Krever ground truth labels, gir Fidelity/NDCG/XDCG metrics |
+| Vil evaluere retrieval uten ground truth | **Retrieval** | LLM-judge vurderer tekstlig kvalitet av context |
+| Vil måle om response svarer på query | **Relevance** | Måler accuracy, completeness, direct relevance |
+
+### Threshold-konfigurering
+
+AI-assisterte evaluatorer bruker **Likert scale (1-5)** og **threshold** for pass/fail:
+
+```python
+groundedness = GroundednessEvaluator(
+    model_config=model_config,
+    threshold=3  # Default: 3 (scores ≥3 = pass, <3 = fail)
+)
+```
+
+**Anbefalinger:**
+- **Threshold 3**: Balansert — god for de fleste use cases
+- **Threshold 4**: Strengere — bruk for high-stakes scenarios (medical, legal)
+- **Threshold 2**: Mer tolerant — bruk for exploratory/creative use cases
+
+### Vanlige feil
+
+| Feil | Konsekvens | Løsning |
+|------|------------|---------|
+| Bruke kun én metric (e.g., kun groundedness) | Mister holistisk bilde av kvalitet | Evaluer minst 3-4 metrics (groundedness, relevance, coherence) |
+| Forvente deterministiske scores | Frustrasjon når samme query gir forskjellige scores | Bruk **target range** (e.g., 4.0-5.0) ikke single target |
+| Bruke LLM-judge uten model config | Evaluering feiler | Alltid send `model_config` med Azure OpenAI endpoint/deployment |
+| Ikke sample production traffic | Mister insight i real-world failures | Implementer sampling + batch evaluation |
+| Ignorere "reason" field i output | Mister kontekst for hvorfor score er lav | Les alltid `*_reason` field for debugging |
+
+### Røde flagg
+
+- **Groundedness score < 2.0**: Response er sannsynligvis hallucinated eller ikke basert på context → sjekk embedding model og chunking
+- **All safety metrics = 0**: Category disabled eller unsupported model → bekreft Content Safety er aktivert
+- **NDCG < 0.3**: Retrieval ranking er veldig dårlig → juster search algorithm (hybrid vs. vector)
+- **Holes > 50%**: Mange dokumenter mangler ground truth labels → forbedre labeling-prosess
+- **Consistency gap**: Metrics scorer høyt i test, lavt i production → test data er ikke representativt for production traffic
+
+## Integrasjon med Microsoft-stakken
+
+### Azure AI Foundry Evaluation
+
+**Pre-production evaluation workflow:**
+```
+[Foundry Portal] → [Evaluations tab]
+    ↓
+1. Configure test data (upload .jsonl eller generer med GPT)
+2. Select metrics (groundedness, relevance, coherence, etc.)
+3. Map dataset columns → evaluator inputs
+4. Submit evaluation run
+    ↓
+[Results dashboard]
+    ├─ Aggregerte metrics (avg score, pass rate)
+    ├─ Row-level results (per query)
+    └─ Reason field (forklaring per score)
+```
+
+**Model evaluation** (sammenlign base models):
+```python
+# Foundry benchmark for model selection
+client.evals.create(
+    name="Compare GPT-4o vs GPT-4.1",
+    data_source_config=data_source_config,
+    testing_criteria=[
+        {"type": "azure_ai_evaluator", "evaluator_name": "builtin.groundedness"},
+        {"type": "azure_ai_evaluator", "evaluator_name": "builtin.relevance"}
+    ]
+)
+```
+
+### Azure Monitor + Log Analytics
+
+**Eksporter evaluation metrics til Log Analytics:**
+```python
+# Diagnostic settings: send logs til Log Analytics workspace
+# KQL query for trends
+AzureDiagnostics
+| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
+| where OperationName == "EvaluationRun"
+| extend groundedness = toint(parse_json(Properties).groundedness)
+| summarize avg(groundedness) by bin(TimeGenerated, 1h)
+| render timechart
+```
+
+### Azure AI Content Safety (Groundedness Pro)
+
+```python
+from azure.ai.evaluation import GroundednessProEvaluator
+
+azure_ai_project = {
+    "subscription_id": os.environ["AZURE_SUBSCRIPTION_ID"],
+    "resource_group_name": os.environ["AZURE_RESOURCE_GROUP"],
+    "project_name": os.environ["AZURE_PROJECT_NAME"]
+}
+
+groundedness_pro = GroundednessProEvaluator(
+    azure_ai_project=azure_ai_project
+)
+
+result = groundedness_pro(
+    query="Is Marie Curie born in Paris?",
+    context="Marie Curie is born in Warsaw.",
+    response="No, Marie Curie is born in Warsaw."
+)
+# Output: {"groundedness_pro_label": True, "groundedness_pro_reason": "All Contents are grounded"}
+```
+
+### Copilot Studio + Prompt Flow
+
+**Evaluering i Prompt Flow:**
+- Bruk `QAEvaluator` (composite evaluator som kjører groundedness, relevance, coherence, fluency, similarity, F1 samtidig)
+- Integrer i CI/CD: kjør evaluation som del av deployment-pipeline
+
+```python
+from azure.ai.evaluation import QAEvaluator
+
+qa_eval = QAEvaluator(
+    model_config=model_config,
+    groundedness_threshold=3,
+    relevance_threshold=3,
+    coherence_threshold=3
+)
+
+result = qa_eval(
+    query="What is the capital of France?",
+    response="Paris is the capital of France.",
+    context="France is a country in Europe. Paris is its capital.",
+    ground_truth="Paris"
+)
+```
+
+### MLflow + Databricks
+
+Azure Databricks støtter MLflow 3 GenAI evaluation:
+
+```python
+import mlflow
+from mlflow.genai.scorers import RetrievalGroundedness, RetrievalRelevance
+
+eval_results = mlflow.genai.evaluate(
+    data=eval_dataset,
+    predict_fn=rag_app,
+    scorers=[
+        RetrievalGroundedness(model="databricks:/databricks-gpt-oss-120b"),
+        RetrievalRelevance(model="databricks:/databricks-gpt-oss-120b")
+    ]
+)
+```
+
+## Offentlig sektor (Norge)
+
+### GDPR og datasuverenitet
+
+**Utfordring:** AI-assisterte evaluatorer sender data til GPT-modeller for scoring — kan inneholde PII fra production traffic.
+
+**Løsning:**
+- Anonymiser/pseudonymiser data **før** evaluering:
+  ```python
+  # Eksempel: erstatt navn med placeholders
+  query = "Hva er status for John Doe sin søknad?"
+  anonymized = "Hva er status for [NAVN] sin søknad?"
+  ```
+- Bruk **Azure OpenAI i Norge-regioner** (Norway East) for data residency
+- Vurder NLP-baserte metrics (BLEU, ROUGE) som **ikke** sender data til LLM — deterministisk, ingen privacy risk
+
+### AI Act compliance (artikkel 10 + 15)
+
+**Artikkel 10 (Data governance):**
+- Dokumenter hvilke metrics som brukes og hvorfor → traceability
+- Logg evaluation runs med metadata: timestamp, dataset versjon, model versjon
+- Bevar evaluation results for audit trail
+
+**Artikkel 15 (Accuracy + robustness):**
+- Bruk **multiple metrics** for å demonstrere testing av accuracy (groundedness, relevance)
+- Implementer **continuous evaluation** for å oppdage degradering over tid
+- Dokumenter threshold-valg og trade-offs (e.g., hvorfor threshold=3 ikke threshold=4)
+
+### Forvaltningsloven § 25 (begrunnelsesplikt)
+
+**Utfordring:** Ved automatiserte vedtak, må system kunne forklare hvorfor et svar ble generert.
+
+**Løsning:**
+- Bruk evaluators med **reason field** (f.eks., `groundedness_reason`) som forklaring
+- Logg: Query → Retrieved documents → Response → Evaluation score + reason
+- Eksempel:
+  ```json
+  {
+    "query": "Er jeg kvalifisert for støtte?",
+    "response": "Ja, basert på din inntekt.",
+    "groundedness_reason": "Response er konsistent med context som viser inntektsgrense.",
+    "groundedness_result": "pass"
+  }
+  ```
+
+### Schrems II og data transfers
+
+**Issue:** Groundedness Pro bruker Azure AI Content Safety service — data kan teoretisk sendes til US-regioner.
+
+**Mitigering:**
+- Bruk **Groundedness (LLM-judge)** i stedet for Groundedness Pro — mer kontroll over model deployment region
+- Deploy GPT-judge i Norge-region (Norway East)
+- Bekreft at Azure AI Foundry project og OpenAI resource er i samme region
+
+### DPIA for response quality metrics
+
+**Vurderinger:**
+- **Privacy risk**: Lav for NLP-metrics (BLEU, ROUGE), Medium for AI-assisterte evaluatorer (sender til GPT)
+- **Mitigating measures**: Anonymisering, data residency i Norge, logging med limited retention
+- **Lawful basis**: Legitimate interest (artikkel 6(1)(f)) for quality assurance, eller public task (artikkel 6(1)(e)) for public sector
+
+## Kostnad og lisensiering
+
+### Evaluation cost model
+
+**AI-assisterte evaluatorer (GPT-judge):**
+- **Cost per evaluator call**: ~500-1500 tokens (prompt for evaluation logic) + response tokens
+- **Eksempel**: 1000 queries × 5 evaluatorer × 1500 tokens = 7.5M tokens
+- **Pricing**: GPT-4o @ $2.50/1M input tokens → ~$18.75 per evaluation run
+
+**NLP-baserte metrics:**
+- **Gratis** (deterministisk beregning, ingen API calls)
+- Bruk for cost-sensitive scenarios eller high-volume evaluation
+
+### PTU vs. PAYG for evaluation
+
+| Model | Anbefaling | Hvorfor |
+|-------|------------|---------|
+| **PAYG** (Pay-as-you-go) | Pre-production testing, ad-hoc evaluations | Fleksibel, kun betal for evaluations kjørt |
+| **PTU** (Provisioned Throughput) | Continuous production evaluation (daglig batch) | Fast månedlig kostnad, garantert kapasitet |
+
+**Break-even beregning:**
+```
+Monthly eval volume: 100K queries × 5 evaluatorer × 1500 tokens = 750M tokens/måned
+PAYG cost: 750M × $2.50/1M = $1875/måned
+PTU equivalent: ~300 PTUs @ $6/PTU = $1800/måned
+
+→ Bruk PTU hvis eval volume > 100K queries/måned
+```
+
+### Groundedness vs. Groundedness Pro cost
+
+| Evaluator | Cost | Latency | Accuracy |
+|-----------|------|---------|----------|
+| **Groundedness** (LLM-judge) | GPT tokens (variable) | 5-10 sek | Nondeterministisk |
+| **Groundedness Pro** (AI Content Safety) | Fixed per call (~$0.002/call) | 2-3 sek | Deterministic |
+
+**Anbefaling:**
+- **Groundedness Pro** for high-volume, cost-sensitive scenarios (fast pris, raskere)
+- **Groundedness** for customizable definition (kan tweake prompt) og edge cases (LLM bedre på edge cases)
+
+### Lisensiering
+
+**Nødvendig:**
+- **Azure OpenAI** (for GPT-judge): Standard/Enterprise Agreement
+- **Azure AI Foundry**: Gratis tier for evaluation UI, betaler kun for underliggende compute (GPT calls)
+- **Azure AI Content Safety** (for Groundedness Pro): Inkludert i Azure subscription, pay-per-transaction
+
+**Ikke nødvendig:**
+- Ingen spesielle lisenser for Azure AI Evaluation SDK (open source Python library)
+
+## For arkitekten (Cosmo)
+
+### Spørsmål å stille kunden
+
+1. **Har dere ground truth data for RAG-systemet?**
+   - Ja → bruk Response Completeness og Document Retrieval for presise metrics
+   - Nei → bruk Groundedness, Relevance, Retrieval (LLM-judge uten ground truth)
+
+2. **Hvor kritisk er correctness i domenet?** (medisinsk, juridisk vs. generell kundeservice)
+   - Høy criticality → strengere threshold (4-5), bruk Groundedness Pro
+   - Medium/lav → standard threshold (3), bruk Groundedness
+
+3. **Hva er evaluation-volumet?**
+   - < 10K queries/måned → bruk PAYG GPT-judge
+   - \> 100K queries/måned → vurder PTU for predictable cost
+
+4. **Trenger dere real-time eller batch evaluation?**
+   - Real-time → bruk Groundedness Pro (raskere, deterministisk)
+   - Batch → bruk multi-dimensional evaluation med flere GPT-judges
+
+5. **Har dere allerede logging av production queries?**
+   - Ja → implementer sampling + scheduled batch evaluation
+   - Nei → sett opp Azure Monitor diagnostics først
+
+6. **Vil dere optimalisere retrieval-parametere?**
+   - Ja → invester i ground truth labeling, bruk Document Retrieval evaluator
+   - Nei → bruk Retrieval evaluator (LLM-judge, ingen ground truth)
+
+7. **Hvilke Microsoft-tjenester bruker dere i dag?**
+   - Azure AI Foundry → bruk innebygd Evaluations UI
+   - Copilot Studio → integrer QAEvaluator i Prompt Flow
+   - Databricks → bruk MLflow GenAI evaluation
+
+8. **Har dere GDPR/privacy concerns med evaluation data?**
+   - Ja → anonymiser før evaluering, bruk Norge-region OpenAI
+   - Nei → standard setup
+
+### Fallgruver
+
+| Fallgruve | Konsekvens | Mitigering |
+|-----------|------------|------------|
+| **Bruke LLM-judge uten re-test ved model upgrade** | Scores kan endre seg når GPT-modell oppdateres | Pin judge model version i config, re-run baseline eval ved upgrade |
+| **Ikke dokumentere threshold-valg** | Kan ikke forklare hvorfor threshold=3 vs. threshold=4 | Dokumenter rationale i ADR (Architecture Decision Record) |
+| **Ignore "reason" field** | Debugging tar lang tid | Alltid inspiser reason field for low scores |
+| **Bruke kun groundedness** | Mister completeness/relevance perspektiv | Bruk minst 3 metrics (groundedness, relevance, coherence) |
+| **Ikke aggregere over tid** | Kan ikke spore quality trends | Lagre eval results i database, visualiser trender i dashboard |
+| **Over-reliance på AI-judge** | Cost kan eksplodere | Kombiner AI-judge med NLP-metrics (BLEU, ROUGE) for å redusere cost |
+
+### Anbefalinger per modenhetsnivå
+
+**Level 1 — Proof of Concept:**
+- Start med **Groundedness** og **Relevance** (to metrics)
+- Bruk Foundry Evaluations UI for rask feedback
+- Kjør ad-hoc evaluations på small dataset (10-50 queries)
+- Kostnadsramme: < $50/måned
+
+**Level 2 — Pilot:**
+- Legg til **Coherence**, **Fluency**, **Retrieval** (5 metrics total)
+- Implementer **Document Retrieval** hvis du har ground truth
+- Kjør scheduled batch evaluation (ukentlig)
+- Kostnadsramme: $200-500/måned
+
+**Level 3 — Production:**
+- Full metric suite (groundedness, relevance, coherence, fluency, retrieval, response completeness)
+- **Continuous evaluation** med sampling av production traffic (5-10%)
+- Integrer metrics i Azure Monitor dashboards
+- Automatiske alerts ved quality degradering
+- Kostnadsramme: $1000-3000/måned (avhengig av volume)
+
+**Level 4 — Enterprise:**
+- Multi-dimensional evaluation med custom evaluators
+- **Parameter sweep** automation for retrieval optimization
+- Integration med MLOps pipeline (eval som gate i deployment)
+- A/B testing av ulike RAG-konfigurasjoner
+- Kostnadsramme: $5000+/måned
+
+## Kilder og verifisering
+
+### Microsoft Learn (Verified via MCP)
+
+1. [Observability in generative AI - What are evaluators?](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/observability#what-are-evaluators) — RAG evaluators (Retrieval, Groundedness, Relevance, Response Completeness)
+2. [Retrieval-Augmented Generation (RAG) evaluators](https://learn.microsoft.com/en-us/azure/ai-foundry/concepts/evaluation-evaluators/rag-evaluators) — Detaljert dokumentasjon for alle RAG-evaluatorer, input/output formats
+3. [Large language model end-to-end evaluation](https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase) — Groundedness, completeness, utilization, relevance, correctness metrics
+4. [Evaluate generative AI models and applications](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/evaluate-generative-ai-app) — Foundry portal evaluation workflow, testing criteria configuration
+5. [Submit a batch run and evaluate a flow](https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/flow-bulk-test-evaluation) — Built-in evaluation methods (QnA Groundedness, Relevance, Coherence)
+6. [Evaluation of RAG performance basics](https://learn.microsoft.com/en-us/fabric/data-science/tutorial-evaluate-rag-performance) — AI-assisted metrics (groundedness, relevance, similarity), top-N retrieval rate
+7. [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai) — Azure Monitor integration, KQL queries, diagnostic settings
+8. [Use Risks & Safety monitoring](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/risks-safety-monitor) — Content filtering metrics, severity distribution
+9. [Azure AI Evaluation SDK - Python samples](https://github.com/Azure-Samples/azureai-samples/blob/main/scenarios/evaluate/) — Code examples for groundedness, relevance evaluators
+
+### Code samples (Verified via MCP)
+
+10. [GroundednessEvaluator Python sample](https://learn.microsoft.com/en-us/python/api/azure-ai-evaluation/azure.ai.evaluation.groundednessevaluator) — Conversation mode evaluation with multi-turn support
+11. [QAEvaluator Python sample](https://learn.microsoft.com/en-us/python/api/azure-ai-evaluation/azure.ai.evaluation.qaevaluator) — Composite evaluator combining multiple quality metrics
+12. [DocumentRetrievalEvaluator usage](https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/ai/azure-ai-projects/samples/evaluations/) — Parameter sweep for retrieval optimization
+
+### Konfidensnivå per seksjon
+
+| Seksjon | Confidence | Kilde |
+|---------|-----------|-------|
+| Kjernekomponenter | **Verified** | Microsoft Learn RAG evaluators doc + SDK samples |
+| Arkitekturmønstre | **Verified** | Microsoft Learn evaluation guides + Azure Architecture Center |
+| Beslutningsveiledning | **Verified** | Microsoft Learn LLM evaluation metrics + best practices |
+| Integrasjon med Microsoft-stakken | **Verified** | Foundry portal docs, Azure Monitor docs, MLflow docs |
+| Offentlig sektor | **Baseline** | Generell GDPR/AI Act kunnskap + Microsoft compliance docs |
+| Kostnad og lisensiering | **Verified** | Azure OpenAI pricing, AI Content Safety pricing |
+
+**MCP research calls:** 3 (microsoft_docs_search × 3, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)
+**Unique URLs:** 12