ms-ai-architect/skills/ms-ai-engineering/references/data-engineering/data-quality-ai-frameworks.md

# Data Quality Frameworks for AI

**Last updated:** 2026-06-24
**Status:** GA (Microsoft Purview Data Quality, Azure ML Model Monitoring, Fabric data quality)
**Category:** Data Engineering for AI
**Type:** reference
**Source:** https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Arkitekturmønstre](#arkitekturmønstre)
- [Beslutningsveiledning](#beslutningsveiledning)
- [Integrasjon med Microsoft-stakken](#integrasjon-med-microsoft-stakken)
- [Offentlig sektor (Norge)](#offentlig-sektor-norge)
- [Kostnad og lisensiering](#kostnad-og-lisensiering)
- [For arkitekten (Cosmo)](#for-arkitekten-cosmo)
- [Kilder og verifisering](#kilder-og-verifisering)

## Introduksjon

Data quality frameworks for AI sikrer at data som brukes til trening, validering og inferens av AI-modeller er nøyaktig, komplett, konsistent og pålitelig. I dagens AI-drevne landskap påvirker datakvalitet direkte AI-ytelsen, modellens nøyaktighet, og tillit til AI-beslutninger.

Dårlig datakvalitet fører til:
- Degradert modellytelse over tid (data drift, prediction drift)
- Feilaktige innsikter og anbefalinger
- Erosjon av tillit til AI-systemer
- Compliance-risiko (GDPR, AI Act, Forvaltningsloven)
- Operasjonelle ineffektiviteter og økte kostnader

Microsoft-stacken tilbyr fire hovedspor for data quality management i AI-kontekst:
1. **Microsoft Purview Data Quality** — enterprise data governance med AI-powered profiling og scoring
2. **Azure Machine Learning Model Monitoring** — production model monitoring med data drift detection
3. **Microsoft Fabric data quality** — lakehouse-native quality constraints i materialized lake views og pipelines
4. **Azure Databricks expectations** — declarative data quality constraints i Delta Live Tables (DLT)

---

## Kjernekomponenter

### 1. Data Quality Dimensions (Six Industry Standards)

| Dimensjon | Definisjon | Microsoft tooling |
|-----------|------------|-------------------|
| **Completeness** | Grad av ikke-null verdier | Purview (null value rate), Databricks expectations, Fabric constraints |
| **Accuracy** | Data reflekterer real-world verdier | Azure ML model performance monitoring, ground truth comparison |
| **Consistency** | Data integrity opprettholdes på tvers av systemer | Purview cross-source validation, Unity Catalog referential integrity |
| **Timeliness** | Data oppdateres og er tilgjengelig i tide | Purview freshness rules, Azure ML prediction drift, Fabric streaming quality checks |
| **Uniqueness** | Ingen duplikater | Purview uniqueness metrics, Databricks DLT deduplication |
| **Conformity** | Data overholder format og schema | Purview data type error rate, Azure ML schema validation |

### 2. Data Quality Lifecycle

```
1. [Data ingestion] → Schema validation, type checks
2. [Enrichment] → Profiling, quality rules application
3. [Training] → Data drift monitoring vs. training baseline
4. [Deployment] → Production inference data quality
5. [Monitoring] → Continuous data quality scoring, alerts
6. [Remediation] → Data quality actions, root cause analysis
```

### 3. Microsoft Purview Data Quality — Enterprise Governance

**Capabilities:**
- **AI-powered profiling:** Anbefaler kolonner for profiling, generer statistikk (distribution, min, max, std dev, uniqueness, completeness)
- **No-code/low-code rules:** Out-of-box rules + AI-generated rules + custom rules
- **Data quality scoring:** Aggregert scoring på column → data asset → data product → governance domain nivå
- **Alerts:** Email notifications ved threshold breaches
- **Actions center:** Diagnostic queries for stewards til å identifisere spesifikke rader som feiler quality checks

**Supported data sources (multi-cloud):**
- Azure: Blob Storage, ADLS Gen2, Azure SQL DB, Synapse, Fabric Lakehouse (Delta/Iceberg format)
- AWS: S3 (Parquet, CSV, Delta), RDS
- GCP: BigQuery (limited support for virtual network)

**Pricing:** Data Governance Processing Units (DGPU) pay-as-you-go

**Limitations:**
- Managed Identity som eneste auth-metode
- Parquet: støtter kun (1) directory med part files eller (2) partitioned Parquet (year=2018/month=Dec) — ikke arbitrary hierarchies
- Virtual network ikke støttet for BigQuery

### 4. Azure Machine Learning Model Monitoring

**Monitoring signals (tabular data):**

| Signal | Beskrivelse | Metrics | Reference data |
|--------|-------------|---------|----------------|
| **Data drift** | Endring i distribution av model input | Jensen-Shannon Distance, PSI, Normalized Wasserstein, KS test, Chi-squared | Training data eller recent production data |
| **Prediction drift** | Endring i distribution av model output | Jensen-Shannon Distance, PSI, Normalized Wasserstein, Chebyshev Distance, KS test, Chi-squared | Validation data eller recent production data |
| **Data quality** | Integrity av model input | Null value rate, Data type error rate, Out-of-bounds rate | Training data eller recent production data |
| **Feature attribution drift** | Endring i feature importance | Normalized Discounted Cumulative Gain (NDCG) | Training data med feature importance |
| **Model performance** | Ground truth vs. predictions | Accuracy, Precision, Recall, F1, AUC, MAE, RMSE | Ground truth labels (actuals) |

**Best practices:**
1. Start monitoring umiddelbart etter production deployment
2. Bruk multiple signals (e.g., data drift + feature attribution drift for early warnings)
3. Set monitoring frequency basert på data accumulation rate (daily hvis heavy traffic, weekly/monthly ellers)
4. Monitor top N features for large feature sets (reduserer cost og noise)
5. Bruk training data som baseline for data drift, validation data for prediction drift
6. Hvis ground truth er tilgjengelig: bruk model performance signal for objective view

**Setup:**
1. Enable production inference data collection (automatic for Azure ML online endpoints, manual for batch/external endpoints)
2. Set up model monitoring via SDK/CLI eller studio UI
3. View/analyze results i Azure ML workspace

### 5. Microsoft Fabric Data Quality

**Materialized Lake Views (MLVs) constraints:**

```sql
-- Example: exclude null customerName rows
CREATE MATERIALIZED LAKE VIEW sales_clean
(CONSTRAINT cust_blank CHECK (customerName IS NOT NULL) ON MISMATCH DROP)  -- or FAIL (default)
AS SELECT * FROM raw_sales;
```

**Actions:**
- `FAIL` — stops refresh ved første constraint violation (default)
- `DROP` — processes og fjerner records som ikke møter constraint, gir count i lineage view

**Limitations:**
- Constraints kan ikke oppdateres etter MLV creation (må recreate)
- Innebygde Spark/SQL-funksjoner (UPPER, TRIM, COALESCE, SUBSTRING …), UDF-er (`spark.udf.register()`), Pandas-UDF-er og Fabric User Data Functions støttes i constraints; ren regex/LIKE er ikke et eget constraint-konstrukt — bruk funksjoner/UDF-er i stedet
- Known issue: MLV med FAIL action kan gi "delta table not found" error (workaround: unngå FAIL, bruk DROP)

**Data quality for Fabric Lakehouse (Purview integration):**
1. Mirror eller load data til Fabric Lakehouse i Delta/Iceberg format
2. Grant Contributor access til workspace for Purview MSI
3. Run Data Map scan på Lakehouse (service principal auth)
4. Associate Lakehouse tables med data product i Purview Unified Catalog
5. Profile + create rules + run data quality scan via Purview

### 6. Azure Databricks Expectations (Lakeflow Spark Declarative Pipelines, tidl. Delta Live Tables/DLT)

> Delta Live Tables (DLT) er rebrandet til **Lakeflow Spark Declarative Pipelines (SDP)**. Ingen migrasjon kreves — `import dlt`/`@dlt.*` fungerer fortsatt, men anbefalt syntaks er nå `from pyspark import pipelines as dp` + `@dp.*` (classic SKU-er og event-log-skjema beholder fortsatt `DLT`-navnet).

**Expectations syntax:**

```python
from pyspark import pipelines as dp  # tidl. import dlt

@dp.expect("valid_timestamp", "timestamp IS NOT NULL")
@dp.expect_or_drop("valid_amount", "amount > 0")
@dp.expect_or_fail("critical_id", "id IS NOT NULL")
def clean_transactions():
    return spark.read.table("raw_transactions")
```

**Actions:**
- `@dp.expect` — track violations, allow records to pass
- `@dp.expect_or_drop` — drop violating records
- `@dp.expect_or_fail` — fail pipeline ved violations

**Benefits:**
- Catch data quality issues ved ingestion før de påvirker downstream data products
- Real-time quality metrics i Lakeflow SDP pipeline observability UI (event log)
- Automatically generated data quality dashboards

---

## Arkitekturmønstre

### Pattern 1: Layered Quality Gates (Medallion Architecture)

```
[Bronze Layer] → Schema validation, null checks
    ↓ (Fabric constraints: FAIL on critical nulls)
[Silver Layer] → Business rules, referential integrity, deduplication
    ↓ (Databricks expectations: DROP invalid records)
[Gold Layer] → Aggregations, data quality scoring
    ↓ (Purview: Profile + score + alert)
[Consumption] → AI model training/inference
```

**Decision criteria:**
- **Bronze:** Fail fast på kritiske schema violations (FAIL action)
- **Silver:** Isoler dårlig data (DROP action), log violations for remediation
- **Gold:** Enterprise-wide governance med Purview scoring

### Pattern 2: Training vs. Production Data Quality Monitoring

**Training phase:**
1. Purview profiling på training dataset
2. Define baseline quality thresholds (e.g., "null rate < 5%")
3. Apply rules, score data quality
4. Iterate: fix issues → retrain → validate

**Production phase:**
1. Azure ML Model Monitoring: track data drift (production vs. training baseline)
2. Set alerts: e.g., "Jensen-Shannon Distance > 0.3"
3. When alert fires → trigger Purview profiling på production data → compare profiles → investigate drift root cause

**Integration:**
- Azure ML Monitoring detekterer drift → triggers Azure Function → calls Purview API for detailed profiling → stores results i Azure Monitor logs

### Pattern 3: Golden Datasets for Consistent Validation

**Concept:** Authoritative datasets som representerer production data patterns, brukt for testing og validation across alle AI workloads.

**Implementation:**
1. Identify representative sample fra production data (stratified sampling)
2. Store i Azure Blob/ADLS Gen2 som immutable dataset (versioned)
3. Register i Azure ML as data asset (type: URIFolder)
4. Use in Azure ML training jobs as validation dataset
5. Update golden dataset quarterly (eller når significant drift detekteres)

**Quality checks on golden dataset:**
- Purview profiling + scoring (monthly)
- Azure ML data quality monitoring (continuous comparison mot latest production data)

### Pattern 4: Real-Time Data Quality for Streaming AI

**Scenario:** Real-time fraud detection model consuming transaction stream.

**Architecture:**
```
[Event Hub] → [Fabric Eventstream] → [Lakehouse (KQL DB)]
                ↓
    [Real-time aggregation pipeline med data quality constraints]
                ↓
    [Model inference endpoint] → [Azure ML Model Monitoring]
```

**Quality checks:**
1. **Ingestion:** Fabric eventstream DQ checks (schema validation, null checks)
2. **Aggregation:** KQL queries med quality assertions (e.g., `assert(count(fraud_score > 1.0) == 0)`)
3. **Inference:** Azure ML Model Monitoring tracks prediction drift i real-time (5-minute windows)

---

## Beslutningsveiledning

### Når velge hvilket verktøy?

| Scenario | Anbefalt verktøy | Rationale |
|----------|------------------|-----------|
| Enterprise-wide data governance på tvers av multi-cloud sources | **Microsoft Purview Data Quality** | Centralized governance domain structure, multi-cloud support, AI-powered profiling |
| Production AI model monitoring (deployed til Azure ML online endpoint) | **Azure ML Model Monitoring** | Out-of-box integration med Azure ML endpoints, automatic inference data collection |
| Lakehouse-native data quality i Fabric | **Fabric materialized lake views constraints** | Native Delta/Iceberg support, lineage integration, no external dependencies |
| Spark-based data pipelines med strict quality enforcement | **Databricks expectations (DLT)** | Declarative syntax, automatic DQ dashboards, fail pipeline on critical violations |
| Early-stage data quality exploration (low maturity) | **Azure ML data quality monitoring signal** | No infrastructure setup, quick start med training dataset comparison |
| Compliance-driven data quality documentation | **Purview Data Quality** | Audit logs, data quality scores per governance domain, alert notifications |
| Real-time streaming data quality | **Fabric Eventstream + KQL DB + Azure ML Monitoring** | Low-latency validation, real-time drift detection |

### Decision tree: Velge data quality framework

```
START
  ↓
Er du i Fabric-økosystemet?
  └─ JA → Bruk Fabric materialized lake views constraints for layered DQ
  └─ NEI → Har du Azure ML deployed models?
      └─ JA → Bruk Azure ML Model Monitoring for production monitoring
      └─ NEI → Har du multi-cloud data sources?
          └─ JA → Bruk Purview Data Quality for enterprise governance
          └─ NEI → Bruk Databricks expectations (DLT) for Spark pipelines
```

---

## Integrasjon med Microsoft-stakken

### 1. Purview + Fabric Integration

**Setup:**
1. Fabric Lakehouse: load data i Delta/Iceberg format
2. Fabric workspace: grant Contributor til Purview MSI
3. Purview Data Map: register Fabric source, run scan (service principal auth)
4. Purview Unified Catalog: add Lakehouse tables til data product
5. Purview Data Quality: configure connection (Tenant ID, Workspace ID, Lakehouse ID, Purview MSI credential)
6. Profile → create rules → run DQ scan

**Virtual Network support:**
- If Fabric tenant uses Private Link: check "Virtual Network" i connection screen, specify Azure region, add Private Link Resource ID
- Set up Purview compute allocation for managed virtual network (separate step)

### 2. Azure ML + Purview Integration (Conceptual — Manual)

**Gap:** Ingen native integration (as of 2026-02).

**Workaround:**
1. Azure ML Model Monitoring detects data drift → logs til Application Insights
2. Azure Function (triggered by App Insights alert) → calls Purview REST API → initiates profiling job on production data asset
3. Purview completes profiling → stores results → sends alert via email/Event Grid
4. Data scientist reviews Purview profile vs. training data profile → decides on retraining

**Code snippet (Azure Function trigger):**

```python
import requests
from azure.identity import DefaultAzureCredential

def trigger_purview_profiling(data_asset_id, connection_id):
    credential = DefaultAzureCredential()
    token = credential.get_token("https://purview.azure.net/.default").token

    headers = {"Authorization": f"Bearer {token}"}
    url = f"https://{purview_account}.purview.azure.com/dataQuality/assets/{data_asset_id}/profile"
    payload = {"connectionId": connection_id}

    response = requests.post(url, headers=headers, json=payload)
    return response.json()
```

### 3. Fabric + Azure ML Integration

**Scenario:** Train model i Azure ML, deploy til Azure ML online endpoint, monitor med Fabric-based data quality.

**Architecture:**
```
[Fabric Lakehouse] → Training data (Delta table)
    ↓ (Azure ML reads via OneLake)
[Azure ML training job] → Model artifact
    ↓ (Deploy)
[Azure ML online endpoint] → Inference data collection
    ↓ (Write back to Fabric Lakehouse via Azure Function)
[Fabric Lakehouse inference table] → Purview DQ scan
```

**Implementation:**
1. Azure ML online endpoint: enable data collection (stores JSON logs i Azure Blob)
2. Azure Function (Event Grid trigger on blob creation) → parses JSON → writes til Fabric Lakehouse via REST API
3. Fabric Lakehouse: Purview DQ scan på inference table → compare med training table profile

### 4. Databricks + Unity Catalog Data Quality

**Unity Catalog features:**
- **Data lineage:** Column-level lineage fra source tables til trained models (captured via `mlflow.log_model()`)
- **Audit logging:** Captures metadata access events (who accessed which dataset, when)
- **Data quality via DLT expectations:** Enforces quality constraints in pipelines

**Best practice:**
1. Define data quality standards i Unity Catalog tags (e.g., `"dq_tier": "gold"` for high-quality tables)
2. Apply expectations i DLT pipelines based on tier (e.g., gold tier → `expect_or_fail`, silver tier → `expect_or_drop`)
3. Monitor lineage: hvis model performance degraderer → trace back til upstream table via lineage → identify quality issues

---

## Offentlig sektor (Norge)

### 1. Compliance-krav

**Forvaltningsloven § 11b (automatisert enkeltvedtak):**
- **Krav:** "Datagrunnlaget må være kvalitetssikret."
- **Implementering:** Purview Data Quality scoring (minimum score: 80/100 for Gold-tier data brukt i vedtakssystemer) + monthly profiling + audit log retention (3 år)

**AI Act (High-Risk AI Systems):**
- **Krav (Article 10):** Data quality management, including data governance, relevance, representativeness, accuracy, completeness.
- **Implementering:**
  - Purview Data Quality: document data quality scores i compliance reports
  - Azure ML Model Monitoring: track data drift, maintain audit trail
  - Quarterly data quality review (documented i DPIA)

**GDPR Article 5(1)(d) — accuracy:**
- **Krav:** Personal data må være "accurate and, where necessary, kept up to date."
- **Implementering:** Purview freshness rules (e.g., "data må være < 24 timer gammel"), alert til data owner ved breach

### 2. Utredningsinstruksen § 4 — kvalitetssikring

**Krav:** "Utredningen skal være basert på best tilgjengelig kunnskap og data."

**Data quality checklist for AI-utredning:**
- [ ] Training data profiling report (Purview) vedlagt utredningen
- [ ] Data quality scores for alle datasets (minimum 70/100)
- [ ] Data lineage dokumentert (Unity Catalog eller Purview lineage view)
- [ ] Training vs. validation data drift analysis (Azure ML Monitoring)
- [ ] Golden dataset versioned og arkivert (Azure Blob immutable storage)
- [ ] Data quality monitoring plan for production phase (frequency, thresholds, alert recipients)

### 3. ROS-analyse: Data quality risks

| Risiko | Sannsynlighet | Konsekvens | Tiltak |
|--------|---------------|------------|--------|
| **Data drift i production** → degradert modellytelse | Høy | Høy (feilaktige vedtak) | Azure ML Model Monitoring med weekly alerts, automated retraining trigger ved drift > threshold |
| **Incomplete training data** → biased model | Middels | Høy (diskriminering) | Purview completeness rules (e.g., "null rate < 5%"), fail training job hvis ikke oppfylt |
| **Stale data** → irrelevante prediksjoner | Middels | Middels | Purview freshness rules med daily checks, alert til data owner |
| **Schema changes i source system** → inference failures | Lav | Høy (system downtime) | Fabric materialized lake views constraints (FAIL action på schema mismatch), integration tests |

---

## Kostnad og lisensiering

### Microsoft Purview Data Quality

**Pricing model:** Data Governance Processing Units (DGPU) pay-as-you-go.

**DGPU consumption:**
- **Profiling:** ~0.5 DGPU per 100 columns profiled
- **Data quality scan:** ~1-2 DGPU per 1M rows scanned (varies by rule complexity)
- **Storage:** Metadata stored i Purview managed storage (included)

**Eksempel (monthly cost estimate, Norway East):**
- 10 data assets (avg 50 columns each) × 4 monthly profiling scans = 100 DGPU
- 10 data assets (avg 10M rows) × 4 monthly DQ scans = 80 DGPU
- Total: ~180 DGPU/month × $0.30/DGPU ≈ **$54/month** (~550 NOK)

**Confidence:** Medium — pricing varierer med rule complexity, data volume.

**Licensing:**
- Purview Data Quality er inkludert i Microsoft Purview accounts (ingen separat lisens)
- Krever: Azure subscription med contributor access til resource group

### Azure Machine Learning Model Monitoring

**Pricing components:**
- **Compute:** Monitoring jobs kjører på Azure ML compute (CPU clusters) — billed per compute hour
- **Storage:** Production inference data stored i Azure Blob — standard storage rates
- **Application Insights:** Monitoring metrics/alerts — standard AI pricing

**Eksempel (monthly cost estimate, Norway East):**
- 1 model, daily monitoring (5-minute compute per run): ~2.5 compute hours/month × $0.10/hour = $0.25
- Inference data storage (10 GB/month): $0.20
- Application Insights (1000 alerts/month): $2.00
- Total: **~$2.50/month** (~25 NOK)

**Confidence:** High.

**Licensing:**
- Krever: Azure ML workspace (included i Azure subscription)
- Model monitoring er included feature i Azure ML (no additional license)

### Microsoft Fabric Data Quality

**Pricing model:** Fabric Capacity Units (CU) for compute + OneLake storage.

**Consumption:**
- **Materialized lake views refresh:** Billed som Spark compute (CU/hour)
- **Data quality constraints:** No additional charge (native feature)
- **OneLake storage:** $0.023/GB/month (standard rate)

**Eksempel (monthly cost estimate, F64 SKU):**
- 10 MLVs, each refreshed daily (5 minutes compute per refresh): ~25 CU hours/month
- F64 SKU (64 CU): $0.54/CU/hour × 25 = $13.50
- OneLake storage (100 GB): $2.30
- Total: **~$16/month** (~165 NOK)

**Confidence:** Medium — varies with MLV complexity, data volume.

**Licensing:**
- Krever: Fabric capacity (F2 minimum, F64 recommended for production)
- Ingen separat data quality license

### Azure Databricks Expectations (Delta Live Tables)

**Pricing model:** DBU (Databricks Units) + Azure VM compute.

**Consumption:**
- **DLT pipelines:** Billed som Jobs Compute (1.0 DBU/hour × Azure VM rate)
- **Expectations:** No additional charge (native DLT feature)

**Eksempel (monthly cost estimate, Standard_DS3_v2):**
- 1 DLT pipeline, 4 daily runs (30 minutes per run): ~60 hours/month
- Jobs Compute: 60 DBU × $0.15 (list price) = $9.00
- Azure VM (DS3_v2): 60 hours × $0.20 = $12.00
- Total: **~$21/month** (~215 NOK)

**Confidence:** High.

**Licensing:**
- Krever: Azure Databricks workspace (Premium tier for Unity Catalog)
- DLT included i Premium tier (no additional license)

---

## For arkitekten (Cosmo)

### Key takeaways

1. **No single tool solves all data quality needs.** Microsoft-stacken krever hybrid approach:
   - **Purview** for enterprise governance, multi-cloud, compliance documentation
   - **Azure ML Monitoring** for deployed model drift detection
   - **Fabric** for lakehouse-native quality enforcement
   - **Databricks** for Spark-based pipeline quality (hvis bruker Databricks)

2. **Layered quality gates are essential.** Apply different quality checks ved hver layer (Bronze/Silver/Gold):
   - Bronze: Fail fast på critical schema violations
   - Silver: Isolate bad data (DROP action), log for remediation
   - Gold: Enterprise-wide scoring med Purview

3. **Automate quality monitoring for production models.** Data drift er inevitable — set up automated alerts + retraining triggers.

4. **Golden datasets reduce model training variability.** Maintain versioned, immutable validation datasets for consistent benchmarking.

5. **Data quality er compliance-kritisk for offentlig sektor.** Forvaltningsloven § 11b + AI Act krever dokumentert data quality management.

### Vanlige fallgruver

| Fallgruve | Symptom | Fix |
|-----------|---------|-----|
| **Manual quality checks** | Data quality issues oppdages for sent (post-inference) | Automate profiling + alerts med Purview eller Azure ML Monitoring |
| **Quality checks kun på training data** | Production data drifter, model performance degraderer | Implement continuous monitoring på production inference data |
| **No baseline for drift detection** | Vanskelig å vite når data quality er acceptable | Establish quality thresholds basert på training data profiling (e.g., "null rate < 5%") |
| **Ignoring data lineage** | Root cause analysis tar for lang tid når issues oppstår | Enable Unity Catalog lineage eller Purview lineage for all data assets |
| **No golden dataset** | Model performance varierer på tvers av testing runs | Create versioned golden dataset, use for consistent validation |

### Anti-patterns å unngå

1. **"Data cleaning fixes all quality issues":** Nei — data cleaning er remediation, ikke prevention. Bruk quality constraints (Fabric, Databricks) for å forhindre dårlig data fra å komme inn i systemet.
2. **"Quality checks kun ved training time":** Production data endrer seg — continuous monitoring er obligatorisk.
3. **"One-size-fits-all quality rules":** Forskjellige data tiers (Bronze/Silver/Gold) krever forskjellige quality levels. Gold-tier data for vedtakssystemer må ha strengere rules enn Bronze-tier raw data.
4. **"Quality monitoring uten alerts":** Metrics uten actionable alerts er bortkastet effort. Set thresholds + notifications.

### Spørsmål å stille kunden

1. **"Har dere eksisterende data governance framework?"** (hvis ja → extend med Purview, hvis nei → start med Purview Data Quality)
2. **"Hvilke data sources brukes til AI?"** (multi-cloud → Purview, Fabric-only → Fabric native DQ, Databricks → DLT expectations)
3. **"Er modellen deployed til production?"** (hvis ja → prioriter Azure ML Model Monitoring, hvis nei → focus på training data profiling)
4. **"Hvor ofte oppdateres source data?"** (real-time → Fabric Eventstream DQ, batch daily → Purview scheduled scans)
5. **"Hvilke compliance-krav må dere møte?"** (Forvaltningsloven § 11b, AI Act, GDPR → Purview for audit logs + documentation)
6. **"Har dere budget for data quality tooling?"** (limited → start med Azure ML DQ monitoring signal (gratis), gradual expansion til Purview)
7. **"Hvem er data owner/steward?"** (sentralisert team → Purview centralized governance, distribuert → Unity Catalog distributed governance model)

### Anbefalte arkitektur-patterns per scenario

| Scenario | Pattern | Tools |
|----------|---------|-------|
| **Offentlig sektor AI-vedtakssystem** | Layered Quality Gates + Compliance Documentation | Fabric (Bronze/Silver constraints) + Purview (Gold scoring + audit logs) + Azure ML Monitoring (production) |
| **Real-time fraud detection** | Real-Time Data Quality for Streaming AI | Fabric Eventstream + KQL DB + Azure ML Monitoring |
| **Multi-cloud data lake** | Enterprise Governance with Multi-Cloud Sources | Purview Data Quality + Unity Catalog (hvis Databricks) |
| **Spark-based ETL pipelines** | Declarative Quality Constraints | Databricks DLT expectations |
| **Quick POC (low maturity)** | Training vs. Production Monitoring | Azure ML data quality monitoring signal |

---

## Kilder og verifisering

### Microsoft dokumentasjon

1. **Microsoft Purview Data Quality Overview**
   https://learn.microsoft.com/en-us/purview/data-quality-overview
   *Lesedato: 2026-02* — Comprehensive guide to Purview DQ lifecycle, features, pricing.

2. **Azure Machine Learning Model Monitoring**
   https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring
   *Lesedato: 2026-02* — Monitoring signals, best practices, setup instructions.

3. **Fabric Materialized Lake Views Data Quality**
   https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/data-quality
   *Lesedato: 2026-02* — MLV constraints syntax, FAIL/DROP actions, limitations.

4. **Databricks Data Governance Best Practices**
   https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
   *Lesedato: 2026-02* — Data quality standards, DLT expectations, Unity Catalog lineage.

5. **Test and Evaluate AI Workloads on Azure**
   https://learn.microsoft.com/en-us/azure/well-architected/ai/test
   *Lesedato: 2026-02* — Testing guidance for model training/fine-tuning, data quality checks.

6. **Purview Data Quality for Fabric Lakehouse**
   https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-fabric-lakehouse
   *Lesedato: 2026-02* — Setup guide for Purview DQ scan on Fabric data.

7. **Azure ML Configure Training, Validation, Cross-Validation Data**
   https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-cross-validation-data-splits
   *Lesedato: 2026-02* — Best practices for data splits, validation dataset setup.

8. **Cloud Adoption Framework: Data Quality**
   https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/govern-data-quality
   *Lesedato: 2026-02* — Data quality considerations, recommendations for enterprise scale.

### Code samples (Microsoft Learn)

- **Azure ML fine-tuning job with validation data**
  https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/fine-tune-serverless
  Python SDK sample for creating validation dataset asset.

- **AutoML training/validation MLTable inputs**
  https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-image-models
  Python SDK sample for Input objects (training/validation data).

- **MLflow traces quality analysis**
  https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/build-eval-dataset
  Python sample for filtering traces by quality score, correlation analysis.

### Confidence markers

- **GA status:** Microsoft Purview Data Quality (GA), Azure ML Model Monitoring (GA), Fabric data quality (GA), Databricks DLT expectations (GA) — alle features er production-ready per 2026-02.
- **Pricing estimates:** Medium-High confidence — basert på official pricing pages (learn.microsoft.com), men actual consumption varierer med data volume og rule complexity.
- **Virtual Network support:** Purview DQ for Fabric Lakehouse via Private Link er bekreftet i official docs (as of 2026-02), men BigQuery virtual network support er eksplisitt dokumentert som "not yet supported."
- **Integration gaps:** Azure ML + Purview native integration finnes ikke (as of 2026-02) — workaround via Azure Functions er architectural recommendation, ikke officially supported pattern.

### Sist verifisert

**2026-02-11** — Alle lenker testet, pricing bekreftet mot official docs.