ms-ai-architect/skills/ms-ai-engineering/references/data-engineering/data-quality-ai-frameworks.md

Data Quality Frameworks for AI

Last updated: 2026-06-24 Status: GA (Microsoft Purview Data Quality, Azure ML Model Monitoring, Fabric data quality) Category: Data Engineering for AI Type: reference Source: https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring

---

Innhold

• Introduksjon
• Kjernekomponenter
• Arkitekturmønstre
• Beslutningsveiledning
• Integrasjon med Microsoft-stakken
• Offentlig sektor (Norge)
• Kostnad og lisensiering
• For arkitekten (Cosmo)
• Kilder og verifisering

Introduksjon

Data quality frameworks for AI sikrer at data som brukes til trening, validering og inferens av AI-modeller er nøyaktig, komplett, konsistent og pålitelig. I dagens AI-drevne landskap påvirker datakvalitet direkte AI-ytelsen, modellens nøyaktighet, og tillit til AI-beslutninger.

Dårlig datakvalitet fører til:

• Degradert modellytelse over tid (data drift, prediction drift)
• Feilaktige innsikter og anbefalinger
• Erosjon av tillit til AI-systemer
• Compliance-risiko (GDPR, AI Act, Forvaltningsloven)
• Operasjonelle ineffektiviteter og økte kostnader

Microsoft-stacken tilbyr fire hovedspor for data quality management i AI-kontekst:

1. Microsoft Purview Data Quality — enterprise data governance med AI-powered profiling og scoring
2. Azure Machine Learning Model Monitoring — production model monitoring med data drift detection
3. Microsoft Fabric data quality — lakehouse-native quality constraints i materialized lake views og pipelines
4. Azure Databricks expectations — declarative data quality constraints i Delta Live Tables (DLT)

---

Kjernekomponenter

1. Data Quality Dimensions (Six Industry Standards)

Dimensjon	Definisjon	Microsoft tooling
Completeness	Grad av ikke-null verdier	Purview (null value rate), Databricks expectations, Fabric constraints
Accuracy	Data reflekterer real-world verdier	Azure ML model performance monitoring, ground truth comparison
Consistency	Data integrity opprettholdes på tvers av systemer	Purview cross-source validation, Unity Catalog referential integrity
Timeliness	Data oppdateres og er tilgjengelig i tide	Purview freshness rules, Azure ML prediction drift, Fabric streaming quality checks
Uniqueness	Ingen duplikater	Purview uniqueness metrics, Databricks DLT deduplication
Conformity	Data overholder format og schema	Purview data type error rate, Azure ML schema validation

2. Data Quality Lifecycle

1. [Data ingestion] → Schema validation, type checks
2. [Enrichment] → Profiling, quality rules application
3. [Training] → Data drift monitoring vs. training baseline
4. [Deployment] → Production inference data quality
5. [Monitoring] → Continuous data quality scoring, alerts
6. [Remediation] → Data quality actions, root cause analysis

3. Microsoft Purview Data Quality — Enterprise Governance

Capabilities:

• AI-powered profiling: Anbefaler kolonner for profiling, generer statistikk (distribution, min, max, std dev, uniqueness, completeness)
• No-code/low-code rules: Out-of-box rules + AI-generated rules + custom rules
• Data quality scoring: Aggregert scoring på column → data asset → data product → governance domain nivå
• Alerts: Email notifications ved threshold breaches
• Actions center: Diagnostic queries for stewards til å identifisere spesifikke rader som feiler quality checks

Supported data sources (multi-cloud):

• Azure: Blob Storage, ADLS Gen2, Azure SQL DB, Synapse, Fabric Lakehouse (Delta/Iceberg format)
• AWS: S3 (Parquet, CSV, Delta), RDS
• GCP: BigQuery (limited support for virtual network)

Pricing: Data Governance Processing Units (DGPU) pay-as-you-go

Limitations:

• Managed Identity som eneste auth-metode
• Parquet: støtter kun (1) directory med part files eller (2) partitioned Parquet (year=2018/month=Dec) — ikke arbitrary hierarchies
• Virtual network ikke støttet for BigQuery

4. Azure Machine Learning Model Monitoring

Monitoring signals (tabular data):

Signal	Beskrivelse	Metrics	Reference data
Data drift	Endring i distribution av model input	Jensen-Shannon Distance, PSI, Normalized Wasserstein, KS test, Chi-squared	Training data eller recent production data
Prediction drift	Endring i distribution av model output	Jensen-Shannon Distance, PSI, Normalized Wasserstein, Chebyshev Distance, KS test, Chi-squared	Validation data eller recent production data
Data quality	Integrity av model input	Null value rate, Data type error rate, Out-of-bounds rate	Training data eller recent production data
Feature attribution drift	Endring i feature importance	Normalized Discounted Cumulative Gain (NDCG)	Training data med feature importance
Model performance	Ground truth vs. predictions	Accuracy, Precision, Recall, F1, AUC, MAE, RMSE	Ground truth labels (actuals)

Best practices:

1. Start monitoring umiddelbart etter production deployment
2. Bruk multiple signals (e.g., data drift + feature attribution drift for early warnings)
3. Set monitoring frequency basert på data accumulation rate (daily hvis heavy traffic, weekly/monthly ellers)
4. Monitor top N features for large feature sets (reduserer cost og noise)
5. Bruk training data som baseline for data drift, validation data for prediction drift
6. Hvis ground truth er tilgjengelig: bruk model performance signal for objective view

Setup:

1. Enable production inference data collection (automatic for Azure ML online endpoints, manual for batch/external endpoints)
2. Set up model monitoring via SDK/CLI eller studio UI
3. View/analyze results i Azure ML workspace

5. Microsoft Fabric Data Quality

Materialized Lake Views (MLVs) constraints:

-- Example: exclude null customerName rows
CREATE MATERIALIZED LAKE VIEW sales_clean
(CONSTRAINT cust_blank CHECK (customerName IS NOT NULL) ON MISMATCH DROP)  -- or FAIL (default)
AS SELECT * FROM raw_sales;

Actions:

• FAIL — stops refresh ved første constraint violation (default)
• DROP — processes og fjerner records som ikke møter constraint, gir count i lineage view

Limitations:

• Constraints kan ikke oppdateres etter MLV creation (må recreate)
• Innebygde Spark/SQL-funksjoner (UPPER, TRIM, COALESCE, SUBSTRING …), UDF-er (spark.udf.register()), Pandas-UDF-er og Fabric User Data Functions støttes i constraints; ren regex/LIKE er ikke et eget constraint-konstrukt — bruk funksjoner/UDF-er i stedet
• Known issue: MLV med FAIL action kan gi "delta table not found" error (workaround: unngå FAIL, bruk DROP)

Data quality for Fabric Lakehouse (Purview integration):

1. Mirror eller load data til Fabric Lakehouse i Delta/Iceberg format
2. Grant Contributor access til workspace for Purview MSI
3. Run Data Map scan på Lakehouse (service principal auth)
4. Associate Lakehouse tables med data product i Purview Unified Catalog
5. Profile + create rules + run data quality scan via Purview

6. Azure Databricks Expectations (Lakeflow Spark Declarative Pipelines, tidl. Delta Live Tables/DLT)

Delta Live Tables (DLT) er rebrandet til Lakeflow Spark Declarative Pipelines (SDP). Ingen migrasjon kreves — import dlt/@dlt.* fungerer fortsatt, men anbefalt syntaks er nå from pyspark import pipelines as dp + @dp.* (classic SKU-er og event-log-skjema beholder fortsatt DLT-navnet).

Expectations syntax:

from pyspark import pipelines as dp  # tidl. import dlt

@dp.expect("valid_timestamp", "timestamp IS NOT NULL")
@dp.expect_or_drop("valid_amount", "amount > 0")
@dp.expect_or_fail("critical_id", "id IS NOT NULL")
def clean_transactions():
    return spark.read.table("raw_transactions")

Actions:

• @dp.expect — track violations, allow records to pass
• @dp.expect_or_drop — drop violating records
• @dp.expect_or_fail — fail pipeline ved violations

Benefits:

• Catch data quality issues ved ingestion før de påvirker downstream data products
• Real-time quality metrics i Lakeflow SDP pipeline observability UI (event log)
• Automatically generated data quality dashboards

---

Arkitekturmønstre

Pattern 1: Layered Quality Gates (Medallion Architecture)

[Bronze Layer] → Schema validation, null checks
    ↓ (Fabric constraints: FAIL on critical nulls)
[Silver Layer] → Business rules, referential integrity, deduplication
    ↓ (Databricks expectations: DROP invalid records)
[Gold Layer] → Aggregations, data quality scoring
    ↓ (Purview: Profile + score + alert)
[Consumption] → AI model training/inference

Decision criteria:

• Bronze: Fail fast på kritiske schema violations (FAIL action)
• Silver: Isoler dårlig data (DROP action), log violations for remediation
• Gold: Enterprise-wide governance med Purview scoring

Pattern 2: Training vs. Production Data Quality Monitoring

Training phase:

1. Purview profiling på training dataset
2. Define baseline quality thresholds (e.g., "null rate < 5%")
3. Apply rules, score data quality
4. Iterate: fix issues → retrain → validate

Production phase:

1. Azure ML Model Monitoring: track data drift (production vs. training baseline)
2. Set alerts: e.g., "Jensen-Shannon Distance > 0.3"
3. When alert fires → trigger Purview profiling på production data → compare profiles → investigate drift root cause

Integration:

• Azure ML Monitoring detekterer drift → triggers Azure Function → calls Purview API for detailed profiling → stores results i Azure Monitor logs

Pattern 3: Golden Datasets for Consistent Validation

Concept: Authoritative datasets som representerer production data patterns, brukt for testing og validation across alle AI workloads.

Implementation:

1. Identify representative sample fra production data (stratified sampling)
2. Store i Azure Blob/ADLS Gen2 som immutable dataset (versioned)
3. Register i Azure ML as data asset (type: URIFolder)
4. Use in Azure ML training jobs as validation dataset
5. Update golden dataset quarterly (eller når significant drift detekteres)

Quality checks on golden dataset:

• Purview profiling + scoring (monthly)
• Azure ML data quality monitoring (continuous comparison mot latest production data)

Pattern 4: Real-Time Data Quality for Streaming AI

Scenario: Real-time fraud detection model consuming transaction stream.

Architecture:

[Event Hub] → [Fabric Eventstream] → [Lakehouse (KQL DB)]
                ↓
    [Real-time aggregation pipeline med data quality constraints]
                ↓
    [Model inference endpoint] → [Azure ML Model Monitoring]

Quality checks:

1. Ingestion: Fabric eventstream DQ checks (schema validation, null checks)
2. Aggregation: KQL queries med quality assertions (e.g., assert(count(fraud_score > 1.0) == 0))
3. Inference: Azure ML Model Monitoring tracks prediction drift i real-time (5-minute windows)

---

Beslutningsveiledning

Når velge hvilket verktøy?

Scenario	Anbefalt verktøy	Rationale
Enterprise-wide data governance på tvers av multi-cloud sources	Microsoft Purview Data Quality	Centralized governance domain structure, multi-cloud support, AI-powered profiling
Production AI model monitoring (deployed til Azure ML online endpoint)	Azure ML Model Monitoring	Out-of-box integration med Azure ML endpoints, automatic inference data collection
Lakehouse-native data quality i Fabric	Fabric materialized lake views constraints	Native Delta/Iceberg support, lineage integration, no external dependencies
Spark-based data pipelines med strict quality enforcement	Databricks expectations (DLT)	Declarative syntax, automatic DQ dashboards, fail pipeline on critical violations
Early-stage data quality exploration (low maturity)	Azure ML data quality monitoring signal	No infrastructure setup, quick start med training dataset comparison
Compliance-driven data quality documentation	Purview Data Quality	Audit logs, data quality scores per governance domain, alert notifications
Real-time streaming data quality	Fabric Eventstream + KQL DB + Azure ML Monitoring	Low-latency validation, real-time drift detection

Decision tree: Velge data quality framework

START
  ↓
Er du i Fabric-økosystemet?
  └─ JA → Bruk Fabric materialized lake views constraints for layered DQ
  └─ NEI → Har du Azure ML deployed models?
      └─ JA → Bruk Azure ML Model Monitoring for production monitoring
      └─ NEI → Har du multi-cloud data sources?
          └─ JA → Bruk Purview Data Quality for enterprise governance
          └─ NEI → Bruk Databricks expectations (DLT) for Spark pipelines

---

Integrasjon med Microsoft-stakken

1. Purview + Fabric Integration

Setup:

1. Fabric Lakehouse: load data i Delta/Iceberg format
2. Fabric workspace: grant Contributor til Purview MSI
3. Purview Data Map: register Fabric source, run scan (service principal auth)
4. Purview Unified Catalog: add Lakehouse tables til data product
5. Purview Data Quality: configure connection (Tenant ID, Workspace ID, Lakehouse ID, Purview MSI credential)
6. Profile → create rules → run DQ scan

Virtual Network support:

• If Fabric tenant uses Private Link: check "Virtual Network" i connection screen, specify Azure region, add Private Link Resource ID
• Set up Purview compute allocation for managed virtual network (separate step)

2. Azure ML + Purview Integration (Conceptual — Manual)

Gap: Ingen native integration (as of 2026-02).

Workaround:

1. Azure ML Model Monitoring detects data drift → logs til Application Insights
2. Azure Function (triggered by App Insights alert) → calls Purview REST API → initiates profiling job on production data asset
3. Purview completes profiling → stores results → sends alert via email/Event Grid
4. Data scientist reviews Purview profile vs. training data profile → decides on retraining

Code snippet (Azure Function trigger):

import requests
from azure.identity import DefaultAzureCredential

def trigger_purview_profiling(data_asset_id, connection_id):
    credential = DefaultAzureCredential()
    token = credential.get_token("https://purview.azure.net/.default").token

    headers = {"Authorization": f"Bearer {token}"}
    url = f"https://{purview_account}.purview.azure.com/dataQuality/assets/{data_asset_id}/profile"
    payload = {"connectionId": connection_id}

    response = requests.post(url, headers=headers, json=payload)
    return response.json()

3. Fabric + Azure ML Integration

Scenario: Train model i Azure ML, deploy til Azure ML online endpoint, monitor med Fabric-based data quality.

Architecture:

[Fabric Lakehouse] → Training data (Delta table)
    ↓ (Azure ML reads via OneLake)
[Azure ML training job] → Model artifact
    ↓ (Deploy)
[Azure ML online endpoint] → Inference data collection
    ↓ (Write back to Fabric Lakehouse via Azure Function)
[Fabric Lakehouse inference table] → Purview DQ scan

Implementation:

1. Azure ML online endpoint: enable data collection (stores JSON logs i Azure Blob)
2. Azure Function (Event Grid trigger on blob creation) → parses JSON → writes til Fabric Lakehouse via REST API
3. Fabric Lakehouse: Purview DQ scan på inference table → compare med training table profile

4. Databricks + Unity Catalog Data Quality

Unity Catalog features:

• Data lineage: Column-level lineage fra source tables til trained models (captured via mlflow.log_model())
• Audit logging: Captures metadata access events (who accessed which dataset, when)
• Data quality via DLT expectations: Enforces quality constraints in pipelines

Best practice:

1. Define data quality standards i Unity Catalog tags (e.g., "dq_tier": "gold" for high-quality tables)
2. Apply expectations i DLT pipelines based on tier (e.g., gold tier → expect_or_fail, silver tier → expect_or_drop)
3. Monitor lineage: hvis model performance degraderer → trace back til upstream table via lineage → identify quality issues

---

Offentlig sektor (Norge)

1. Compliance-krav

Forvaltningsloven § 11b (automatisert enkeltvedtak):

• Krav: "Datagrunnlaget må være kvalitetssikret."
• Implementering: Purview Data Quality scoring (minimum score: 80/100 for Gold-tier data brukt i vedtakssystemer) + monthly profiling + audit log retention (3 år)

AI Act (High-Risk AI Systems):

• Krav (Article 10): Data quality management, including data governance, relevance, representativeness, accuracy, completeness.
• Implementering:
  • Purview Data Quality: document data quality scores i compliance reports
  • Azure ML Model Monitoring: track data drift, maintain audit trail
  • Quarterly data quality review (documented i DPIA)

GDPR Article 5(1)(d) — accuracy:

• Krav: Personal data må være "accurate and, where necessary, kept up to date."
• Implementering: Purview freshness rules (e.g., "data må være < 24 timer gammel"), alert til data owner ved breach

2. Utredningsinstruksen § 4 — kvalitetssikring

Krav: "Utredningen skal være basert på best tilgjengelig kunnskap og data."

Data quality checklist for AI-utredning:

• Training data profiling report (Purview) vedlagt utredningen
• Data quality scores for alle datasets (minimum 70/100)
• Data lineage dokumentert (Unity Catalog eller Purview lineage view)
• Training vs. validation data drift analysis (Azure ML Monitoring)
• Golden dataset versioned og arkivert (Azure Blob immutable storage)
• Data quality monitoring plan for production phase (frequency, thresholds, alert recipients)

3. ROS-analyse: Data quality risks

Risiko	Sannsynlighet	Konsekvens	Tiltak
Data drift i production → degradert modellytelse	Høy	Høy (feilaktige vedtak)	Azure ML Model Monitoring med weekly alerts, automated retraining trigger ved drift > threshold
Incomplete training data → biased model	Middels	Høy (diskriminering)	Purview completeness rules (e.g., "null rate < 5%"), fail training job hvis ikke oppfylt
Stale data → irrelevante prediksjoner	Middels	Middels	Purview freshness rules med daily checks, alert til data owner
Schema changes i source system → inference failures	Lav	Høy (system downtime)	Fabric materialized lake views constraints (FAIL action på schema mismatch), integration tests

---

Kostnad og lisensiering

Microsoft Purview Data Quality

Pricing model: Data Governance Processing Units (DGPU) pay-as-you-go.

DGPU consumption:

• Profiling: ~0.5 DGPU per 100 columns profiled
• Data quality scan: ~1-2 DGPU per 1M rows scanned (varies by rule complexity)
• Storage: Metadata stored i Purview managed storage (included)

Eksempel (monthly cost estimate, Norway East):

• 10 data assets (avg 50 columns each) × 4 monthly profiling scans = 100 DGPU
• 10 data assets (avg 10M rows) × 4 monthly DQ scans = 80 DGPU
• Total: ~180 DGPU/month × $0.30/DGPU ≈ $54/month (~550 NOK)

Confidence: Medium — pricing varierer med rule complexity, data volume.

Licensing:

• Purview Data Quality er inkludert i Microsoft Purview accounts (ingen separat lisens)
• Krever: Azure subscription med contributor access til resource group

Azure Machine Learning Model Monitoring

Pricing components:

• Compute: Monitoring jobs kjører på Azure ML compute (CPU clusters) — billed per compute hour
• Storage: Production inference data stored i Azure Blob — standard storage rates
• Application Insights: Monitoring metrics/alerts — standard AI pricing

Eksempel (monthly cost estimate, Norway East):

• 1 model, daily monitoring (5-minute compute per run): ~2.5 compute hours/month × $0.10/hour = $0.25
• Inference data storage (10 GB/month): $0.20
• Application Insights (1000 alerts/month): $2.00
• Total: ~$2.50/month (~25 NOK)

Confidence: High.

Licensing:

• Krever: Azure ML workspace (included i Azure subscription)
• Model monitoring er included feature i Azure ML (no additional license)

Microsoft Fabric Data Quality

Pricing model: Fabric Capacity Units (CU) for compute + OneLake storage.

Consumption:

• Materialized lake views refresh: Billed som Spark compute (CU/hour)
• Data quality constraints: No additional charge (native feature)
• OneLake storage: $0.023/GB/month (standard rate)

Eksempel (monthly cost estimate, F64 SKU):

• 10 MLVs, each refreshed daily (5 minutes compute per refresh): ~25 CU hours/month
• F64 SKU (64 CU): $0.54/CU/hour × 25 = $13.50
• OneLake storage (100 GB): $2.30
• Total: ~$16/month (~165 NOK)

Confidence: Medium — varies with MLV complexity, data volume.

Licensing:

• Krever: Fabric capacity (F2 minimum, F64 recommended for production)
• Ingen separat data quality license

Azure Databricks Expectations (Delta Live Tables)

Pricing model: DBU (Databricks Units) + Azure VM compute.

Consumption:

• DLT pipelines: Billed som Jobs Compute (1.0 DBU/hour × Azure VM rate)
• Expectations: No additional charge (native DLT feature)

Eksempel (monthly cost estimate, Standard_DS3_v2):

• 1 DLT pipeline, 4 daily runs (30 minutes per run): ~60 hours/month
• Jobs Compute: 60 DBU × $0.15 (list price) = $9.00
• Azure VM (DS3_v2): 60 hours × $0.20 = $12.00
• Total: ~$21/month (~215 NOK)

Confidence: High.

Licensing:

• Krever: Azure Databricks workspace (Premium tier for Unity Catalog)
• DLT included i Premium tier (no additional license)

---

For arkitekten (Cosmo)

Key takeaways

1. No single tool solves all data quality needs. Microsoft-stacken krever hybrid approach:

   • Purview for enterprise governance, multi-cloud, compliance documentation
   • Azure ML Monitoring for deployed model drift detection
   • Fabric for lakehouse-native quality enforcement
   • Databricks for Spark-based pipeline quality (hvis bruker Databricks)

2. Layered quality gates are essential. Apply different quality checks ved hver layer (Bronze/Silver/Gold):

   • Bronze: Fail fast på critical schema violations
   • Silver: Isolate bad data (DROP action), log for remediation
   • Gold: Enterprise-wide scoring med Purview

3. Automate quality monitoring for production models. Data drift er inevitable — set up automated alerts + retraining triggers.

4. Golden datasets reduce model training variability. Maintain versioned, immutable validation datasets for consistent benchmarking.

5. Data quality er compliance-kritisk for offentlig sektor. Forvaltningsloven § 11b + AI Act krever dokumentert data quality management.

Vanlige fallgruver

Fallgruve	Symptom	Fix
Manual quality checks	Data quality issues oppdages for sent (post-inference)	Automate profiling + alerts med Purview eller Azure ML Monitoring
Quality checks kun på training data	Production data drifter, model performance degraderer	Implement continuous monitoring på production inference data
No baseline for drift detection	Vanskelig å vite når data quality er acceptable	Establish quality thresholds basert på training data profiling (e.g., "null rate < 5%")
Ignoring data lineage	Root cause analysis tar for lang tid når issues oppstår	Enable Unity Catalog lineage eller Purview lineage for all data assets
No golden dataset	Model performance varierer på tvers av testing runs	Create versioned golden dataset, use for consistent validation

Anti-patterns å unngå

1. "Data cleaning fixes all quality issues": Nei — data cleaning er remediation, ikke prevention. Bruk quality constraints (Fabric, Databricks) for å forhindre dårlig data fra å komme inn i systemet.
2. "Quality checks kun ved training time": Production data endrer seg — continuous monitoring er obligatorisk.
3. "One-size-fits-all quality rules": Forskjellige data tiers (Bronze/Silver/Gold) krever forskjellige quality levels. Gold-tier data for vedtakssystemer må ha strengere rules enn Bronze-tier raw data.
4. "Quality monitoring uten alerts": Metrics uten actionable alerts er bortkastet effort. Set thresholds + notifications.

Spørsmål å stille kunden

1. "Har dere eksisterende data governance framework?" (hvis ja → extend med Purview, hvis nei → start med Purview Data Quality)
2. "Hvilke data sources brukes til AI?" (multi-cloud → Purview, Fabric-only → Fabric native DQ, Databricks → DLT expectations)
3. "Er modellen deployed til production?" (hvis ja → prioriter Azure ML Model Monitoring, hvis nei → focus på training data profiling)
4. "Hvor ofte oppdateres source data?" (real-time → Fabric Eventstream DQ, batch daily → Purview scheduled scans)
5. "Hvilke compliance-krav må dere møte?" (Forvaltningsloven § 11b, AI Act, GDPR → Purview for audit logs + documentation)
6. "Har dere budget for data quality tooling?" (limited → start med Azure ML DQ monitoring signal (gratis), gradual expansion til Purview)
7. "Hvem er data owner/steward?" (sentralisert team → Purview centralized governance, distribuert → Unity Catalog distributed governance model)

Anbefalte arkitektur-patterns per scenario

Scenario	Pattern	Tools
Offentlig sektor AI-vedtakssystem	Layered Quality Gates + Compliance Documentation	Fabric (Bronze/Silver constraints) + Purview (Gold scoring + audit logs) + Azure ML Monitoring (production)
Real-time fraud detection	Real-Time Data Quality for Streaming AI	Fabric Eventstream + KQL DB + Azure ML Monitoring
Multi-cloud data lake	Enterprise Governance with Multi-Cloud Sources	Purview Data Quality + Unity Catalog (hvis Databricks)
Spark-based ETL pipelines	Declarative Quality Constraints	Databricks DLT expectations
Quick POC (low maturity)	Training vs. Production Monitoring	Azure ML data quality monitoring signal

---

Kilder og verifisering

Microsoft dokumentasjon

1. Microsoft Purview Data Quality Overview https://learn.microsoft.com/en-us/purview/data-quality-overview Lesedato: 2026-02 — Comprehensive guide to Purview DQ lifecycle, features, pricing.

2. Azure Machine Learning Model Monitoring https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring Lesedato: 2026-02 — Monitoring signals, best practices, setup instructions.

3. Fabric Materialized Lake Views Data Quality https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/data-quality Lesedato: 2026-02 — MLV constraints syntax, FAIL/DROP actions, limitations.

4. Databricks Data Governance Best Practices https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices Lesedato: 2026-02 — Data quality standards, DLT expectations, Unity Catalog lineage.

5. Test and Evaluate AI Workloads on Azure https://learn.microsoft.com/en-us/azure/well-architected/ai/test Lesedato: 2026-02 — Testing guidance for model training/fine-tuning, data quality checks.

6. Purview Data Quality for Fabric Lakehouse https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-fabric-lakehouse Lesedato: 2026-02 — Setup guide for Purview DQ scan on Fabric data.

7. Azure ML Configure Training, Validation, Cross-Validation Data https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-cross-validation-data-splits Lesedato: 2026-02 — Best practices for data splits, validation dataset setup.

8. Cloud Adoption Framework: Data Quality https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/govern-data-quality Lesedato: 2026-02 — Data quality considerations, recommendations for enterprise scale.

Code samples (Microsoft Learn)

• Azure ML fine-tuning job with validation data https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/fine-tune-serverless Python SDK sample for creating validation dataset asset.

• AutoML training/validation MLTable inputs https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-image-models Python SDK sample for Input objects (training/validation data).

• MLflow traces quality analysis https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/build-eval-dataset Python sample for filtering traces by quality score, correlation analysis.

Confidence markers

• GA status: Microsoft Purview Data Quality (GA), Azure ML Model Monitoring (GA), Fabric data quality (GA), Databricks DLT expectations (GA) — alle features er production-ready per 2026-02.
• Pricing estimates: Medium-High confidence — basert på official pricing pages (learn.microsoft.com), men actual consumption varierer med data volume og rule complexity.
• Virtual Network support: Purview DQ for Fabric Lakehouse via Private Link er bekreftet i official docs (as of 2026-02), men BigQuery virtual network support er eksplisitt dokumentert som "not yet supported."
• Integration gaps: Azure ML + Purview native integration finnes ikke (as of 2026-02) — workaround via Azure Functions er architectural recommendation, ikke officially supported pattern.

Sist verifisert

2026-02-11 — Alle lenker testet, pricing bekreftet mot official docs.