feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,573 @@
|
|||
# Data Quality Frameworks for AI
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Status:** GA (Microsoft Purview Data Quality, Azure ML Model Monitoring, Fabric data quality)
|
||||
**Category:** Data Engineering for AI
|
||||
|
||||
---
|
||||
|
||||
## Introduksjon
|
||||
|
||||
Data quality frameworks for AI sikrer at data som brukes til trening, validering og inferens av AI-modeller er nøyaktig, komplett, konsistent og pålitelig. I dagens AI-drevne landskap påvirker datakvalitet direkte AI-ytelsen, modellens nøyaktighet, og tillit til AI-beslutninger.
|
||||
|
||||
Dårlig datakvalitet fører til:
|
||||
- Degradert modellytelse over tid (data drift, prediction drift)
|
||||
- Feilaktige innsikter og anbefalinger
|
||||
- Erosjon av tillit til AI-systemer
|
||||
- Compliance-risiko (GDPR, AI Act, Forvaltningsloven)
|
||||
- Operasjonelle ineffektiviteter og økte kostnader
|
||||
|
||||
Microsoft-stacken tilbyr fire hovedspor for data quality management i AI-kontekst:
|
||||
1. **Microsoft Purview Data Quality** — enterprise data governance med AI-powered profiling og scoring
|
||||
2. **Azure Machine Learning Model Monitoring** — production model monitoring med data drift detection
|
||||
3. **Microsoft Fabric data quality** — lakehouse-native quality constraints i materialized lake views og pipelines
|
||||
4. **Azure Databricks expectations** — declarative data quality constraints i Delta Live Tables (DLT)
|
||||
|
||||
---
|
||||
|
||||
## Kjernekomponenter
|
||||
|
||||
### 1. Data Quality Dimensions (Six Industry Standards)
|
||||
|
||||
| Dimensjon | Definisjon | Microsoft tooling |
|
||||
|-----------|------------|-------------------|
|
||||
| **Completeness** | Grad av ikke-null verdier | Purview (null value rate), Databricks expectations, Fabric constraints |
|
||||
| **Accuracy** | Data reflekterer real-world verdier | Azure ML model performance monitoring, ground truth comparison |
|
||||
| **Consistency** | Data integrity opprettholdes på tvers av systemer | Purview cross-source validation, Unity Catalog referential integrity |
|
||||
| **Timeliness** | Data oppdateres og er tilgjengelig i tide | Purview freshness rules, Azure ML prediction drift, Fabric streaming quality checks |
|
||||
| **Uniqueness** | Ingen duplikater | Purview uniqueness metrics, Databricks DLT deduplication |
|
||||
| **Conformity** | Data overholder format og schema | Purview data type error rate, Azure ML schema validation |
|
||||
|
||||
### 2. Data Quality Lifecycle
|
||||
|
||||
```
|
||||
1. [Data ingestion] → Schema validation, type checks
|
||||
2. [Enrichment] → Profiling, quality rules application
|
||||
3. [Training] → Data drift monitoring vs. training baseline
|
||||
4. [Deployment] → Production inference data quality
|
||||
5. [Monitoring] → Continuous data quality scoring, alerts
|
||||
6. [Remediation] → Data quality actions, root cause analysis
|
||||
```
|
||||
|
||||
### 3. Microsoft Purview Data Quality — Enterprise Governance
|
||||
|
||||
**Capabilities:**
|
||||
- **AI-powered profiling:** Anbefaler kolonner for profiling, generer statistikk (distribution, min, max, std dev, uniqueness, completeness)
|
||||
- **No-code/low-code rules:** Out-of-box rules + AI-generated rules + custom rules
|
||||
- **Data quality scoring:** Aggregert scoring på column → data asset → data product → governance domain nivå
|
||||
- **Alerts:** Email notifications ved threshold breaches
|
||||
- **Actions center:** Diagnostic queries for stewards til å identifisere spesifikke rader som feiler quality checks
|
||||
|
||||
**Supported data sources (multi-cloud):**
|
||||
- Azure: Blob Storage, ADLS Gen2, Azure SQL DB, Synapse, Fabric Lakehouse (Delta/Iceberg format)
|
||||
- AWS: S3 (Parquet, CSV, Delta), RDS
|
||||
- GCP: BigQuery (limited support for virtual network)
|
||||
|
||||
**Pricing:** Data Governance Processing Units (DGPU) pay-as-you-go
|
||||
|
||||
**Limitations:**
|
||||
- Managed Identity som eneste auth-metode
|
||||
- Parquet: støtter kun (1) directory med part files eller (2) partitioned Parquet (year=2018/month=Dec) — ikke arbitrary hierarchies
|
||||
- Virtual network ikke støttet for BigQuery
|
||||
|
||||
### 4. Azure Machine Learning Model Monitoring
|
||||
|
||||
**Monitoring signals (tabular data):**
|
||||
|
||||
| Signal | Beskrivelse | Metrics | Reference data |
|
||||
|--------|-------------|---------|----------------|
|
||||
| **Data drift** | Endring i distribution av model input | Jensen-Shannon Distance, PSI, Normalized Wasserstein, KS test, Chi-squared | Training data eller recent production data |
|
||||
| **Prediction drift** | Endring i distribution av model output | Jensen-Shannon Distance, PSI, Normalized Wasserstein, Chebyshev Distance, KS test, Chi-squared | Validation data eller recent production data |
|
||||
| **Data quality** | Integrity av model input | Null value rate, Data type error rate, Out-of-bounds rate | Training data eller recent production data |
|
||||
| **Feature attribution drift** | Endring i feature importance | Normalized Discounted Cumulative Gain (NDCG) | Training data med feature importance |
|
||||
| **Model performance** | Ground truth vs. predictions | Accuracy, Precision, Recall, F1, AUC, MAE, RMSE | Ground truth labels (actuals) |
|
||||
|
||||
**Best practices:**
|
||||
1. Start monitoring umiddelbart etter production deployment
|
||||
2. Bruk multiple signals (e.g., data drift + feature attribution drift for early warnings)
|
||||
3. Set monitoring frequency basert på data accumulation rate (daily hvis heavy traffic, weekly/monthly ellers)
|
||||
4. Monitor top N features for large feature sets (reduserer cost og noise)
|
||||
5. Bruk training data som baseline for data drift, validation data for prediction drift
|
||||
6. Hvis ground truth er tilgjengelig: bruk model performance signal for objective view
|
||||
|
||||
**Setup:**
|
||||
1. Enable production inference data collection (automatic for Azure ML online endpoints, manual for batch/external endpoints)
|
||||
2. Set up model monitoring via SDK/CLI eller studio UI
|
||||
3. View/analyze results i Azure ML workspace
|
||||
|
||||
### 5. Microsoft Fabric Data Quality
|
||||
|
||||
**Materialized Lake Views (MLVs) constraints:**
|
||||
|
||||
```sql
|
||||
-- Example: exclude null customerName rows
|
||||
CREATE MATERIALIZED VIEW sales_clean
|
||||
CONSTRAINT cust_blank CHECK customerName IS NOT NULL
|
||||
WITH ACTION DROP -- or FAIL
|
||||
AS SELECT * FROM raw_sales;
|
||||
```
|
||||
|
||||
**Actions:**
|
||||
- `FAIL` — stops refresh ved første constraint violation (default)
|
||||
- `DROP` — processes og fjerner records som ikke møter constraint, gir count i lineage view
|
||||
|
||||
**Limitations:**
|
||||
- Constraints kan ikke oppdateres etter MLV creation (må recreate)
|
||||
- Functions og pattern search (LIKE, regex) i constraints er ikke støttet
|
||||
- Known issue: MLV med FAIL action kan gi "delta table not found" error (workaround: unngå FAIL, bruk DROP)
|
||||
|
||||
**Data quality for Fabric Lakehouse (Purview integration):**
|
||||
1. Mirror eller load data til Fabric Lakehouse i Delta/Iceberg format
|
||||
2. Grant Contributor access til workspace for Purview MSI
|
||||
3. Run Data Map scan på Lakehouse (service principal auth)
|
||||
4. Associate Lakehouse tables med data product i Purview Unified Catalog
|
||||
5. Profile + create rules + run data quality scan via Purview
|
||||
|
||||
### 6. Azure Databricks Expectations (Delta Live Tables)
|
||||
|
||||
**Expectations syntax:**
|
||||
|
||||
```python
|
||||
@dlt.expect("valid_timestamp", "timestamp IS NOT NULL")
|
||||
@dlt.expect_or_drop("valid_amount", "amount > 0")
|
||||
@dlt.expect_or_fail("critical_id", "id IS NOT NULL")
|
||||
def clean_transactions():
|
||||
return spark.read.table("raw_transactions")
|
||||
```
|
||||
|
||||
**Actions:**
|
||||
- `@dlt.expect` — track violations, allow records to pass
|
||||
- `@dlt.expect_or_drop` — drop violating records
|
||||
- `@dlt.expect_or_fail` — fail pipeline ved violations
|
||||
|
||||
**Benefits:**
|
||||
- Catch data quality issues ved ingestion før de påvirker downstream data products
|
||||
- Real-time quality metrics i DLT pipeline observability UI
|
||||
- Automatically generated data quality dashboards
|
||||
|
||||
---
|
||||
|
||||
## Arkitekturmønstre
|
||||
|
||||
### Pattern 1: Layered Quality Gates (Medallion Architecture)
|
||||
|
||||
```
|
||||
[Bronze Layer] → Schema validation, null checks
|
||||
↓ (Fabric constraints: FAIL on critical nulls)
|
||||
[Silver Layer] → Business rules, referential integrity, deduplication
|
||||
↓ (Databricks expectations: DROP invalid records)
|
||||
[Gold Layer] → Aggregations, data quality scoring
|
||||
↓ (Purview: Profile + score + alert)
|
||||
[Consumption] → AI model training/inference
|
||||
```
|
||||
|
||||
**Decision criteria:**
|
||||
- **Bronze:** Fail fast på kritiske schema violations (FAIL action)
|
||||
- **Silver:** Isoler dårlig data (DROP action), log violations for remediation
|
||||
- **Gold:** Enterprise-wide governance med Purview scoring
|
||||
|
||||
### Pattern 2: Training vs. Production Data Quality Monitoring
|
||||
|
||||
**Training phase:**
|
||||
1. Purview profiling på training dataset
|
||||
2. Define baseline quality thresholds (e.g., "null rate < 5%")
|
||||
3. Apply rules, score data quality
|
||||
4. Iterate: fix issues → retrain → validate
|
||||
|
||||
**Production phase:**
|
||||
1. Azure ML Model Monitoring: track data drift (production vs. training baseline)
|
||||
2. Set alerts: e.g., "Jensen-Shannon Distance > 0.3"
|
||||
3. When alert fires → trigger Purview profiling på production data → compare profiles → investigate drift root cause
|
||||
|
||||
**Integration:**
|
||||
- Azure ML Monitoring detekterer drift → triggers Azure Function → calls Purview API for detailed profiling → stores results i Azure Monitor logs
|
||||
|
||||
### Pattern 3: Golden Datasets for Consistent Validation
|
||||
|
||||
**Concept:** Authoritative datasets som representerer production data patterns, brukt for testing og validation across alle AI workloads.
|
||||
|
||||
**Implementation:**
|
||||
1. Identify representative sample fra production data (stratified sampling)
|
||||
2. Store i Azure Blob/ADLS Gen2 som immutable dataset (versioned)
|
||||
3. Register i Azure ML as data asset (type: URIFolder)
|
||||
4. Use in Azure ML training jobs as validation dataset
|
||||
5. Update golden dataset quarterly (eller når significant drift detekteres)
|
||||
|
||||
**Quality checks on golden dataset:**
|
||||
- Purview profiling + scoring (monthly)
|
||||
- Azure ML data quality monitoring (continuous comparison mot latest production data)
|
||||
|
||||
### Pattern 4: Real-Time Data Quality for Streaming AI
|
||||
|
||||
**Scenario:** Real-time fraud detection model consuming transaction stream.
|
||||
|
||||
**Architecture:**
|
||||
```
|
||||
[Event Hub] → [Fabric Eventstream] → [Lakehouse (KQL DB)]
|
||||
↓
|
||||
[Real-time aggregation pipeline med data quality constraints]
|
||||
↓
|
||||
[Model inference endpoint] → [Azure ML Model Monitoring]
|
||||
```
|
||||
|
||||
**Quality checks:**
|
||||
1. **Ingestion:** Fabric eventstream DQ checks (schema validation, null checks)
|
||||
2. **Aggregation:** KQL queries med quality assertions (e.g., `assert(count(fraud_score > 1.0) == 0)`)
|
||||
3. **Inference:** Azure ML Model Monitoring tracks prediction drift i real-time (5-minute windows)
|
||||
|
||||
---
|
||||
|
||||
## Beslutningsveiledning
|
||||
|
||||
### Når velge hvilket verktøy?
|
||||
|
||||
| Scenario | Anbefalt verktøy | Rationale |
|
||||
|----------|------------------|-----------|
|
||||
| Enterprise-wide data governance på tvers av multi-cloud sources | **Microsoft Purview Data Quality** | Centralized governance domain structure, multi-cloud support, AI-powered profiling |
|
||||
| Production AI model monitoring (deployed til Azure ML online endpoint) | **Azure ML Model Monitoring** | Out-of-box integration med Azure ML endpoints, automatic inference data collection |
|
||||
| Lakehouse-native data quality i Fabric | **Fabric materialized lake views constraints** | Native Delta/Iceberg support, lineage integration, no external dependencies |
|
||||
| Spark-based data pipelines med strict quality enforcement | **Databricks expectations (DLT)** | Declarative syntax, automatic DQ dashboards, fail pipeline on critical violations |
|
||||
| Early-stage data quality exploration (low maturity) | **Azure ML data quality monitoring signal** | No infrastructure setup, quick start med training dataset comparison |
|
||||
| Compliance-driven data quality documentation | **Purview Data Quality** | Audit logs, data quality scores per governance domain, alert notifications |
|
||||
| Real-time streaming data quality | **Fabric Eventstream + KQL DB + Azure ML Monitoring** | Low-latency validation, real-time drift detection |
|
||||
|
||||
### Decision tree: Velge data quality framework
|
||||
|
||||
```
|
||||
START
|
||||
↓
|
||||
Er du i Fabric-økosystemet?
|
||||
└─ JA → Bruk Fabric materialized lake views constraints for layered DQ
|
||||
└─ NEI → Har du Azure ML deployed models?
|
||||
└─ JA → Bruk Azure ML Model Monitoring for production monitoring
|
||||
└─ NEI → Har du multi-cloud data sources?
|
||||
└─ JA → Bruk Purview Data Quality for enterprise governance
|
||||
└─ NEI → Bruk Databricks expectations (DLT) for Spark pipelines
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Integrasjon med Microsoft-stakken
|
||||
|
||||
### 1. Purview + Fabric Integration
|
||||
|
||||
**Setup:**
|
||||
1. Fabric Lakehouse: load data i Delta/Iceberg format
|
||||
2. Fabric workspace: grant Contributor til Purview MSI
|
||||
3. Purview Data Map: register Fabric source, run scan (service principal auth)
|
||||
4. Purview Unified Catalog: add Lakehouse tables til data product
|
||||
5. Purview Data Quality: configure connection (Tenant ID, Workspace ID, Lakehouse ID, Purview MSI credential)
|
||||
6. Profile → create rules → run DQ scan
|
||||
|
||||
**Virtual Network support:**
|
||||
- If Fabric tenant uses Private Link: check "Virtual Network" i connection screen, specify Azure region, add Private Link Resource ID
|
||||
- Set up Purview compute allocation for managed virtual network (separate step)
|
||||
|
||||
### 2. Azure ML + Purview Integration (Conceptual — Manual)
|
||||
|
||||
**Gap:** Ingen native integration (as of 2026-02).
|
||||
|
||||
**Workaround:**
|
||||
1. Azure ML Model Monitoring detects data drift → logs til Application Insights
|
||||
2. Azure Function (triggered by App Insights alert) → calls Purview REST API → initiates profiling job on production data asset
|
||||
3. Purview completes profiling → stores results → sends alert via email/Event Grid
|
||||
4. Data scientist reviews Purview profile vs. training data profile → decides on retraining
|
||||
|
||||
**Code snippet (Azure Function trigger):**
|
||||
|
||||
```python
|
||||
import requests
|
||||
from azure.identity import DefaultAzureCredential
|
||||
|
||||
def trigger_purview_profiling(data_asset_id, connection_id):
|
||||
credential = DefaultAzureCredential()
|
||||
token = credential.get_token("https://purview.azure.net/.default").token
|
||||
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
url = f"https://{purview_account}.purview.azure.com/dataQuality/assets/{data_asset_id}/profile"
|
||||
payload = {"connectionId": connection_id}
|
||||
|
||||
response = requests.post(url, headers=headers, json=payload)
|
||||
return response.json()
|
||||
```
|
||||
|
||||
### 3. Fabric + Azure ML Integration
|
||||
|
||||
**Scenario:** Train model i Azure ML, deploy til Azure ML online endpoint, monitor med Fabric-based data quality.
|
||||
|
||||
**Architecture:**
|
||||
```
|
||||
[Fabric Lakehouse] → Training data (Delta table)
|
||||
↓ (Azure ML reads via OneLake)
|
||||
[Azure ML training job] → Model artifact
|
||||
↓ (Deploy)
|
||||
[Azure ML online endpoint] → Inference data collection
|
||||
↓ (Write back to Fabric Lakehouse via Azure Function)
|
||||
[Fabric Lakehouse inference table] → Purview DQ scan
|
||||
```
|
||||
|
||||
**Implementation:**
|
||||
1. Azure ML online endpoint: enable data collection (stores JSON logs i Azure Blob)
|
||||
2. Azure Function (Event Grid trigger on blob creation) → parses JSON → writes til Fabric Lakehouse via REST API
|
||||
3. Fabric Lakehouse: Purview DQ scan på inference table → compare med training table profile
|
||||
|
||||
### 4. Databricks + Unity Catalog Data Quality
|
||||
|
||||
**Unity Catalog features:**
|
||||
- **Data lineage:** Column-level lineage fra source tables til trained models (captured via `mlflow.log_model()`)
|
||||
- **Audit logging:** Captures metadata access events (who accessed which dataset, when)
|
||||
- **Data quality via DLT expectations:** Enforces quality constraints in pipelines
|
||||
|
||||
**Best practice:**
|
||||
1. Define data quality standards i Unity Catalog tags (e.g., `"dq_tier": "gold"` for high-quality tables)
|
||||
2. Apply expectations i DLT pipelines based on tier (e.g., gold tier → `expect_or_fail`, silver tier → `expect_or_drop`)
|
||||
3. Monitor lineage: hvis model performance degraderer → trace back til upstream table via lineage → identify quality issues
|
||||
|
||||
---
|
||||
|
||||
## Offentlig sektor (Norge)
|
||||
|
||||
### 1. Compliance-krav
|
||||
|
||||
**Forvaltningsloven § 11b (automatisert enkeltvedtak):**
|
||||
- **Krav:** "Datagrunnlaget må være kvalitetssikret."
|
||||
- **Implementering:** Purview Data Quality scoring (minimum score: 80/100 for Gold-tier data brukt i vedtakssystemer) + monthly profiling + audit log retention (3 år)
|
||||
|
||||
**AI Act (High-Risk AI Systems):**
|
||||
- **Krav (Article 10):** Data quality management, including data governance, relevance, representativeness, accuracy, completeness.
|
||||
- **Implementering:**
|
||||
- Purview Data Quality: document data quality scores i compliance reports
|
||||
- Azure ML Model Monitoring: track data drift, maintain audit trail
|
||||
- Quarterly data quality review (documented i DPIA)
|
||||
|
||||
**GDPR Article 5(1)(d) — accuracy:**
|
||||
- **Krav:** Personal data må være "accurate and, where necessary, kept up to date."
|
||||
- **Implementering:** Purview freshness rules (e.g., "data må være < 24 timer gammel"), alert til data owner ved breach
|
||||
|
||||
### 2. Utredningsinstruksen § 4 — kvalitetssikring
|
||||
|
||||
**Krav:** "Utredningen skal være basert på best tilgjengelig kunnskap og data."
|
||||
|
||||
**Data quality checklist for AI-utredning:**
|
||||
- [ ] Training data profiling report (Purview) vedlagt utredningen
|
||||
- [ ] Data quality scores for alle datasets (minimum 70/100)
|
||||
- [ ] Data lineage dokumentert (Unity Catalog eller Purview lineage view)
|
||||
- [ ] Training vs. validation data drift analysis (Azure ML Monitoring)
|
||||
- [ ] Golden dataset versioned og arkivert (Azure Blob immutable storage)
|
||||
- [ ] Data quality monitoring plan for production phase (frequency, thresholds, alert recipients)
|
||||
|
||||
### 3. ROS-analyse: Data quality risks
|
||||
|
||||
| Risiko | Sannsynlighet | Konsekvens | Tiltak |
|
||||
|--------|---------------|------------|--------|
|
||||
| **Data drift i production** → degradert modellytelse | Høy | Høy (feilaktige vedtak) | Azure ML Model Monitoring med weekly alerts, automated retraining trigger ved drift > threshold |
|
||||
| **Incomplete training data** → biased model | Middels | Høy (diskriminering) | Purview completeness rules (e.g., "null rate < 5%"), fail training job hvis ikke oppfylt |
|
||||
| **Stale data** → irrelevante prediksjoner | Middels | Middels | Purview freshness rules med daily checks, alert til data owner |
|
||||
| **Schema changes i source system** → inference failures | Lav | Høy (system downtime) | Fabric materialized lake views constraints (FAIL action på schema mismatch), integration tests |
|
||||
|
||||
---
|
||||
|
||||
## Kostnad og lisensiering
|
||||
|
||||
### Microsoft Purview Data Quality
|
||||
|
||||
**Pricing model:** Data Governance Processing Units (DGPU) pay-as-you-go.
|
||||
|
||||
**DGPU consumption:**
|
||||
- **Profiling:** ~0.5 DGPU per 100 columns profiled
|
||||
- **Data quality scan:** ~1-2 DGPU per 1M rows scanned (varies by rule complexity)
|
||||
- **Storage:** Metadata stored i Purview managed storage (included)
|
||||
|
||||
**Eksempel (monthly cost estimate, Norway East):**
|
||||
- 10 data assets (avg 50 columns each) × 4 monthly profiling scans = 100 DGPU
|
||||
- 10 data assets (avg 10M rows) × 4 monthly DQ scans = 80 DGPU
|
||||
- Total: ~180 DGPU/month × $0.30/DGPU ≈ **$54/month** (~550 NOK)
|
||||
|
||||
**Confidence:** Medium — pricing varierer med rule complexity, data volume.
|
||||
|
||||
**Licensing:**
|
||||
- Purview Data Quality er inkludert i Microsoft Purview accounts (ingen separat lisens)
|
||||
- Krever: Azure subscription med contributor access til resource group
|
||||
|
||||
### Azure Machine Learning Model Monitoring
|
||||
|
||||
**Pricing components:**
|
||||
- **Compute:** Monitoring jobs kjører på Azure ML compute (CPU clusters) — billed per compute hour
|
||||
- **Storage:** Production inference data stored i Azure Blob — standard storage rates
|
||||
- **Application Insights:** Monitoring metrics/alerts — standard AI pricing
|
||||
|
||||
**Eksempel (monthly cost estimate, Norway East):**
|
||||
- 1 model, daily monitoring (5-minute compute per run): ~2.5 compute hours/month × $0.10/hour = $0.25
|
||||
- Inference data storage (10 GB/month): $0.20
|
||||
- Application Insights (1000 alerts/month): $2.00
|
||||
- Total: **~$2.50/month** (~25 NOK)
|
||||
|
||||
**Confidence:** High.
|
||||
|
||||
**Licensing:**
|
||||
- Krever: Azure ML workspace (included i Azure subscription)
|
||||
- Model monitoring er included feature i Azure ML (no additional license)
|
||||
|
||||
### Microsoft Fabric Data Quality
|
||||
|
||||
**Pricing model:** Fabric Capacity Units (CU) for compute + OneLake storage.
|
||||
|
||||
**Consumption:**
|
||||
- **Materialized lake views refresh:** Billed som Spark compute (CU/hour)
|
||||
- **Data quality constraints:** No additional charge (native feature)
|
||||
- **OneLake storage:** $0.023/GB/month (standard rate)
|
||||
|
||||
**Eksempel (monthly cost estimate, F64 SKU):**
|
||||
- 10 MLVs, each refreshed daily (5 minutes compute per refresh): ~25 CU hours/month
|
||||
- F64 SKU (64 CU): $0.54/CU/hour × 25 = $13.50
|
||||
- OneLake storage (100 GB): $2.30
|
||||
- Total: **~$16/month** (~165 NOK)
|
||||
|
||||
**Confidence:** Medium — varies with MLV complexity, data volume.
|
||||
|
||||
**Licensing:**
|
||||
- Krever: Fabric capacity (F2 minimum, F64 recommended for production)
|
||||
- Ingen separat data quality license
|
||||
|
||||
### Azure Databricks Expectations (Delta Live Tables)
|
||||
|
||||
**Pricing model:** DBU (Databricks Units) + Azure VM compute.
|
||||
|
||||
**Consumption:**
|
||||
- **DLT pipelines:** Billed som Jobs Compute (1.0 DBU/hour × Azure VM rate)
|
||||
- **Expectations:** No additional charge (native DLT feature)
|
||||
|
||||
**Eksempel (monthly cost estimate, Standard_DS3_v2):**
|
||||
- 1 DLT pipeline, 4 daily runs (30 minutes per run): ~60 hours/month
|
||||
- Jobs Compute: 60 DBU × $0.15 (list price) = $9.00
|
||||
- Azure VM (DS3_v2): 60 hours × $0.20 = $12.00
|
||||
- Total: **~$21/month** (~215 NOK)
|
||||
|
||||
**Confidence:** High.
|
||||
|
||||
**Licensing:**
|
||||
- Krever: Azure Databricks workspace (Premium tier for Unity Catalog)
|
||||
- DLT included i Premium tier (no additional license)
|
||||
|
||||
---
|
||||
|
||||
## For arkitekten (Cosmo)
|
||||
|
||||
### Key takeaways
|
||||
|
||||
1. **No single tool solves all data quality needs.** Microsoft-stacken krever hybrid approach:
|
||||
- **Purview** for enterprise governance, multi-cloud, compliance documentation
|
||||
- **Azure ML Monitoring** for deployed model drift detection
|
||||
- **Fabric** for lakehouse-native quality enforcement
|
||||
- **Databricks** for Spark-based pipeline quality (hvis bruker Databricks)
|
||||
|
||||
2. **Layered quality gates are essential.** Apply different quality checks ved hver layer (Bronze/Silver/Gold):
|
||||
- Bronze: Fail fast på critical schema violations
|
||||
- Silver: Isolate bad data (DROP action), log for remediation
|
||||
- Gold: Enterprise-wide scoring med Purview
|
||||
|
||||
3. **Automate quality monitoring for production models.** Data drift er inevitable — set up automated alerts + retraining triggers.
|
||||
|
||||
4. **Golden datasets reduce model training variability.** Maintain versioned, immutable validation datasets for consistent benchmarking.
|
||||
|
||||
5. **Data quality er compliance-kritisk for offentlig sektor.** Forvaltningsloven § 11b + AI Act krever dokumentert data quality management.
|
||||
|
||||
### Vanlige fallgruver
|
||||
|
||||
| Fallgruve | Symptom | Fix |
|
||||
|-----------|---------|-----|
|
||||
| **Manual quality checks** | Data quality issues oppdages for sent (post-inference) | Automate profiling + alerts med Purview eller Azure ML Monitoring |
|
||||
| **Quality checks kun på training data** | Production data drifter, model performance degraderer | Implement continuous monitoring på production inference data |
|
||||
| **No baseline for drift detection** | Vanskelig å vite når data quality er acceptable | Establish quality thresholds basert på training data profiling (e.g., "null rate < 5%") |
|
||||
| **Ignoring data lineage** | Root cause analysis tar for lang tid når issues oppstår | Enable Unity Catalog lineage eller Purview lineage for all data assets |
|
||||
| **No golden dataset** | Model performance varierer på tvers av testing runs | Create versioned golden dataset, use for consistent validation |
|
||||
|
||||
### Anti-patterns å unngå
|
||||
|
||||
1. **"Data cleaning fixes all quality issues":** Nei — data cleaning er remediation, ikke prevention. Bruk quality constraints (Fabric, Databricks) for å forhindre dårlig data fra å komme inn i systemet.
|
||||
2. **"Quality checks kun ved training time":** Production data endrer seg — continuous monitoring er obligatorisk.
|
||||
3. **"One-size-fits-all quality rules":** Forskjellige data tiers (Bronze/Silver/Gold) krever forskjellige quality levels. Gold-tier data for vedtakssystemer må ha strengere rules enn Bronze-tier raw data.
|
||||
4. **"Quality monitoring uten alerts":** Metrics uten actionable alerts er bortkastet effort. Set thresholds + notifications.
|
||||
|
||||
### Spørsmål å stille kunden
|
||||
|
||||
1. **"Har dere eksisterende data governance framework?"** (hvis ja → extend med Purview, hvis nei → start med Purview Data Quality)
|
||||
2. **"Hvilke data sources brukes til AI?"** (multi-cloud → Purview, Fabric-only → Fabric native DQ, Databricks → DLT expectations)
|
||||
3. **"Er modellen deployed til production?"** (hvis ja → prioriter Azure ML Model Monitoring, hvis nei → focus på training data profiling)
|
||||
4. **"Hvor ofte oppdateres source data?"** (real-time → Fabric Eventstream DQ, batch daily → Purview scheduled scans)
|
||||
5. **"Hvilke compliance-krav må dere møte?"** (Forvaltningsloven § 11b, AI Act, GDPR → Purview for audit logs + documentation)
|
||||
6. **"Har dere budget for data quality tooling?"** (limited → start med Azure ML DQ monitoring signal (gratis), gradual expansion til Purview)
|
||||
7. **"Hvem er data owner/steward?"** (sentralisert team → Purview centralized governance, distribuert → Unity Catalog distributed governance model)
|
||||
|
||||
### Anbefalte arkitektur-patterns per scenario
|
||||
|
||||
| Scenario | Pattern | Tools |
|
||||
|----------|---------|-------|
|
||||
| **Offentlig sektor AI-vedtakssystem** | Layered Quality Gates + Compliance Documentation | Fabric (Bronze/Silver constraints) + Purview (Gold scoring + audit logs) + Azure ML Monitoring (production) |
|
||||
| **Real-time fraud detection** | Real-Time Data Quality for Streaming AI | Fabric Eventstream + KQL DB + Azure ML Monitoring |
|
||||
| **Multi-cloud data lake** | Enterprise Governance with Multi-Cloud Sources | Purview Data Quality + Unity Catalog (hvis Databricks) |
|
||||
| **Spark-based ETL pipelines** | Declarative Quality Constraints | Databricks DLT expectations |
|
||||
| **Quick POC (low maturity)** | Training vs. Production Monitoring | Azure ML data quality monitoring signal |
|
||||
|
||||
---
|
||||
|
||||
## Kilder og verifisering
|
||||
|
||||
### Microsoft dokumentasjon
|
||||
|
||||
1. **Microsoft Purview Data Quality Overview**
|
||||
https://learn.microsoft.com/en-us/purview/data-quality-overview
|
||||
*Lesedato: 2026-02* — Comprehensive guide to Purview DQ lifecycle, features, pricing.
|
||||
|
||||
2. **Azure Machine Learning Model Monitoring**
|
||||
https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring
|
||||
*Lesedato: 2026-02* — Monitoring signals, best practices, setup instructions.
|
||||
|
||||
3. **Fabric Materialized Lake Views Data Quality**
|
||||
https://learn.microsoft.com/en-us/fabric/data-engineering/materialized-lake-views/data-quality
|
||||
*Lesedato: 2026-02* — MLV constraints syntax, FAIL/DROP actions, limitations.
|
||||
|
||||
4. **Databricks Data Governance Best Practices**
|
||||
https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices
|
||||
*Lesedato: 2026-02* — Data quality standards, DLT expectations, Unity Catalog lineage.
|
||||
|
||||
5. **Test and Evaluate AI Workloads on Azure**
|
||||
https://learn.microsoft.com/en-us/azure/well-architected/ai/test
|
||||
*Lesedato: 2026-02* — Testing guidance for model training/fine-tuning, data quality checks.
|
||||
|
||||
6. **Purview Data Quality for Fabric Lakehouse**
|
||||
https://learn.microsoft.com/en-us/purview/unified-catalog-data-quality-fabric-lakehouse
|
||||
*Lesedato: 2026-02* — Setup guide for Purview DQ scan on Fabric data.
|
||||
|
||||
7. **Azure ML Configure Training, Validation, Cross-Validation Data**
|
||||
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-cross-validation-data-splits
|
||||
*Lesedato: 2026-02* — Best practices for data splits, validation dataset setup.
|
||||
|
||||
8. **Cloud Adoption Framework: Data Quality**
|
||||
https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/cloud-scale-analytics/govern-data-quality
|
||||
*Lesedato: 2026-02* — Data quality considerations, recommendations for enterprise scale.
|
||||
|
||||
### Code samples (Microsoft Learn)
|
||||
|
||||
- **Azure ML fine-tuning job with validation data**
|
||||
https://learn.microsoft.com/en-us/azure/ai-foundry/how-to/fine-tune-serverless
|
||||
Python SDK sample for creating validation dataset asset.
|
||||
|
||||
- **AutoML training/validation MLTable inputs**
|
||||
https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-auto-train-image-models
|
||||
Python SDK sample for Input objects (training/validation data).
|
||||
|
||||
- **MLflow traces quality analysis**
|
||||
https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/build-eval-dataset
|
||||
Python sample for filtering traces by quality score, correlation analysis.
|
||||
|
||||
### Confidence markers
|
||||
|
||||
- **GA status:** Microsoft Purview Data Quality (GA), Azure ML Model Monitoring (GA), Fabric data quality (GA), Databricks DLT expectations (GA) — alle features er production-ready per 2026-02.
|
||||
- **Pricing estimates:** Medium-High confidence — basert på official pricing pages (learn.microsoft.com), men actual consumption varierer med data volume og rule complexity.
|
||||
- **Virtual Network support:** Purview DQ for Fabric Lakehouse via Private Link er bekreftet i official docs (as of 2026-02), men BigQuery virtual network support er eksplisitt dokumentert som "not yet supported."
|
||||
- **Integration gaps:** Azure ML + Purview native integration finnes ikke (as of 2026-02) — workaround via Azure Functions er architectural recommendation, ikke officially supported pattern.
|
||||
|
||||
### Sist verifisert
|
||||
|
||||
**2026-02-11** — Alle lenker testet, pricing bekreftet mot official docs.
|
||||
Loading…
Add table
Add a link
Reference in a new issue