open/ms-ai-architect
1
0
Fork 0
Code Issues Pull requests Releases Activity

docs(ms-ai-architect): KB-refresh LOW 7/52 — data-quality-ai-frameworks: MLV-syntaks ON MISMATCH (ikke WITH ACTION), funksjoner/UDF stottes i constraints, DLT to Lakeflow SDP rebrand

Browse source
This commit is contained in:
Kjell Tore Guttormsen 2026-06-24 11:59:27 +02:00
commit 2bb12de705
1 changed files with 16 additions and 13 deletions
Download patch file Download diff file Expand all files Collapse all files

29
skills/ms-ai-engineering/references/data-engineering/data-quality-ai-frameworks.md
View file

@ -1,6 +1,6 @@
# Data Quality Frameworks for AI
**Last updated:** 2026-02
**Last updated:** 2026-06-24
**Status:** GA (Microsoft Purview Data Quality, Azure ML Model Monitoring, Fabric data quality)
**Category:** Data Engineering for AI
@ -101,9 +101,8 @@ Microsoft-stacken tilbyr fire hovedspor for data quality management i AI-konteks
```sql
-- Example: exclude null customerName rows
CREATE MATERIALIZED VIEW sales_clean
CONSTRAINT cust_blank CHECK customerName IS NOT NULL
WITH ACTION DROP -- or FAIL
CREATE MATERIALIZED LAKE VIEW sales_clean
(CONSTRAINT cust_blank CHECK (customerName IS NOT NULL) ON MISMATCH DROP) -- or FAIL (default)
AS SELECT * FROM raw_sales;
```
@ -113,7 +112,7 @@ AS SELECT * FROM raw_sales;
**Limitations:**
- Constraints kan ikke oppdateres etter MLV creation (må recreate)
- Functions og pattern search (LIKE, regex) i constraints er ikke støttet
- Innebygde Spark/SQL-funksjoner (UPPER, TRIM, COALESCE, SUBSTRING …), UDF-er (`spark.udf.register()`), Pandas-UDF-er og Fabric User Data Functions støttes i constraints; ren regex/LIKE er ikke et eget constraint-konstrukt — bruk funksjoner/UDF-er i stedet
- Known issue: MLV med FAIL action kan gi "delta table not found" error (workaround: unngå FAIL, bruk DROP)
**Data quality for Fabric Lakehouse (Purview integration):**
@ -123,26 +122,30 @@ AS SELECT * FROM raw_sales;
4. Associate Lakehouse tables med data product i Purview Unified Catalog
5. Profile + create rules + run data quality scan via Purview
### 6. Azure Databricks Expectations (Delta Live Tables)
### 6. Azure Databricks Expectations (Lakeflow Spark Declarative Pipelines, tidl. Delta Live Tables/DLT)
> Delta Live Tables (DLT) er rebrandet til **Lakeflow Spark Declarative Pipelines (SDP)**. Ingen migrasjon kreves — `import dlt`/`@dlt.*` fungerer fortsatt, men anbefalt syntaks er nå `from pyspark import pipelines as dp` + `@dp.*` (classic SKU-er og event-log-skjema beholder fortsatt `DLT`-navnet).
**Expectations syntax:**
```python
@dlt.expect("valid_timestamp", "timestamp IS NOT NULL")
@dlt.expect_or_drop("valid_amount", "amount > 0")
@dlt.expect_or_fail("critical_id", "id IS NOT NULL")
from pyspark import pipelines as dp # tidl. import dlt
@dp.expect("valid_timestamp", "timestamp IS NOT NULL")
@dp.expect_or_drop("valid_amount", "amount > 0")
@dp.expect_or_fail("critical_id", "id IS NOT NULL")
def clean_transactions():
return spark.read.table("raw_transactions")
```
**Actions:**
- `@dlt.expect` — track violations, allow records to pass
- `@dlt.expect_or_drop` — drop violating records
- `@dlt.expect_or_fail` — fail pipeline ved violations
- `@dp.expect` — track violations, allow records to pass
- `@dp.expect_or_drop` — drop violating records
- `@dp.expect_or_fail` — fail pipeline ved violations
**Benefits:**
- Catch data quality issues ved ingestion før de påvirker downstream data products
- Real-time quality metrics i DLT pipeline observability UI
- Real-time quality metrics i Lakeflow SDP pipeline observability UI (event log)
- Automatically generated data quality dashboards
---