open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 4 Activity

feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Browse source 
Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

513
skills/ms-ai-governance/references/responsible-ai/data-quality-responsible-ai.md Normal file
View file

@ -0,0 +1,513 @@
# Data Quality for Responsible AI - Ensuring Training Data Integrity
**Last updated:** 2026-02
**Status:** GA
**Category:** Responsible AI & Governance
---
## Introduksjon
Datakvalitet er grunnmuren for ansvarlig AI. Machine learning-modeller lærer fra historiske beslutninger og handlinger fanget i treningsdata, og deres ytelse i produksjon er direkte avhengig av kvaliteten på disse dataene. Dårlig datakvalitet fører til bias, unfairness, feilprediksjoner og tap av tillit.
Denne referansen dekker Microsofts tilnærming til å sikre dataintegritet gjennom hele ML-livssyklusen — fra datainnsamling og preprosessering til vedlikehold og lineage tracking. For organisasjoner i offentlig sektor (spesielt Norge) er dette kritisk for å oppfylle krav til etterrettelighet, åpenhet og rettferdig behandling.
**Kjerneprinsipp:** Trustworthy training data har høyere sannsynlighet for å generere trustworthy outcomes. Data quality er ikke en engangsjobb, men en kontinuerlig prosess som må integreres i MLOps-praksis.
---
## Kjernekomponenter
### 1. Data Sources og Diversitet
**Kilder til treningsdata:**
| Type | Beskrivelse | Kvalitetsrisiko |
|------|-------------|-----------------|
| **Proprietary data** | Organisasjonens egen data | Label bias, underrepresentasjon |
| **Public sources** | Wikipedia, PubMed, offentlige datasett | Variabel kvalitet, mangelfull kurering |
| **User-generated data** | Brukerinteraksjoner, feedback, samarbeid | Støy, malicious inputs, drifting patterns |
**Kvalitetsutfordringer:**
- **Imbalanced datasets** → modeller som favoriserer majoritetsklasser
- **Underrepresentasjon** → dårlig ytelse for minoritetsgrupper
- **Skewed feature distribution** → feilprediksjoner for underrepresenterte segmenter
**Teknikker for balansering:**
- **SMOTE** (Synthetic Minority Oversampling Technique) — genererer syntetiske eksempler for minoritetsklasser
- **Undersampling** — reduserer majoritetsklasser
- **Synthetic data generation** (Azure AI Foundry) — genererer representative datasett
### 2. Exploratory Data Analysis (EDA)
Gjennomfør EDA **tidlig** i feature design for å identifisere:
- Karakteristikker, relasjoner, mønstre
- Kvalitetsproblemer (missing values, outliers, noise)
- Over-/underrepresentasjon
- Statistisk bias
**Plattformstøtte:**
- **Azure Machine Learning Responsible AI dashboard** → Data Analysis-komponent
- Visualiseringer: aggregate plots, scatter plots, cohort-basert analyse
- Filtrer på predicted outcome, dataset features, error groups
### 3. Data Preprocessing
**Fire nøkkelteknikker (Verified fra Microsoft Docs):**
| Teknikk | Formål | Eksempel |
|---------|--------|----------|
| **Quality filtering** | Fjern støy, ufullstendige observasjoner | Eliminer produktanmeldelser som er for korte |
| **Rescoping** | Broadening overly specific fields | Adresse → by/stat i stedet for gate/husnummer |
| **Deduplication** | Fjern redundans | 1000 identiske loggoppføringer → 1 observasjon |
| **Sensitive data handling** | Eliminer persondata hvis ikke kritisk | Anonymiser PII, fjern unødvendige personopplysninger |
**Standardized transformation:**
- Konverter til ML-kompatible formater
- Image → text (OCR for scanned documents)
- Adjust orientations/aspect ratios for modellkompatibilitet
### 4. Data Validation og Guardrails
**Azure Machine Learning AutoML Data Guardrails:**
| Guardrail | Status | Condition |
|-----------|--------|-----------|
| **Class balancing detection** | Alerted/Passed | Detekterer ubalanserte klasser |
| **Memory issues detection** | Done/Passed | Sjekker at horizon/lag/rolling window ikke forårsaker OOM |
| **Frequency detection** | Done/Passed | Verifiserer time-series alignment |
**Data quality expectations (Azure Databricks):**
```python
valid_pages = {
"valid_count": "count > 0",
"valid_current_page": "current_page_id IS NOT NULL AND current_page_title IS NOT NULL"
}
@dp.table
@dp.expect_all_or_drop(valid_pages)
def prepared_data():
# Dropper records som feiler expectations
```
### 5. Feature Stores
Sentralisert repository for features som sikrer:
- Konsistens mellom training og inference
- Feature reuse på tvers av modeller og team
- Versjonering og immutability
- Automated data drift detection
**Implementeringsmønstre:**
- **Centralized** → single source of truth, sterk governance
- **Distributed** → team-autonomi, krever koordinering
- **Hybrid** → common features sentralt, domain-specific features distribuert
### 6. Data Lineage Tracking
Spor dataens vei fra kilde til modelltrening for:
- Explainability og åpenhet
- Debugging og root cause analysis
- Identifisere bias introdusert i preprocessing
- Compliance og auditability
**Plattformintegrasjon:**
- **Azure Machine Learning + Microsoft Purview** → automatisk lineage tracking
- **Version control** (Git, Azure DevOps) → track changes til training datasets
### 7. Decision Integrity og Security
**Threats til training data (Verified fra Microsoft Security whitepaper):**
| Threat | Beskrivelse | Mitigasjon |
|--------|-------------|------------|
| **Malicious data injection** | Angripere introduserer crafted inputs | Data resilience, decision integrity checks |
| **Target leakage** | Modellen "jukser" med data fra fremtiden | Validate features, temporal consistency |
| **Training data tampering** | Modifikasjon av trusted training data | Access controls, immutable datasets |
**Overtraining pitfalls:**
- **Overfitting** → modellen memorerer trening, feiler på test
- **Target leakage** → abnormally høy accuracy (95%+) → sannsynligvis leakage
---
## Arkitekturmønstre
### Pattern 1: Centralized Training Data Pipeline
```
Source Data (Production/External)
↓
Data Collection Store (localized)
↓
Exploratory Data Analysis (EDA)
↓
Preprocessing (quality, rescoping, deduplication, PII removal)
↓
Feature Store (versioned, immutable features)
↓
Training Data (train/validation/test split)
↓
Model Training
↓
Responsible AI Dashboard → Data Analysis
```
**Når bruke:**
- Sterk data governance
- Compliance-krav (GDPR, offentlig sektor)
- Flere team deler samme datasett
### Pattern 2: Segmented Data Pipeline
**Use case:** Separate pipelines for data med distinct security requirements.
```
Geo Region A Data → Pipeline A → Model A
Geo Region B Data → Pipeline B → Model B
↓
(Optional) Federated Training → Combined Model
```
**Krav:**
- Access controls per segment
- Same security rigor på alle segmenter
- Regulatory constraints (data residency)
### Pattern 3: Continuous Data Quality Monitoring
```
Production Data → Real-time Ingestion
↓
Data Quality Checks (expectations, guardrails)
↓
[Pass] → Feature Store → Retraining
[Fail] → Alert → Manual Review
↓
Monitor for Data Drift / Concept Drift
↓
Trigger Retraining (condition-based or scheduled)
```
**Plattform:**
- **Azure Machine Learning Model Monitoring** → data drift, data quality signals
- **Databricks Expectations** → inline quality checks
### Pattern 4: Foundation Model Fine-Tuning Data Pipeline
Mindre volum, høyere kvalitetskrav:
```
High-Quality Domain-Specific Examples
↓
Manual Curation / Expert Review
↓
Small Training Set (100-1000s examples)
↓
Fine-Tune Pre-Trained Model
↓
Validate on Hold-Out Test Set
```
**Eksempel:** Fine-tune GPT-4 for medical documentation → training examples må accurately representere medical terminology og clinical reasoning.
---
## Beslutningsveiledning
### Når bruke sentralisert vs. distribuert feature store?
| Kriterium | Centralized | Distributed |
|-----------|-------------|-------------|
| **Organization size** | Large, standardized | Multiple autonomous teams |
| **Governance maturity** | High | Moderate |
| **Feature overlap** | High (many shared features) | Low (domain-specific) |
| **Compliance** | Strict centralized control | Team-level flexibility |
### Hvor ofte gjøre retraining?
| Trigger | Frequency | Use Case |
|---------|-----------|----------|
| **Scheduled** | Daily/weekly | Routine maintenance, stable data |
| **Trigger-based** | On data drift detection | Dynamic environments, rapid change |
| **Hybrid** | Both | Fail-proof operations (scheduled + triggered) |
### Hvor lenge beholde training data?
| Scenario | Retention Policy | Rationale |
|----------|------------------|-----------|
| **Data unchanged** | Delete after training | Reduce storage costs, minimize risk |
| **Model drift detected** | Retain for comparison | Rebuild/retrain with historical data |
| **Compliance** | Follow RTBF (Right to Be Forgotten) | Remove personal data on request |
| **Disaster recovery** | Secondary pipeline with redundancy | Regenerate model exactly as before |
### Hvordan håndtere imbalanced data?
```
IF minority class < 10% THEN
IF synthetic data acceptable THEN
Apply SMOTE
ELSE
Oversample minority OR Undersample majority
END IF
ELSE IF 10-30% THEN
Use class weights in model training
ELSE
Standard training (sufficient balance)
END IF
```
---
## Integrasjon med Microsoft-stakken
### Azure Machine Learning
| Komponent | Kapabilitet | Data Quality Support |
|-----------|-------------|----------------------|
| **Responsible AI Dashboard** | Data analysis, fairness, error analysis | Visualize distribution, identify bias |
| **AutoML Data Guardrails** | Class balancing, memory, frequency checks | Automated alerts |
| **Model Monitoring** | Data drift, data quality signals | Continuous monitoring |
| **ML Datasets** | Versioned, registered datasets | Lineage tracking |
**Code Sample (Data Quality Signal):**
```python
from azure.ai.ml.entities import DataQualitySignal, DataQualityMetricThreshold
metric_thresholds = DataQualityMetricThreshold(
numerical=DataQualityMetricsNumerical(null_value_rate=0.01),
categorical=DataQualityMetricsCategorical(out_of_bounds_rate=0.02)
)
data_quality_signal = DataQualitySignal(
production_data=production_data,
reference_data=reference_data_training,
features=['feature_A', 'feature_B', 'feature_C'],
metric_thresholds=metric_thresholds,
alert_enabled=True
)
```
### Azure AI Foundry
- **Evaluation tools** → assess data quality before training
- **Synthetic data generation** → generate balanced datasets
- **Content Safety** → filter harmful training data (protected material detection)
### Azure Databricks
**Expectations pattern:**
```python
@dp.table
@dp.expect_all_or_fail({"valid_count": "count > 0"})
def customer_facing_data():
# Pipeline fails if expectation not met
```
### Microsoft Purview
- **Data discovery and classification** → automated tagging
- **Lineage tracking** → full data provenance
- **Compliance policies** → enforce GDPR/data residency
### Azure DevOps
- **Version control** for training datasets
- **CI/CD pipelines** → automated data validation
- **Rollback** → revert to previous dataset version if quality degrades
---
## Offentlig sektor (Norge)
### Særlige krav
| Krav | Implementasjon | Microsoft-støtte |
|------|----------------|------------------|
| **Etterrettelighet** | Full lineage tracking fra kilde til modell | Azure ML + Purview |
| **Åpenhet** | Responsible AI Scorecard (PDF for stakeholders) | Azure ML RAI dashboard |
| **Rettferdig behandling** | Fairness assessment, class balancing | AutoML guardrails, RAI dashboard |
| **Personvern (GDPR)** | PII removal, RTBF compliance | Data preprocessing, anonymization |
| **Data residency** | Segmented pipelines per region | Norway East/West regions |
### Eksempel: NAV (arbeids- og velferdsetaten)
**Scenario:** Prediksjonsmodell for uføretrygd.
**Data quality challenges:**
- Historiske data kan inneholde bias (underrepresentasjon av grupper)
- Personopplysninger må anonymiseres
- Modellen må være transparent for revisorer
**Løsning:**
1. **EDA** → identifiser underrepresentasjon (alder, kjønn, region)
2. **Balancing** → SMOTE for minoritetsgrupper
3. **PII removal** → anonymiser fødselsnummer, adresser
4. **Fairness assessment** → RAI dashboard → verifiser at accuracy er lik på tvers av demografiske grupper
5. **Scorecard** → generer PDF for politiske stakeholders og revisorer
6. **Lineage** → Purview → dokumenter at all data er lovlig innsamlet
---
## Kostnad og lisensiering
### Kostnadskomponenter
| Komponent | Kostnadsfaktor | Estimat (NOK/måned) |
|-----------|----------------|---------------------|
| **Data storage (localized)** | Azure Blob/ADLS Gen2 | 500-5000 (avhenger av volum) |
| **Compute (EDA, preprocessing)** | Databricks/Synapse Spark | 2000-20000 (avhenger av scale) |
| **Feature store** | Azure ML Feature Store | Inkludert i Azure ML |
| **Purview (lineage)** | Data governance scanning | 3000-10000 (avhenger av data sources) |
| **AutoML (guardrails)** | Compute for training | 1000-10000 per experiment |
**Optimeringstips:**
- Bruk serverless Spark (pay-per-use) for EDA
- Delete stale training data
- Share feature stores på tvers av team
### Lisenskrav
| Kapabilitet | Lisens | Kommentar |
|-------------|--------|-----------|
| **Azure Machine Learning** | Azure subscription | Pay-as-you-go compute |
| **Responsible AI Dashboard** | Inkludert i Azure ML | Ingen ekstra kostnad |
| **Microsoft Purview** | Separate license | Data governance add-on |
| **Databricks Expectations** | Databricks license | Premium/Enterprise tier |
| **Azure AI Foundry** | Azure subscription | Separate compute charges |
---
## For arkitekten (Cosmo)
### Quick Decision Tree
```
START: Kunde trenger AI-modell
1. Har de eksisterende training data?
NO → Anbefal data collection strategy (proprietary vs. public vs. synthetic)
YES → Fortsett til 2
2. Er datasettet balansert?
NO → Anbefal SMOTE/oversampling/synthetic data
YES → Fortsett til 3
3. Har de gjort EDA?
NO → Anbefal Azure ML RAI Dashboard → Data Analysis
YES → Fortsett til 4
4. Er det PII i datasettet?
YES → KRITISK: Anbefal preprocessing (anonymization/removal)
NO → Fortsett til 5
5. Trenger de compliance/auditability?
YES → Anbefal Purview + RAI Scorecard
NO → Fortsett til 6
6. Har de data drift i produksjon?
YES → Anbefal Model Monitoring med data quality signals
NO → Basic training pipeline OK
7. Er dette foundation model fine-tuning?
YES → Anbefal small, high-quality curated dataset
NO → Standard training pipeline
```
### Red Flags (Varsle umiddelbart)
| Symptom | Problem | Løsning |
|---------|---------|---------|
| "Vi har 95%+ accuracy på test" | Sannsynlig target leakage | Validate features, temporal consistency |
| "Brukerdata går rett i training" | Malicious injection risk | Data validation, guardrails |
| "Vi slettet dårlige eksempler" | Selection bias | Behold representative samples |
| "Vi trener på all historisk data" | Overfitting, stale data | Implement temporal windowing |
| "Vi har ikke test set" | Kan ikke validere generalisering | 80/10/10 split (train/val/test) |
### Cosmo's Talking Points
**Når kunden sier:** "Vi har mye data, så kvalitet er ikke så viktig."
**Svar:** "Det er motsatt — mer data forsterker bias hvis kvaliteten er dårlig. En modell trent på 1M dårlige eksempler er verre enn 10K gode. La oss starte med EDA for å forstå hva dere faktisk har."
**Når kunden sier:** "Vi kan ikke slette persondata, det er viktig for modellen."
**Svar:** "Det er to spørsmål: 1) Er det *kritisk* for prediktiv kraft, eller kan vi anonymisere? 2) Hvis kritisk, må dere ha GDPR-compliance (RTBF-policy, consent management). Jeg anbefaler Purview for å tracke dette."
**Når kunden sier:** "Modellen fungerer dårlig på noen grupper."
**Svar:** "Det er sannsynligvis underrepresentasjon i training data. La oss kjøre Fairness Assessment i RAI Dashboard og se om vi trenger oversampling eller mer data."
### Verktøyvalg per scenario
| Scenario | Anbefalt verktøy | Alternativ |
|----------|------------------|------------|
| **EDA** | Azure ML Notebooks + RAI Dashboard | Databricks Notebooks |
| **Data validation** | AutoML Guardrails | Databricks Expectations |
| **Lineage tracking** | Purview | Manual documentation (ikke anbefalt) |
| **Class balancing** | SMOTE (Azure ML) | Synthetic data (AI Foundry) |
| **PII removal** | Custom preprocessing scripts | Azure Cognitive Services (PII detection) |
| **Monitoring** | Azure ML Model Monitoring | Custom dashboards (Grafana) |
---
## Kilder og verifisering
**Verified (fra Microsoft Learn MCP):**
1. **Design training data for AI workloads on Azure**
https://learn.microsoft.com/en-us/azure/well-architected/ai/training-data-design
*Confidence: Verified* → Covering data sources, preprocessing, feature stores, lineage, maintenance
2. **Understand your datasets (Responsible AI)**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-analysis?view=azureml-api-2
*Confidence: Verified* → Data analysis component, cohorts, over/underrepresentation
3. **What is Responsible AI?**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai?view=azureml-api-2
*Confidence: Verified* → Six principles, RAI dashboard components, transparency
4. **Responsible AI in Azure workloads**
https://learn.microsoft.com/en-us/azure/well-architected/ai/responsible-ai
*Confidence: Verified* → User data handling, RTBF, explainability, privacy
5. **Prevent overfitting and imbalanced data with Automated ML**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-manage-ml-pitfalls?view=azureml-api-2
*Confidence: Verified* → Overfitting, target leakage, class imbalance, SMOTE
6. **Securing AI and Machine Learning at Microsoft**
https://learn.microsoft.com/en-us/security/engineering/securing-artificial-intelligence-machine-learning
*Confidence: Verified* → Malicious data injection, decision integrity, training data security
7. **Govern Azure platform services for AI**
https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/platform/governance
*Confidence: Verified* → Data discovery, classification, Purview, version control
8. **Model performance and fairness**
https://learn.microsoft.com/en-us/azure/machine-learning/concept-fairness-ml?view=azureml-api-2
*Confidence: Verified* → Parity constraints, mitigation algorithms (Fairlearn)
9. **Data featurization in AutoML**
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-auto-features?view=azureml-api-1
*Confidence: Verified* → Data guardrails (class balancing, memory, frequency detection)
10. **Azure Databricks Data Expectations**
https://learn.microsoft.com/en-us/azure/databricks/ldp/expectations
*Confidence: Verified* → expect_all, expect_all_or_drop, expect_all_or_fail patterns
**Baseline (modellkunnskap):**
- Feature store patterns (centralized/distributed/hybrid)
- Decision trees for trigger-based vs. scheduled retraining
- Norwegian public sector requirements (etterrettelighet, GDPR)
**Code samples:**
- Azure ML Data Quality Signal (Python SDK) → Verified
- Databricks Expectations decorator pattern → Verified
---
**Sist oppdatert:** 2026-02
**Neste review:** 2026-08 (eller ved større Microsoft AI-oppdateringer)