feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,765 @@
|
|||
# Model Monitoring and Drift Detection - Ongoing Compliance
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Status:** GA
|
||||
**Category:** Responsible AI & Governance
|
||||
|
||||
---
|
||||
|
||||
## Introduksjon
|
||||
|
||||
Model monitoring er siste steg i machine learning-livssyklusen og sporer modellytelse i produksjon fra både datavitenskapelige og operasjonelle perspektiver. I motsetning til tradisjonelle programvaresystemer, avhenger ML-systemers oppførsel ikke bare av regler i kode, men også av data. Endringer i datadistribusjon, training-serving skew, datakvalitetsproblemer og miljøendringer kan alle føre til at modeller blir utdaterte.
|
||||
|
||||
Azure Machine Learning model monitoring detekterer disse problemene kontinuerlig ved å sammenligne produksjonsdata med referansedata (training data eller nylige produksjonsdata) og varsle når metriske terskler overskrides.
|
||||
|
||||
**Verified** (MCP-research jan 2026): Azure Machine Learning model monitoring er GA (Generally Available) for tabular data med support for både online endpoints og batch/external deployments.
|
||||
|
||||
### Kjernetyper av drift
|
||||
|
||||
| Drifttype | Beskrivelse | Eksempel |
|
||||
|-----------|-------------|----------|
|
||||
| **Data drift** | Endringer i input-data distribusjon som gjør modellen utdatert | Demografiske endringer etter redistricting påvirker stemmeprediksjon |
|
||||
| **Concept drift** | Eksterne forhold endrer seg slik at modellens prediksjoner ikke lenger reflekterer virkeligheten | Konkurrent lanserer nytt produkt → salgsmodell blir irrelevant |
|
||||
| **Prediction drift** | Endringer i modellens output-distribusjon sammenlignet med validation/test data | Fraud detection-modell predikerer plutselig høyere fraud rate |
|
||||
| **Data quality drift** | Degradering av dataintegritet (null values, type errors, out-of-bounds) | Sensor begynner alltid å rapportere 0 (broken sensor) |
|
||||
| **Feature attribution drift** | Endringer i feature importance i produksjon vs. training | Temperature blir mindre viktig for prediction over tid |
|
||||
|
||||
---
|
||||
|
||||
## Kjernekomponenter
|
||||
|
||||
### Built-in Monitoring Signals (Azure ML)
|
||||
|
||||
**Verified** (microsoft-learn): Azure Machine Learning tilbyr følgende innebygde signaler for tabular data:
|
||||
|
||||
| Signal | Metrics | Production Data | Reference Data | ML Task Support |
|
||||
|--------|---------|-----------------|----------------|-----------------|
|
||||
| **Data drift** | Jensen-Shannon Distance, Population Stability Index, Normalized Wasserstein Distance, Two-Sample Kolmogorov-Smirnov Test, Pearson's Chi-Squared Test | Model inputs | Training data or recent production | Classification, Regression (tabular) |
|
||||
| **Prediction drift** | Jensen-Shannon Distance, Population Stability Index, Normalized Wasserstein Distance, Chebyshev Distance, Two-Sample Kolmogorov-Smirnov Test, Pearson's Chi-Squared Test | Model outputs | Validation data or recent production | Classification, Regression (tabular) |
|
||||
| **Data quality** | Null value rate, Data type error rate, Out-of-bounds rate | Model inputs | Training data or recent production | Classification, Regression (tabular) |
|
||||
| **Feature attribution drift** (preview) | Normalized discounted cumulative gain (NDCG) | Model inputs + outputs | Training data (required) | Classification, Regression (tabular) |
|
||||
| **Model performance** (preview) | Accuracy, Precision, Recall (classification); MAE, MSE, RMSE (regression) | Model outputs | Ground truth data (required) | Classification, Regression (tabular) |
|
||||
| **Generation safety/quality** (preview) | Groundedness, Relevance, Fluency, Similarity, Coherence | Prompt, completion, context | Annotation template | Generative AI (Q&A) |
|
||||
|
||||
### Data Quality Metrics (Detaljer)
|
||||
|
||||
**Verified** (microsoft-learn): Azure ML støtter opptil 0.00001 precision for data quality calculations:
|
||||
|
||||
1. **Null value rate**: Andel null-verdier per feature (støttes for alle datatyper)
|
||||
2. **Data type error rate**: Andel verdier som ikke matcher inferred data type fra reference data
|
||||
- Støttede PySpark typer: `ShortType`, `BooleanType`, `BinaryType`, `DoubleType`, `TimestampType`, `StringType`, `IntegerType`, `FloatType`, `ByteType`, `LongType`, `DateType`
|
||||
3. **Out-of-bounds rate**: Andel verdier utenfor acceptable range/set fra reference data
|
||||
- Numerical features: intervall [min, max] fra reference dataset
|
||||
- Categorical features: sett av alle verdier i reference dataset
|
||||
- Støttede typer: `StringType`, `IntegerType`, `DoubleType`, `ByteType`, `LongType`, `FloatType`
|
||||
|
||||
### Lookback Windows og Data Windowing
|
||||
|
||||
**Verified** (microsoft-learn): Azure ML bruker ISO 8601 format for time windows:
|
||||
|
||||
```yaml
|
||||
# Eksempel: Monitor kjører 31. januar kl 15:15 UTC
|
||||
production_data:
|
||||
data_window:
|
||||
lookback_window_size: P7D # 7 dager produksjonsdata
|
||||
lookback_window_offset: P0D # Ingen offset (data frem til run time)
|
||||
# Resultat: 24. jan 15:15 → 31. jan 15:15
|
||||
|
||||
reference_data:
|
||||
data_window:
|
||||
lookback_window_size: P24D # 24 dager referansedata
|
||||
lookback_window_offset: P7D # 7 dagers offset (ingen overlap)
|
||||
# Resultat: 1. jan 15:15 → 24. jan 15:15
|
||||
```
|
||||
|
||||
**Best practice** (Verified): Reference data offset bør være ≥ (production lookback size + production offset) for å unngå overlap.
|
||||
|
||||
---
|
||||
|
||||
## Arkitekturmønstre
|
||||
|
||||
### Pattern 1: Out-of-Box Monitoring (Online Endpoints)
|
||||
|
||||
**Verified** (microsoft-learn): For modeller deployed til Azure ML online endpoints med data collection enabled:
|
||||
|
||||
```python
|
||||
# Serverless Spark compute (Required for all monitoring)
|
||||
spark_compute = ServerlessSparkCompute(
|
||||
instance_type="standard_e4s_v3", # Supported: e4s, e8s, e16s, e32s, e64s (v3)
|
||||
runtime_version="3.3"
|
||||
)
|
||||
|
||||
# Minimal konfigurasjon - automatisk data drift, prediction drift, data quality
|
||||
monitoring_target = MonitoringTarget(
|
||||
ml_task="classification", # eller "regression"
|
||||
endpoint_deployment_id="azureml:credit-default:main"
|
||||
)
|
||||
|
||||
monitor_definition = MonitorDefinition(
|
||||
compute=spark_compute,
|
||||
monitoring_target=monitoring_target,
|
||||
alert_notification=AlertNotification(emails=['admin@example.com'])
|
||||
)
|
||||
|
||||
# Schedule (daglig kl 03:15)
|
||||
recurrence_trigger = RecurrenceTrigger(
|
||||
frequency="day",
|
||||
interval=1,
|
||||
schedule=RecurrencePattern(hours=3, minutes=15)
|
||||
)
|
||||
```
|
||||
|
||||
**Hva skjer automatisk:**
|
||||
- Azure ML detekterer production inference data asset fra online deployment
|
||||
- Reference data = recent past production data
|
||||
- Data drift, prediction drift, data quality signals med smart defaults
|
||||
- Email alerts ved threshold breach
|
||||
|
||||
### Pattern 2: Advanced Monitoring (Training Data som Baseline)
|
||||
|
||||
**Verified** (microsoft-learn): For å bruke training data som comparison baseline og aktivere feature importance:
|
||||
|
||||
```python
|
||||
# Production data (automatisk fra online endpoint eller manuelt registrert)
|
||||
production_data = ProductionData(
|
||||
input_data=Input(type="uri_folder", path="azureml:prod_data:1"),
|
||||
data_context=MonitorDatasetContext.MODEL_INPUTS,
|
||||
data_window=BaselineDataRange(
|
||||
lookback_window_size="P1D",
|
||||
lookback_window_offset="P0D"
|
||||
)
|
||||
)
|
||||
|
||||
# Reference data (training data)
|
||||
reference_data = ReferenceData(
|
||||
input_data=Input(type="mltable", path="azureml:training_data:1"),
|
||||
data_context=MonitorDatasetContext.TRAINING,
|
||||
data_column_names={"target_column": "is_fraud"} # Required for feature importance
|
||||
)
|
||||
|
||||
# Data drift med feature importance (top 10)
|
||||
data_drift = DataDriftSignal(
|
||||
production_data=production_data,
|
||||
reference_data=reference_data,
|
||||
features=MonitorFeatureFilter(top_n_feature_importance=10),
|
||||
metric_thresholds=DataDriftMetricThreshold(
|
||||
numerical=NumericalDriftMetrics(jensen_shannon_distance=0.01),
|
||||
categorical=CategoricalDriftMetrics(pearsons_chi_squared_test=0.02)
|
||||
),
|
||||
alert_enabled=True
|
||||
)
|
||||
|
||||
# Feature attribution drift (krever både input og output data)
|
||||
feature_attribution = FeatureAttributionDriftSignal(
|
||||
reference_data=reference_data, # Training data (required)
|
||||
metric_thresholds=FeatureAttributionDriftMetricThreshold(
|
||||
normalized_discounted_cumulative_gain=0.9
|
||||
),
|
||||
alert_enabled=True
|
||||
)
|
||||
```
|
||||
|
||||
**Viktig** (Verified): For feature attribution drift må Azure ML online endpoint samle både `model_inputs` og `model_outputs`. Systemet joiner automatisk via `correlationid`.
|
||||
|
||||
### Pattern 3: Model Performance Monitoring (Ground Truth)
|
||||
|
||||
**Verified** (microsoft-learn): For objective performance tracking med ground truth data:
|
||||
|
||||
**Prerequisites:**
|
||||
- Output data (predictions) med unique ID per rad
|
||||
- Ground truth data (actuals) med samme unique ID
|
||||
- Matching IDs brukes til join før metric computation
|
||||
|
||||
```python
|
||||
# Production output data
|
||||
production_output = ProductionData(
|
||||
input_data=Input(type="uri_folder", path="azureml:model_outputs:1"),
|
||||
data_column_names={
|
||||
"target_column": "is_fraud", # Prediction column
|
||||
"join_column": "correlation_id" # Unique ID for join
|
||||
},
|
||||
data_window=BaselineDataRange(
|
||||
lookback_window_offset="P0D",
|
||||
lookback_window_size="P10D"
|
||||
)
|
||||
)
|
||||
|
||||
# Ground truth data
|
||||
reference_ground_truth = ReferenceData(
|
||||
input_data=Input(type="mltable", path="azureml:ground_truth:1"),
|
||||
data_column_names={
|
||||
"target_column": "actual_fraud", # Actual column
|
||||
"join_column": "correlation_id" # Same unique ID
|
||||
},
|
||||
data_context=MonitorDatasetContext.GROUND_TRUTH_DATA
|
||||
)
|
||||
|
||||
# Model performance signal
|
||||
model_performance = ModelPerformanceSignal(
|
||||
production_data=production_output,
|
||||
reference_data=reference_ground_truth,
|
||||
metric_thresholds=ModelPerformanceMetricThreshold(
|
||||
classification=ModelPerformanceClassificationThresholds(
|
||||
accuracy=0.50,
|
||||
precision=0.50,
|
||||
recall=0.50
|
||||
)
|
||||
),
|
||||
alert_enabled=True
|
||||
)
|
||||
```
|
||||
|
||||
**Correlation ID best practice** (Verified):
|
||||
- Hvis du bruker Azure ML data collector uten egen ID → systemet genererer `correlationid`
|
||||
- Data collector batcher requests → samme `correlationid` for alle rader i batch
|
||||
- Systemet bruker indexing: `correlationid_0`, `correlationid_1`, osv.
|
||||
- **Anbefaling**: Logg egen unique ID i separat kolonne for å unngå indexing-kompleksitet
|
||||
|
||||
### Pattern 4: Custom Signals (Egendefinerte Metrics)
|
||||
|
||||
**Verified** (microsoft-learn): For metrics som ikke er innebygde:
|
||||
|
||||
**Component Input Signature:**
|
||||
```yaml
|
||||
inputs:
|
||||
production_data:
|
||||
type: mltable
|
||||
std_deviation_threshold: # Egen metric threshold
|
||||
type: string
|
||||
default: "2"
|
||||
```
|
||||
|
||||
**Component Output Signature:**
|
||||
```yaml
|
||||
outputs:
|
||||
signal_metrics:
|
||||
type: mltable
|
||||
# Schema: group, metric_name, metric_value, threshold_value
|
||||
```
|
||||
|
||||
**Output format (eksempel):**
|
||||
| group | metric_value | metric_name | threshold_value |
|
||||
|-------|--------------|-------------|-----------------|
|
||||
| TRANSACTIONAMOUNT | 44896.082 | std_deviation | 2 |
|
||||
| LOCALHOUR | 3.983 | std_deviation | 2 |
|
||||
|
||||
**Registrer component:**
|
||||
```bash
|
||||
az ml component create --file custom_signal.yaml
|
||||
```
|
||||
|
||||
**Bruk i monitor:**
|
||||
```yaml
|
||||
monitoring_signals:
|
||||
customSignal:
|
||||
type: custom
|
||||
component_id: azureml:my_custom_signal:1.0.0
|
||||
input_data:
|
||||
production_data:
|
||||
input_data:
|
||||
type: uri_folder
|
||||
path: azureml:production_data:1
|
||||
data_window:
|
||||
lookback_window_size: P30D
|
||||
lookback_window_offset: P7D
|
||||
metric_thresholds:
|
||||
- metric_name: std_deviation
|
||||
threshold: 2
|
||||
```
|
||||
|
||||
### Pattern 5: External/Batch Deployments (Custom Preprocessing)
|
||||
|
||||
**Verified** (microsoft-learn): For modeller deployed utenfor Azure ML eller til batch endpoints:
|
||||
|
||||
**Preprocessing Component Requirements:**
|
||||
|
||||
| Input/Output | Name | Type | Description | Example |
|
||||
|--------------|------|------|-------------|---------|
|
||||
| input | `data_window_start` | literal, string | ISO8601 start time | 2023-05-01T04:31:57.012Z |
|
||||
| input | `data_window_end` | literal, string | ISO8601 end time | 2023-05-01T04:31:57.012Z |
|
||||
| input | `input_data` | uri_folder | Production inference data asset | azureml:prod_data:1 |
|
||||
| output | `preprocessed_data` | mltable | Tabular data matching reference schema | - |
|
||||
|
||||
**Eksempel preprocessing component:**
|
||||
```python
|
||||
# custom_preprocessing/run.py
|
||||
import argparse
|
||||
from datetime import datetime
|
||||
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--data_window_start", type=str)
|
||||
parser.add_argument("--data_window_end", type=str)
|
||||
parser.add_argument("--input_data", type=str)
|
||||
parser.add_argument("--preprocessed_data", type=str)
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Filter data basert på time window
|
||||
start = datetime.fromisoformat(args.data_window_start)
|
||||
end = datetime.fromisoformat(args.data_window_end)
|
||||
|
||||
# Process input_data → output mltable til preprocessed_data path
|
||||
# ... din logikk her ...
|
||||
```
|
||||
|
||||
**Bruk i monitor:**
|
||||
```yaml
|
||||
monitoring_signals:
|
||||
advanced_data_drift:
|
||||
type: data_drift
|
||||
production_data:
|
||||
input_data:
|
||||
path: azureml:my_production_data:1
|
||||
type: uri_folder
|
||||
data_context: model_inputs
|
||||
pre_processing_component: azureml:custom_preprocessor:1.0.0 # Din component
|
||||
reference_data:
|
||||
input_data:
|
||||
path: azureml:training_data:1
|
||||
type: mltable
|
||||
data_context: training
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Beslutningsveiledning
|
||||
|
||||
### Når bruke hvilken monitoring signal?
|
||||
|
||||
| Scenario | Anbefalt Signal | Reference Data | Rationale |
|
||||
|----------|-----------------|----------------|-----------|
|
||||
| Nylig deployed model, bekymret for input data endringer | **Data drift** | Training data | Tidlig varsel om modellytelse degradering |
|
||||
| Modell i produksjon, output distribusjoner endrer seg | **Prediction drift** | Validation/test data | Detekterer når modellen predikerer annerledes enn forventet |
|
||||
| Datakvalitet problemer (missing values, type errors) | **Data quality** | Training data eller recent production | Fanger opp upstream data pipeline issues |
|
||||
| Vil forstå hvilke features som drifter mest | **Feature attribution drift** | Training data (required) | Identifiserer features med endret importance |
|
||||
| Har tilgang til ground truth data | **Model performance** | Ground truth data (required) | Objektiv measure av actual performance |
|
||||
| Spesifikke metrics som ikke er innebygde | **Custom signal** | Valgfritt | Full kontroll over metric definitions |
|
||||
|
||||
### Monitoring Frequency Guidance
|
||||
|
||||
**Verified** (microsoft-learn best practices):
|
||||
|
||||
| Production Traffic | Data Accumulation | Anbefalt Frequency | Rationale |
|
||||
|--------------------|-------------------|-------------------|-----------|
|
||||
| Høy (daglig) | Sufficient daily data | **Daily** (`frequency: day`, `interval: 1`) | Rask deteksjon av issues |
|
||||
| Medium (ukentlig) | Sufficient weekly data | **Weekly** (`frequency: week`, `interval: 1`) | Balanse mellom cost og coverage |
|
||||
| Lav (månedlig) | Sufficient monthly data | **Monthly** (`frequency: month`, `interval: 1`) | Unngå noise fra små datasets |
|
||||
|
||||
**Best practice** (Verified): Monitor frekvens bør matche production data vekst over tid. For modeller med store feature sets, vurder å monitorere subset av features for å redusere compute cost og noise.
|
||||
|
||||
### Threshold Setting Strategy
|
||||
|
||||
**Baseline** (model knowledge): Riktige terskler avhenger av business context og modellens kritikalitet:
|
||||
|
||||
| Kritikalitet | Threshold Strategy | Eksempel |
|
||||
|--------------|-------------------|----------|
|
||||
| **Høy** (fraud, medical) | Konservative terskler (lavere) | Data drift JS distance: 0.01 |
|
||||
| **Medium** (recommendation) | Moderate terskler | Data drift JS distance: 0.05 |
|
||||
| **Lav** (exploratory) | Liberal terskler (høyere) | Data drift JS distance: 0.10 |
|
||||
|
||||
**Anbefaling** (Verified from docs): Arbeid med data scientists som kjenner modellen for å sette riktige terskler og unngå alert fatigue.
|
||||
|
||||
---
|
||||
|
||||
## Integrasjon med Microsoft-stakken
|
||||
|
||||
### Azure Event Grid Integration
|
||||
|
||||
**Verified** (microsoft-learn): Koble model monitoring til event-driven workflows:
|
||||
|
||||
**Setup Event Subscription:**
|
||||
1. Opprett Event Grid system topic (hvis ikke eksisterer)
|
||||
2. Opprett event subscription i Azure ML workspace
|
||||
3. Velg event type: **Run status changed** (IKKE "Dataset drift detected" - det er v1)
|
||||
4. Legg til advanced filter:
|
||||
- **Key**: `data.RunTags.azureml_modelmonitor_threshold_breached`
|
||||
- **Operator**: String contains
|
||||
- **Value**: `has failed due to one or more features violating metric thresholds`
|
||||
5. (Optional) Filter på specific monitor:
|
||||
- **Value**: `<monitor-name>_<signal-description>` (f.eks. `credit_fraud_monitor_data_drift`)
|
||||
|
||||
**Event Handlers:**
|
||||
- **Azure Event Hubs**: Stream events for processing
|
||||
- **Azure Functions**: Trigger serverless retraining pipeline
|
||||
- **Azure Logic Apps**: Orkestrer kompleks retraining workflow
|
||||
|
||||
**Eksempel workflow:**
|
||||
```
|
||||
Drift detected → Event Grid → Azure Function →
|
||||
→ Trigger Azure ML pipeline (retraining) →
|
||||
→ Deploy new model version →
|
||||
→ Update monitoring til ny versjon
|
||||
```
|
||||
|
||||
### Azure Monitor og Application Insights
|
||||
|
||||
**Verified** (microsoft-learn):
|
||||
- Monitoring metrics sendes til Azure Blob Storage (JSON format)
|
||||
- Application Insights for custom alerting på alle metrics
|
||||
- Azure Monitor Metrics for performance visualization
|
||||
|
||||
### Compute og Resource Management
|
||||
|
||||
**Verified** (microsoft-learn):
|
||||
|
||||
| Component | Resource Type | Supported Sizes |
|
||||
|-----------|---------------|-----------------|
|
||||
| Monitoring jobs | Serverless Spark compute | standard_e4s_v3, e8s_v3, e16s_v3, e32s_v3, e64s_v3 |
|
||||
| Data storage | Azure Blob Storage | Auto-managed av Azure ML |
|
||||
| Metrics storage | Azure Monitor time-series DB | Auto-managed |
|
||||
|
||||
**Begrensninger** (Verified):
|
||||
- Støtter IKKE `AllowOnlyApprovedOutbound` managed VNet isolation
|
||||
- Avhenger av Spark → unngå `MLTable` for komplekse operasjoner (bruk Spark API direkte)
|
||||
- Kun basic `MLTable` har garantert support
|
||||
|
||||
### Authentication Options
|
||||
|
||||
**Verified** (microsoft-learn):
|
||||
|
||||
| Method | Setup | Use Case |
|
||||
|--------|-------|----------|
|
||||
| **Credential-based** | Legg til credentials i datastore | Legacy systems |
|
||||
| **Credential-less (UAMI)** | 1. Opprett User-Assigned Managed Identity<br>2. Attach til workspace<br>3. Grant permissions til datastore<br>4. Set `systemDatastoresAuthMode='identity'` | Modern, sikker (anbefalt) |
|
||||
|
||||
---
|
||||
|
||||
## Offentlig sektor (Norge)
|
||||
|
||||
### Compliance og Regulatoriske Krav
|
||||
|
||||
**Baseline** (AI Act, offentlig sektor best practices):
|
||||
|
||||
| Krav | Hvordan Model Monitoring Hjelper | Azure ML Capability |
|
||||
|------|-----------------------------------|---------------------|
|
||||
| **Kontinuerlig overvåking** (AI Act Art. 61) | Automatisk scheduled monitoring jobs | RecurrenceTrigger (daily/weekly/monthly) |
|
||||
| **Dokumentasjon av ytelse** | Metrics logges automatisk til Azure Monitor | Automatic metrics storage + JSON export |
|
||||
| **Varsling ved avvik** | Email alerts ved threshold breach | AlertNotification + Event Grid |
|
||||
| **Audit trail** | Full history av monitoring runs | Azure ML experiment tracking |
|
||||
| **Data quality krav** | Null value rate, type errors, out-of-bounds | Data quality signal (built-in) |
|
||||
| **Ground truth validation** | Sammenligning mot faktiske verdier | Model performance signal |
|
||||
|
||||
### Personvern og GDPR
|
||||
|
||||
**Baseline** (GDPR compliance):
|
||||
|
||||
| Concern | Mitigering | Azure ML Feature |
|
||||
|---------|------------|------------------|
|
||||
| **Logging av persondata** | Bruk pseudonymiserte IDs (correlation_id) | Data collector med custom ID column |
|
||||
| **Data retention** | Slett gamle monitoring data assets | Automated data lifecycle policies i Azure Blob Storage |
|
||||
| **Access control** | RBAC til monitoring dashboards | Azure ML workspace RBAC |
|
||||
| **Data minimization** | Monitor kun nødvendige features | `features` parameter (subset eller top N) |
|
||||
|
||||
### Sektorspesifikke Anbefalinger
|
||||
|
||||
**Baseline** (offentlig sektor best practices):
|
||||
|
||||
| Sektor | Monitoring Focus | Anbefalt Frekvens | Kritiske Signals |
|
||||
|--------|------------------|-------------------|------------------|
|
||||
| **Helse** | Patient safety, data quality | Daglig | Data quality, Model performance (ground truth fra EHR) |
|
||||
| **NAV** | Fairness, ytelsesmonitorering | Ukentlig | Data drift, Feature attribution drift (sjekk protected attributes) |
|
||||
| **Politi/Justis** | Bias detection, transparency | Ukentlig | Feature attribution drift, Custom fairness metrics |
|
||||
| **Utdanning** | Performance equity | Månedlig | Data drift, Prediction drift |
|
||||
| **Samferdsel** | Safety-critical predictions | Daglig | Model performance, Data quality |
|
||||
|
||||
**Eksempel (NAV søknadsbehandling):**
|
||||
```python
|
||||
# Monitor for bias i protected attributes
|
||||
fairness_signal = CustomSignal(
|
||||
component_id="azureml:fairness_metrics:1.0.0",
|
||||
input_data=production_data,
|
||||
metric_thresholds=[
|
||||
{"metric_name": "demographic_parity_difference", "threshold": 0.05},
|
||||
{"metric_name": "equalized_odds_difference", "threshold": 0.05}
|
||||
]
|
||||
)
|
||||
|
||||
# Monitor data quality (mange manuelle søknader → data quality issues)
|
||||
data_quality = DataQualitySignal(
|
||||
reference_data=training_data,
|
||||
features=['søkers_alder', 'arbeidserfaring', 'utdanning'],
|
||||
metric_thresholds=DataQualityMetricThreshold(
|
||||
numerical=DataQualityMetricsNumerical(null_value_rate=0.02),
|
||||
categorical=DataQualityMetricsCategorical(out_of_bounds_rate=0.01)
|
||||
),
|
||||
alert_enabled=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Kostnad og lisensiering
|
||||
|
||||
### Compute Costs (Serverless Spark)
|
||||
|
||||
**Baseline** (Azure pricing model):
|
||||
|
||||
| VM Size | vCPUs | RAM | Typical Use Case | Estimert Cost/Time |
|
||||
|---------|-------|-----|------------------|-------------------|
|
||||
| standard_e4s_v3 | 4 | 32 GB | Small datasets (<1M rows) | Lavest |
|
||||
| standard_e8s_v3 | 8 | 64 GB | Medium datasets (1M-10M rows) | Medium |
|
||||
| standard_e16s_v3 | 16 | 128 GB | Large datasets (10M-100M rows) | Høy |
|
||||
| standard_e32s_v3 | 32 | 256 GB | Very large datasets (100M+ rows) | Veldig høy |
|
||||
| standard_e64s_v3 | 64 | 512 GB | Enterprise scale | Svært høy |
|
||||
|
||||
**Cost Optimization Strategies:**
|
||||
|
||||
1. **Monitor subset av features** (ikke alle):
|
||||
```python
|
||||
features=MonitorFeatureFilter(top_n_feature_importance=10) # Ikke 100+ features
|
||||
```
|
||||
|
||||
2. **Juster monitoring frequency** basert på data vekst:
|
||||
- High traffic → daily (men større window size)
|
||||
- Low traffic → weekly eller monthly
|
||||
|
||||
3. **Bruk lookback windows strategisk**:
|
||||
```python
|
||||
# Større window = mindre frequent runs
|
||||
data_window=BaselineDataRange(
|
||||
lookback_window_size="P7D", # 7 dager i stedet for P1D
|
||||
lookback_window_offset="P0D"
|
||||
)
|
||||
```
|
||||
|
||||
4. **Limit number of monitoring signals** per monitor:
|
||||
- Start med data drift + data quality
|
||||
- Legg til feature attribution drift bare hvis nødvendig
|
||||
|
||||
### Licensing Requirements
|
||||
|
||||
**Verified** (Azure ML pricing):
|
||||
|
||||
| Component | License/SKU Required | Notes |
|
||||
|-----------|---------------------|-------|
|
||||
| Azure ML workspace | Azure subscription | Ingen ekstra license |
|
||||
| Model monitoring | Inkludert i Azure ML | Ingen ekstra cost utover compute |
|
||||
| Serverless Spark | Pay-per-use (compute timer) | Charged per vCPU-hour |
|
||||
| Data storage | Azure Blob Storage standard pricing | Pay for storage used |
|
||||
| Event Grid | Standard Event Grid pricing | Første 100k operations/måned gratis |
|
||||
|
||||
### Estimert Monthly Cost (Eksempel)
|
||||
|
||||
**Scenario**: Fraud detection model, 1M transactions/day, monitor daily
|
||||
|
||||
| Component | Details | Estimert Monthly Cost (NOK) |
|
||||
|-----------|---------|----------------------------|
|
||||
| Serverless Spark | standard_e4s_v3, ~15 min/dag | ~2000-3000 |
|
||||
| Blob Storage | ~100 GB production data | ~20-30 |
|
||||
| Event Grid | ~30 events/måned | Gratis (under limit) |
|
||||
| **Total** | | **~2500-3500 NOK/måned** |
|
||||
|
||||
**Baseline**: For enterprise deployments med multiple modeller, regn ~3000-5000 NOK/modell/måned avhengig av data volume og frequency.
|
||||
|
||||
---
|
||||
|
||||
## For arkitekten (Cosmo)
|
||||
|
||||
### Når anbefale model monitoring?
|
||||
|
||||
**Obligatorisk scenarios:**
|
||||
1. ✅ Produksjonsmodeller i regulerte domener (helse, finans, justis)
|
||||
2. ✅ High-stakes decisions (fraud detection, credit scoring, medical diagnosis)
|
||||
3. ✅ Modeller med kjent risk for drift (seasonality, market changes)
|
||||
4. ✅ Compliance requirements (AI Act, GDPR, internal governance)
|
||||
5. ✅ Long-lived models (deployed >6 måneder)
|
||||
|
||||
**Nice-to-have scenarios:**
|
||||
- Medium-stakes models (recommendations, content filtering)
|
||||
- Exploratory models i pilot phase
|
||||
- Models med infrequent retraining cycles
|
||||
|
||||
**Ikke nødvendig:**
|
||||
- Prototype/POC models uten production traffic
|
||||
- Models med continuous retraining (daily/weekly)
|
||||
- Simple rule-based systems (ikke ML)
|
||||
|
||||
### Beslutningstre for signal selection
|
||||
|
||||
```
|
||||
START: Hvilke signals trenger kunden?
|
||||
|
||||
1. Er modellen deployed til Azure ML online endpoint?
|
||||
JA → Bruk out-of-box monitoring (data drift + prediction drift + data quality automatic)
|
||||
NEI → Fortsett til 2
|
||||
|
||||
2. Er modellen deployed utenfor Azure ML?
|
||||
JA → Krever custom preprocessing component (Pattern 5)
|
||||
NEI → Modellen er i batch endpoint → custom preprocessing (Pattern 5)
|
||||
|
||||
3. Har kunden tilgang til ground truth data?
|
||||
JA → Inkluder model performance signal (Pattern 3)
|
||||
NEI → Fortsett til 4
|
||||
|
||||
4. Er feature importance kritisk for forståelsen?
|
||||
JA → Inkluder feature attribution drift (Pattern 2) - krever training data + both inputs/outputs
|
||||
NEI → Fortsett til 5
|
||||
|
||||
5. Finnes det domene-spesifikke metrics som ikke er innebygde?
|
||||
JA → Utvikle custom signal component (Pattern 4)
|
||||
NEI → Standard signals er sufficient
|
||||
|
||||
6. Hva er production traffic volume?
|
||||
Høy (daglig data) → Daily monitoring
|
||||
Medium (ukentlig data) → Weekly monitoring
|
||||
Lav (månedlig data) → Monthly monitoring
|
||||
```
|
||||
|
||||
### Typical Consulting Conversation Flow
|
||||
|
||||
**Fase 1: Discover (Forstå modellen)**
|
||||
- "Hva slags modell er dette? (classification/regression/generative)"
|
||||
- "Hvor er modellen deployed? (Azure ML online/batch/external)"
|
||||
- "Hvor mye production traffic har dere? (requests/dag)"
|
||||
- "Har dere tilgang til ground truth data? Hvor raskt er det tilgjengelig?"
|
||||
- "Hvilke features er mest kritiske for business?"
|
||||
|
||||
**Fase 2: Design (Foreslå løsning)**
|
||||
- "Based på at dere har X traffic og Y deployment, anbefaler jeg Z monitoring frequency"
|
||||
- "For deres use case (fraud/health/etc), er data quality og model performance kritisk"
|
||||
- "Vi setter opp data drift med training data som baseline for å få feature importance"
|
||||
- "For ground truth integration, trenger vi correlation ID strategy - har dere unique transaction IDs?"
|
||||
|
||||
**Fase 3: Implementation Guidance**
|
||||
- "Start med out-of-box for å få baseline, deretter tune thresholds basert på første runs"
|
||||
- "For Event Grid integration, anbefaler jeg Azure Functions for retraining trigger"
|
||||
- "Vi må registrere preprocessing component hvis dere samler data utenfor Azure ML"
|
||||
- "For compliance, dokumenter threshold rationale i ADR (Architecture Decision Record)"
|
||||
|
||||
**Fase 4: Operationalization**
|
||||
- "Hvem skal motta alerts? Sett opp alert_notification emails"
|
||||
- "Definer runbook for hva teamet gjør når drift detekteres"
|
||||
- "Integrer med Linear/Jira for incident tracking via Event Grid"
|
||||
- "Schedule monthly review av monitoring metrics med data science team"
|
||||
|
||||
### Red Flags (Når kunden trenger mer enn monitoring)
|
||||
|
||||
| Red Flag | Implikasjon | Anbefaling |
|
||||
|----------|-------------|------------|
|
||||
| "Vi retrainer aldri modellen" | Model vil degrade over tid | Sett opp retraining pipeline FØRST, deretter monitoring |
|
||||
| "Vi har ingen ground truth" | Kan ikke måle actual performance | Utvikle ground truth collection strategy (async) |
|
||||
| "Vi vet ikke hvilke features som er viktige" | Vanskelig å prioritere monitoring | Kjør feature importance analysis før setup |
|
||||
| "Modellen er deployed for 2 år siden uten endringer" | Sannsynligvis allerede degraded | Start med ad-hoc monitoring run for å assess current state |
|
||||
| "Vi har 500+ features" | Compute cost vil bli høy | Monitor top 20-30 features, ikke alle |
|
||||
|
||||
### Integration med Responsible AI Framework
|
||||
|
||||
Model monitoring er **ongoing compliance layer** i Responsible AI framework:
|
||||
|
||||
```
|
||||
Training Phase:
|
||||
↓ Feature importance analysis → Baseline for monitoring
|
||||
↓ Fairness evaluation → Custom fairness signals
|
||||
↓ Model cards documentation → Reference for threshold setting
|
||||
|
||||
Deployment Phase:
|
||||
↓ Data collection setup → Production data for monitoring
|
||||
↓ Initial monitoring setup → Out-of-box signals
|
||||
|
||||
Production Phase:
|
||||
↓ Continuous monitoring → This document
|
||||
↓ Drift detection → Trigger retraining
|
||||
↓ Ground truth validation → Model performance tracking
|
||||
↓ Event Grid integration → Automated remediation
|
||||
|
||||
Governance Phase:
|
||||
↓ Audit trail → Monitoring history for compliance
|
||||
↓ Metrics reporting → Quarterly reviews
|
||||
↓ Threshold adjustments → Based on business feedback
|
||||
```
|
||||
|
||||
### Quick Reference: Pattern Selection Matrix
|
||||
|
||||
| Deployment Type | Data Collection | Ground Truth | Recommended Pattern |
|
||||
|-----------------|-----------------|--------------|-------------------|
|
||||
| Azure ML online endpoint | Auto (data collector) | ❌ | Pattern 1 (Out-of-box) |
|
||||
| Azure ML online endpoint | Auto (data collector) | ✅ | Pattern 1 + Pattern 3 (Performance) |
|
||||
| Azure ML online endpoint | Auto (data collector) | ✅ + Feature importance needed | Pattern 2 (Advanced) + Pattern 3 |
|
||||
| Azure ML batch endpoint | Manual | ❌ | Pattern 5 (Custom preprocessing) |
|
||||
| External (AKS/ACI/on-prem) | Manual | ✅ | Pattern 5 + Pattern 3 |
|
||||
| Any | Any | Custom metrics needed | Pattern 4 (Custom signals) |
|
||||
|
||||
### Sample Architecture Decision Record (ADR) Template
|
||||
|
||||
Når du anbefaler monitoring setup, dokumenter med ADR:
|
||||
|
||||
```markdown
|
||||
# ADR-XXX: Model Monitoring Setup for [Model Name]
|
||||
|
||||
## Status
|
||||
Proposed / Accepted
|
||||
|
||||
## Context
|
||||
- Model type: Classification/Regression
|
||||
- Deployment: Azure ML online endpoint / Batch / External
|
||||
- Production traffic: X requests/day
|
||||
- Business criticality: High/Medium/Low
|
||||
- Regulatory requirements: AI Act / GDPR / Sector-specific
|
||||
|
||||
## Decision
|
||||
Implement Azure Machine Learning model monitoring with:
|
||||
- Signals: Data drift, Data quality, [Model performance if ground truth available]
|
||||
- Reference data: Training data
|
||||
- Frequency: Daily/Weekly/Monthly
|
||||
- Thresholds: [Specific values with rationale]
|
||||
- Event Grid integration: Yes/No
|
||||
|
||||
## Consequences
|
||||
- Positive: Early detection of drift, compliance coverage, automated alerts
|
||||
- Negative: Monthly cost ~X NOK, requires serverless Spark compute
|
||||
- Mitigation: Monitor top N features only, adjust frequency based on learnings
|
||||
|
||||
## Implementation
|
||||
- Phase 1: Out-of-box setup (week 1)
|
||||
- Phase 2: Threshold tuning based on initial runs (week 2-4)
|
||||
- Phase 3: Event Grid + retraining pipeline integration (week 5-6)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Kilder og verifisering
|
||||
|
||||
### Verified Sources (MCP Research)
|
||||
|
||||
1. **Azure Machine Learning model monitoring** (Concept)
|
||||
- URL: https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2
|
||||
- Verified: Capabilities, signals, metrics, best practices
|
||||
- Confidence: High (official docs, jan 2026)
|
||||
|
||||
2. **Monitor the performance of models deployed to production** (How-to)
|
||||
- URL: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2
|
||||
- Verified: Setup procedures, Event Grid integration, lookback windows
|
||||
- Confidence: High (official docs, jan 2026)
|
||||
|
||||
3. **Data drift (preview) will be retired, and replaced by Model Monitor** (Legacy)
|
||||
- URL: https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1
|
||||
- Verified: Legacy v1 concepts, migration context
|
||||
- Confidence: Medium (deprecated, but useful for understanding evolution)
|
||||
|
||||
4. **Test and evaluate AI workloads on Azure** (Guidance)
|
||||
- URL: https://learn.microsoft.com/en-us/azure/well-architected/ai/test#guidance-for-testing-model-training-and-fine-tuning
|
||||
- Verified: Data drift vs concept drift definitions, testing best practices
|
||||
- Confidence: High (Azure Well-Architected Framework)
|
||||
|
||||
### Code Samples (Verified)
|
||||
|
||||
- **Python SDK examples**: azureml-datadrift package (v1), azure-ai-ml (v2)
|
||||
- **YAML configurations**: Model monitoring schedule definitions
|
||||
- **Custom component examples**: azureml-examples GitHub repo
|
||||
|
||||
### Baseline Sources (Model Knowledge)
|
||||
|
||||
- AI Act compliance requirements (European Parliament, 2024)
|
||||
- GDPR data protection principles (GDPR Art. 5, Art. 25)
|
||||
- MLOps best practices (Azure AI Playbook)
|
||||
- Offentlig sektor AI governance (KS/Difi retningslinjer)
|
||||
- Fairness metrics (demographic parity, equalized odds)
|
||||
|
||||
### Total MCP Calls: 4
|
||||
- microsoft_docs_search: 3 queries
|
||||
- microsoft_docs_fetch: 2 deep reads
|
||||
- microsoft_code_sample_search: 1 query
|
||||
|
||||
### Total Unique URLs: 9
|
||||
- Primary: 4 (concept, how-to, legacy, well-architected)
|
||||
- Secondary: 5 (referenced in code samples and related docs)
|
||||
Loading…
Add table
Add a link
Reference in a new issue