feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-08 08:58:35 +02:00 · 2026-04-08 08:58:35 +02:00 · baa2d0220b
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
--- a/skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md
+++ b/skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md
@ -0,0 +1,506 @@
+# Anomaly Detection for AI Systems
+
+**Dato:** 5. februar 2026
+**Kategori:** Monitoring & Observability
+**Målgruppe:** AI-arkitekter, DevOps-team, MLOps-ingeniører
+
+## Oversikt
+
+Anomaly detection er kritisk for proaktiv overvåking av AI-systemer. Microsoft Azure tilbyr flere mekanismer for å oppdage avvikende oppførsel i AI-applikasjoner, fra innebygde ML-baserte funksjoner til dedikerte tjenester. Effektiv anomaly detection reduserer tiden fra et problem oppstår til det blir oppdaget (dwell time) og muliggjør raskere respons på trusler og systemfeil.
+
+## Smart Detection Capabilities
+
+### Application Insights Smart Detection
+
+Application Insights inkluderer automatisk smart detection som bruker maskinlæring til å oppdage avvik uten konfigurasjon. Systemet analyserer telemetri kontinuerlig og varsler automatisk ved potensielle problemer.
+
+**Hovedfunksjoner:**
+
+1. **Failure Anomalies Detection**
+   - Oppdager unormal økning i feilrate
+   - Korrelerer feilrater med last og andre faktorer
+   - Bruker maskinlæring til å etablere forventet baseline
+   - Trenger 24 timer med data før aktivering
+
+2. **Performance Anomalies Detection**
+   - Detekterer degradering i responstid
+   - Analyserer både requests og dependencies
+   - Identifiserer mønstre i page load time
+   - Sammenligner med historisk baseline
+
+3. **General Degradations**
+   - Trace severity degradation
+   - Memory leaks
+   - Abnormal exception volume
+   - Security anti-patterns
+
+**Konfigurasjon:**
+
+Smart detection krever ingen oppsett hvis Application Insights sender nok telemetri. Default e-postvarsler sendes til Monitoring Reader og Monitoring Contributor-roller.
+
+```json
+// Azure Resource Manager template for konfigurasjon
+{
+  "type": "Microsoft.Insights/components/ProactiveDetectionConfigs",
+  "properties": {
+    "enabled": true,
+    "sendEmailsToSubscriptionOwners": true,
+    "customEmails": ["ops-team@example.com"]
+  }
+}
+```
+
+### Migrering til Alert-Based Smart Detection
+
+Microsoft anbefaler å migrere smart detection til alerts-basert system for bedre kontroll:
+
+- Oppretter alert rules for hver deteksjonsmodul
+- Muliggjør action groups for notifikasjoner
+- Gir bedre integrasjon med Azure Monitor alerts
+- Støtter multiple notification methods
+
+**Migreringsmåter:**
+1. Via Azure Portal (manuell migrering)
+2. Via Azure CLI med REST API
+3. Via ARM templates for batch-migrering
+
+## Custom Anomaly Rules for AI
+
+### KQL Machine Learning Functions
+
+Azure Monitor Logs støtter KQL-baserte ML-funksjoner for anomaly detection uten behov for datascience-ekspertise.
+
+**series_decompose_anomalies() - Hovedfunksjon:**
+
+```kusto
+// Detect anomalies i AI-telemetri
+let starttime = 21d;
+let endtime = 0d;
+let timeframe = 1h; // Sample frequency
+AIRequests
+| where TimeGenerated between (startofday(ago(starttime))..startofday(ago(endtime)))
+| make-series RequestRate=count() default=0
+    on TimeGenerated
+    from startofday(ago(starttime))
+    to startofday(ago(endtime))
+    step timeframe
+    by ModelName
+| extend (Anomalies, AnomalyScore, ExpectedRate) =
+    series_decompose_anomalies(RequestRate, 1.5, -1, 'avg', 1)
+| mv-expand RequestRate to typeof(double),
+    TimeGenerated to typeof(datetime),
+    Anomalies to typeof(double),
+    AnomalyScore to typeof(double),
+    ExpectedRate to typeof(long)
+| where Anomalies != 0
+| project TimeGenerated, ModelName, RequestRate, ExpectedRate, AnomalyScore, Anomalies
+| sort by abs(AnomalyScore) desc
+```
+
+**Parametere for tuning:**
+
+- **Threshold** (default 1.5): Justerer sensitivitet – lavere verdi gir flere anomalier
+- **Seasonality** (default -1): Auto-detect sesongvariasjoner
+- **Trend** (default 'avg'): 'avg', 'linefit', eller 'none'
+- **Test_points**: Antall punkter å ekskludere fra learning (for outliers)
+- **AD_method**: Anomaly detection-metode
+
+### Root Cause Analysis med diffpatterns()
+
+Når anomalier oppdages, bruk `diffpatterns()` plugin for å identifisere årsaker:
+
+```kusto
+let anomalyDate = datetime(2026-02-05T12:00:00Z);
+AIRequests
+| extend AnomalyDate = iff(TimeGenerated == anomalyDate, "AnomalyDate", "OtherDates")
+| where TimeGenerated between (ago(7d)..now())
+| project AnomalyDate, Operation, ResultCode, ModelVersion, Region
+| evaluate diffpatterns(AnomalyDate, "OtherDates", "AnomalyDate", "~", 0.20)
+```
+
+**Output:** Tabell som viser hvilke dimensjoner (operation, resultcode, etc.) som varierer mest mellom normal og anomal periode.
+
+## Behavioral Baseline Detection
+
+### Etablering av Baseline
+
+Smart detection etablerer automatisk behavioral baselines over tid:
+
+1. **Learning Period**: Minimum 24 timer (ofte 7-14 dager for robust baseline)
+2. **Continuous Learning**: Modellen oppdateres kontinuerlig med nye data
+3. **Context-Aware**: Korrelerer med faktorer som load, tid på døgnet, ukedag
+4. **Adaptive Thresholds**: Dynamiske terskler basert på historikk
+
+### Dynamic Thresholds for Metric Alerts
+
+Azure Monitor tilbyr dynamiske terskler basert på maskinlæring for metric alerts:
+
+```json
+{
+  "criteria": {
+    "allOf": [{
+      "name": "AI Model Response Time",
+      "metricName": "ResponseTime",
+      "operator": "GreaterThan",
+      "threshold": "dynamic",
+      "sensitivity": "Medium",
+      "failingPeriods": {
+        "numberOfEvaluationPeriods": 4,
+        "minFailingPeriodsToAlert": 3
+      }
+    }]
+  }
+}
+```
+
+**Sensitivity levels:**
+- **High**: Lavere toleranse, fanger flere anomalier (mer false positives)
+- **Medium**: Balansert (anbefalt for de fleste scenarioer)
+- **Low**: Høyere toleranse, færre varsler
+
+### Behavioral Patterns for AI Systems
+
+Spesifikke mønstre å overvåke for AI-systemer:
+
+1. **Input Anomalies**
+   - Uventede prompt-lengder
+   - Høy forekomst av special characters
+   - Repetitive patterns (potensielt angrep)
+
+2. **Output Anomalies**
+   - Plutselig endring i response-lengder
+   - Avvik i token consumption patterns
+   - Uventede confidence scores
+
+3. **Performance Anomalies**
+   - Latency spikes
+   - Throughput degradation
+   - Rate limit hits
+
+4. **Resource Anomalies**
+   - Abnormal compute usage
+   - Memory consumption spikes
+   - Storage I/O patterns
+
+## Drift Detection Patterns
+
+Model drift er en spesiell form for anomaly detection kritisk for AI-systemer.
+
+### Data Drift Detection
+
+Overvåk endringer i input-distribusjon:
+
+```kusto
+// Detect distribution shifts i prompt-karakteristikker
+let baseline_period = 7d;
+let current_period = 1d;
+let baseline = AIRequests
+    | where TimeGenerated between (ago(baseline_period + current_period)..ago(current_period))
+    | summarize
+        AvgTokens=avg(PromptTokens),
+        StdDevTokens=stdev(PromptTokens),
+        P50=percentile(PromptTokens, 50),
+        P95=percentile(PromptTokens, 95)
+    | extend Period = "Baseline";
+let current = AIRequests
+    | where TimeGenerated > ago(current_period)
+    | summarize
+        AvgTokens=avg(PromptTokens),
+        StdDevTokens=stdev(PromptTokens),
+        P50=percentile(PromptTokens, 50),
+        P95=percentile(PromptTokens, 95)
+    | extend Period = "Current";
+union baseline, current
+| evaluate pivot(Period, sum(AvgTokens), sum(P50), sum(P95))
+| extend
+    AvgDrift = (Current_AvgTokens - Baseline_AvgTokens) / Baseline_AvgTokens * 100,
+    P95Drift = (Current_P95 - Baseline_P95) / Baseline_P95 * 100
+| where abs(AvgDrift) > 15 or abs(P95Drift) > 20 // Threshold: 15% avg eller 20% P95
+```
+
+### Concept Drift Detection
+
+Overvåk endringer i modell-utdata:
+
+```python
+from azure.ai.anomalydetector import AnomalyDetectorClient
+from azure.core.credentials import AzureKeyCredential
+
+# Azure AI Anomaly Detector for univariate series
+client = AnomalyDetectorClient(endpoint, AzureKeyCredential(api_key))
+
+# Time series av confidence scores
+series = [
+    TimeSeriesPoint(timestamp=row[0], value=row[1])  # confidence score
+    for row in data
+]
+
+request = UnivariateDetectionOptions(
+    series=series,
+    granularity=TimeGranularity.HOURLY,
+    sensitivity=90
+)
+
+# Detect både anomalies og change points
+anomaly_response = client.detect_univariate_entire_series(request)
+changepoint_response = client.detect_univariate_change_point(request)
+
+for i, (is_anomaly, is_changepoint) in enumerate(
+    zip(anomaly_response.is_anomaly, changepoint_response.is_change_point)
+):
+    if is_changepoint:
+        # Persistent shift - potential concept drift
+        alert_drift(timestamp=series[i].timestamp)
+    elif is_anomaly:
+        # Temporary spike - potential transient issue
+        alert_anomaly(timestamp=series[i].timestamp)
+```
+
+## Alert Correlation
+
+### Korrelere Anomalier med Hendelser
+
+Best practice er å korrelere anomalier med andre events:
+
+```kusto
+// Korrelere performance anomalies med deployment events
+let anomalies = AIRequests
+    | where TimeGenerated > ago(7d)
+    | make-series RequestRate=count() default=0 on TimeGenerated step 5m
+    | extend (Anomalies, Score, Expected) = series_decompose_anomalies(RequestRate)
+    | mv-expand TimeGenerated to typeof(datetime), Anomalies to typeof(double), Score to typeof(double)
+    | where Anomalies != 0
+    | project AnomalyTime=TimeGenerated, Score;
+let deployments = AzureActivity
+    | where OperationNameValue == "MICROSOFT.RESOURCES/DEPLOYMENTS/WRITE"
+    | where ActivityStatusValue == "Success"
+    | project DeploymentTime=TimeGenerated, ResourceGroup, Deployment=Properties.deployment;
+anomalies
+| join kind=inner (deployments) on $left.AnomalyTime == $right.DeploymentTime
+| where abs(datetime_diff('minute', AnomalyTime, DeploymentTime)) < 30
+| project AnomalyTime, DeploymentTime, Score, ResourceGroup, Deployment
+| order by Score desc
+```
+
+### Multi-Signal Correlation
+
+Korrelere anomalier på tvers av signaler:
+
+1. **Application-level metrics** (latency, throughput, errors)
+2. **Infrastructure metrics** (CPU, memory, network)
+3. **Model metrics** (confidence scores, token usage)
+4. **Security signals** (authentication failures, suspicious patterns)
+
+### Action Groups for Automated Response
+
+Konfigurer action groups for koordinerte responser:
+
+```json
+{
+  "actionGroups": [
+    {
+      "actionGroupId": "/subscriptions/{sub}/resourceGroups/{rg}/providers/Microsoft.Insights/actionGroups/AIAnomalyResponse",
+      "webhookProperties": {
+        "anomaly_type": "performance",
+        "severity": "high",
+        "auto_scale": "true"
+      }
+    }
+  ]
+}
+```
+
+**Mulige actions:**
+- Email/SMS/Push notifications
+- Webhook til incident management system
+- Azure Function for automated remediation
+- Logic App for workflow orchestration
+- ITSM connector (ServiceNow, etc.)
+
+## Azure AI-Specific Detection
+
+### Defender for AI Services
+
+Microsoft Defender for AI tilbyr spesialisert anomaly detection for AI-spesifikke trusler:
+
+1. **Jailbreak Attempt Detection**
+   - Mønstergjenkjenning av jailbreak-teknikker
+   - Analyser prompt injection patterns
+   - Korrelere med MITRE ATLAS framework
+
+2. **Model Inference Anomalies**
+   - Uvanlige API call patterns
+   - Excessive inference requests
+   - Suspicious input/output correlations
+
+3. **Data Exfiltration Patterns**
+   - Abnormal data access via model queries
+   - High-volume low-latency requests
+   - Sensitive data in prompts/responses
+
+**Aktivering:**
+
+Defender for AI aktiveres via Security Center:
+
+```bash
+# Azure CLI
+az security pricing create \
+  --name AIServices \
+  --tier Standard \
+  --subscription <subscription-id>
+```
+
+### Azure AI Anomaly Detector Service
+
+Dedikert service for anomaly detection (NB: Retired 1. oktober 2026 – bruk alternativene nedenfor):
+
+**Alternativer etter retirement:**
+
+1. **Azure ML model monitoring** – for model-spesifikk anomaly detection
+2. **Azure Monitor KQL-baserte funksjoner** – for log-basert detection
+3. **Azure Stream Analytics** – for real-time streaming anomaly detection
+4. **Custom models** i Azure ML – for spesialiserte use cases
+
+### Real-Time Intelligence Anomaly Detection (Fabric)
+
+For organisasjoner med Microsoft Fabric:
+
+```python
+# Python plugin i Eventhouse
+from synapse.ml.services import SimpleDetectAnomalies
+
+anomaly_detector = (SimpleDetectAnomalies()
+  .setTimestampCol("timestamp")
+  .setValueCol("model_confidence")
+  .setOutputCol("anomalies")
+  .setGroupbyCol("model_name")
+  .setGranularity("hourly"))
+
+result = anomaly_detector.transform(df)
+display(result.select("timestamp", "model_confidence", "anomalies.isAnomaly"))
+```
+
+## Implementeringsmønster
+
+### 1. Etabler Baseline (Uke 1-2)
+
+```kusto
+// Etabler baseline for key metrics
+AIRequests
+| where TimeGenerated between (ago(14d)..now())
+| summarize
+    P50_Latency=percentile(Duration, 50),
+    P95_Latency=percentile(Duration, 95),
+    P99_Latency=percentile(Duration, 99),
+    AvgTokens=avg(TotalTokens),
+    ErrorRate=countif(Success == false) * 100.0 / count()
+  by bin(TimeGenerated, 1h), ModelName
+| render timechart
+```
+
+### 2. Konfigurer Anomaly Detection
+
+```bash
+# Opprett alert rule med dynamic threshold
+az monitor metrics alert create \
+  --name "AI-Latency-Anomaly" \
+  --resource-group <rg> \
+  --scopes <app-insights-id> \
+  --condition "avg requests/duration > dynamic High 4 of 4" \
+  --window-size 5m \
+  --evaluation-frequency 1m \
+  --action <action-group-id>
+```
+
+### 3. Implementer Root Cause Analysis Automation
+
+```python
+# Azure Function triggered av alert
+import azure.functions as func
+from azure.monitor.query import LogsQueryClient
+
+def main(req: func.HttpRequest) -> func.HttpResponse:
+    alert_data = req.get_json()
+    anomaly_time = alert_data['data']['context']['timestamp']
+
+    # Query for root cause
+    query = f"""
+    AIRequests
+    | where TimeGenerated between (datetime({anomaly_time}) - 30m .. datetime({anomaly_time}) + 30m)
+    | summarize ErrorCount=countif(Success==false) by Operation, ResultCode
+    | top 10 by ErrorCount desc
+    """
+
+    result = logs_client.query_workspace(workspace_id, query)
+
+    # Send enriched alert
+    send_enriched_alert(result)
+```
+
+### 4. Continuous Tuning
+
+Juster sensitivitet basert på false positive rate:
+
+- Hvis > 30% false positives: øk threshold eller sensitivity
+- Hvis < 5% false positives: reduser threshold for tidligere detection
+- Revurder baseline hver måned ved sesongrelaterte endringer
+
+## For Cosmo
+
+### Når anbefale anomaly detection
+
+**ALLTID anbefal** for:
+- Produksjons-AI-applikasjoner med høy trafikk
+- AI-systemer med sensitive data eller compliance-krav
+- Multimodal AI-løsninger med komplekse dependencies
+- AI-agenter med autonom beslutningskraft
+
+**Ikke kritisk** for:
+- Proof-of-concepts under utvikling
+- Lavtrafikks prototype-løsninger uten produksjonsdata
+
+### Platform-spesifikke anbefalinger
+
+| Plattform | Primær Metode | Sekundær Metode |
+|-----------|---------------|-----------------|
+| Azure AI Foundry | Application Insights Smart Detection | KQL-baserte custom queries |
+| Copilot Studio | M365 audit logs + KQL | Application Insights (via plugin) |
+| Power Platform AI | Application Insights + Power Platform analytics | Custom Dataverse queries |
+| Azure OpenAI Service | Application Insights + Defender for AI | Azure Monitor metric alerts |
+
+### Arkitekturdialog
+
+**Spørsmål å stille:**
+
+1. "Hvilke typer avvik er viktigst å oppdage for deres AI-applikasjon – performance, sikkerhet, eller datakvalitet?"
+2. "Har dere eksisterende alert-systemer dette må integreres med?"
+3. "Hva er akseptabel responstid fra anomaly til varsling?"
+4. "Trenger dere automated remediation eller kun notifikasjoner?"
+
+**Typiske trade-offs:**
+
+- **Sensitivity vs. Alert Fatigue**: Høyere sensitivitet gir flere false positives
+- **Real-time vs. Batch**: Real-time detection krever mer ressurser
+- **Custom vs. Built-in**: Custom ML-modeller gir bedre presisjon men høyere vedlikeholdskostnad
+
+### Kostnadsestimat
+
+Anomaly detection koster primært via:
+1. **Log Analytics ingestion**: ~NOK 30/GB
+2. **Application Insights**: Inkludert i Basic-tier (gratis til 5 GB/mnd)
+3. **Alert rules**: Gratis for første 10 metric alerts, NOK 1/mnd per ekstra
+4. **Action groups**: Gratis for de fleste notification types
+
+**Tommelfingerregel:** Budsjetter NOK 500-2000/mnd for typisk produksjons-AI-app med comprehensive anomaly detection.
+
+---
+
+**Sources:**
+- [Tutorial: Detect and analyze anomalies using KQL](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/kql-machine-learning-azure-monitor)
+- [Smart detection in Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics)
+- [Detect and mitigate potential issues using AIOps and machine learning](https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/aiops-machine-learning)
+- [Azure Monitor dynamic thresholds](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-dynamic-thresholds)
+- [Microsoft Defender for AI Services](https://learn.microsoft.com/en-us/azure/defender-for-cloud/ai-threat-protection)
+- [Anomaly detection in Real-Time Intelligence (Fabric)](https://learn.microsoft.com/en-us/fabric/real-time-intelligence/anomaly-detection)
+- [Azure AI Anomaly Detector](https://learn.microsoft.com/en-us/azure/ai-services/anomaly-detector/overview) (retired Oct 2026)
+- [Azure Stream Analytics anomaly detection](https://learn.microsoft.com/en-us/azure/stream-analytics/stream-analytics-machine-learning-anomaly-detection)