feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions

View file

@ -0,0 +1,576 @@
# Alerting Strategies and Escalation Policies for AI Incidents
**Last updated:** 2026-02
**Status:** GA
**Category:** Monitoring & Observability
---
## Introduksjon
Effektive alerting-strategier og eskaleringsrutiner er kritiske for å sikre rask respons på AI-relaterte hendelser. I motsetning til tradisjonelle applikasjoner introduserer AI-systemer unike utfordringer: modeller kan degradere over tid (drift), prompt injection-angrep kan oppstå plutselig, og token-kostnader kan eksplodere uten varsel. En robust alerting-arkitektur må derfor kombinere reaktive varsler (noe gikk galt) med proaktive varsler (noe er i ferd med å gå galt).
Azure Monitor gir et omfattende rammeverk for alerting gjennom action groups, alert processing rules, og integrasjon med Azure Automation, Logic Apps, og ITSM-systemer. For AI-løsninger må denne infrastrukturen konfigureres med forståelse av både business impact og teknisk kompleksitet — en kritisk alert kan være en modell som returnerer bias-innhold, eller en Azure OpenAI-deployment som nærmer seg rate limit.
Eskaleringsrutiner må reflektere organisasjonens modenhetsnivå. En Minimum Viable Product (MVP)-løsning kan starte med e-postvarsler til et lite team, mens en produksjonsløsning for offentlig sektor krever Multi-tier escalation med clear ownership, automated incident response, og compliance logging for AI Act Article 72 (incident reporting).
---
## Kjernekomponenter
### Azure Monitor Alert Architecture
| Komponent | Beskrivelse | AI-relevans |
|-----------|-------------|-------------|
| **Alert Rules** | Definerer betingelser som trigger alerts (metrics, logs, activity log) | Token rate limits, model latency, failed requests |
| **Action Groups** | Samling av notifications og actions som kjøres når alert fires | Email, SMS, webhook, runbook, Logic App |
| **Alert Processing Rules** | Overstyr eller beriket alert-oppførsel (f.eks. suppression under maintenance) | Prevent alert fatigue under model redeployments |
| **Common Alert Schema** | Uniform JSON payload på tvers av alle alert-typer | Forenkler webhook-integrasjoner og ITSM-connectors |
| **Severity Levels** | Sev 0 (Critical) til Sev 4 (Informational) | Map til business impact (Sev 0 = PII leak, Sev 4 = latency spike) |
### Notification Channels
| Channel | Bruksområde | Rate Limits | Global Resilience |
|---------|-------------|-------------|-------------------|
| **Email** | Standard notification (opptil 1000 mottakere per action group) | Best practices: Ikke send til personlige adresser, bruk distribution lists | ✅ Yes |
| **SMS** | Kritiske alerts (begrensede land) | 1 SMS per 5 min per nummer | ✅ Yes |
| **Voice Call** | Sev 0 incidents (natt/helg) | 1 call per 5 min per nummer | ✅ Yes |
| **Webhook** | Integration med eksterne systemer (PagerDuty, Slack) | Retry: 5 retries med 5-40s delay | ❌ Endpoint-dependent |
| **Azure App Push** | Mobile notifications til Azure-appen | Begrenset til Azure mobile app | ✅ Yes |
| **Event Hub** | Stream alerts til analytics/SIEM | Supports Private Link og NSP | ✅ Yes (regional) |
**Rate Limiting:** Azure Monitor rate-limiter notifications for å hindre spam. Hvis samme email/SMS/telefonnummer mottar for mange alerts, suspenderes notifications midlertidig. For AI-systemer som kan generere høy alert-volum (f.eks. per-request failures), bruk programmatic actions (Logic Apps, Automation Runbooks) i stedet.
### Action Types for AI Incidents
```json
{
"actionType": "AutomationRunbook",
"runbookName": "ScaleDownOpenAI",
"webhookResourceId": "/subscriptions/.../runbooktest/webhooks/Alert...",
"useCommonAlertSchema": true,
"isGlobalRunbook": false
}
```
| Action Type | AI Use Case | Authentication | Cross-tenant Support |
|-------------|-------------|----------------|---------------------|
| **Automation Runbook** | Auto-scale Azure OpenAI TPM, restart failing deployments | Managed Identity (Automation Contributor role) | ❌ No |
| **Logic App** | Enrich alert med model metadata, post til Teams/Slack | Managed Identity (Logic App Contributor) | ❌ No |
| **Azure Function** | Custom logic (e.g., invoke model rollback API) | HTTP trigger med access key | ❌ No |
| **Webhook** | Invoke external incident mgmt (PagerDuty, ServiceNow) | Basic auth via URI eller secure webhook (Entra ID) | ✅ Yes (limited) |
| **Event Hub** | Stream til SIEM (Microsoft Sentinel) for correlation | Managed Identity (Event Hubs Data Sender) | ✅ Yes (up to API 2023-09) |
| **ITSM Connector** | Create incidents i ServiceNow, Cherwell | ITSM connection credentials | ❌ No |
**Managed Identity Best Practice:** For Automation Runbooks og Logic Apps, bruk managed identity i stedet for service principals. Azure Portal legger automatisk til role assignments. For PowerShell/CLI/SDK må du manuelt tildele roller (se tabell over).
---
## Arkitekturmønstre
### 1. Multi-Tier Escalation for AI Incidents
**Bruk når:** Produksjonsløsninger med SLA-krav og 24/7 support.
**Implementering:**
```plaintext
Tier 1: On-Call Developer (Email + SMS)
├─ Sev 3-4 alerts → Respond within 4 hours
└─ Auto-escalate to Tier 2 if no ACK within 30 min
Tier 2: AI Platform Team (Voice Call + PagerDuty)
├─ Sev 1-2 alerts → Respond within 30 min
└─ Auto-escalate to Tier 3 if no resolution within 2 hours
Tier 3: Management + Legal (Email + Teams)
└─ Sev 0 alerts → Data breach, AI Act violation, PII leak
```
**Azure Monitor Implementering:**
1. **Action Group per Tier:**
- `AG-Tier1-Developers`: Email til dev-team distribution list
- `AG-Tier2-Platform`: SMS + PagerDuty webhook
- `AG-Tier3-Executive`: Voice call til on-call manager + Teams notification
2. **Alert Processing Rule for Auto-Escalation:**
```json
{
"rules": {
"if": "alert.severity == 0 AND alert.state == 'New' FOR 30 minutes",
"then": "add action group AG-Tier3-Executive"
}
}
```
3. **Time-Based Escalation (via Logic App):**
- Webhook til Logic App som sjekker alert timestamp
- Hvis ikke acknowledged innen threshold → invoke Tier 2/3 action groups
**Fordeler:**
- Clear ownership per severity level
- Reduserer alert fatigue for Tier 3
- Automatisk eskalering hindrer at kritiske alerts "faller mellom stolene"
**Ulemper:**
- Kompleks konfigurasjon (krever Logic Apps for time-based escalation)
- Krever testing og dokumentasjon av eskaleringsrutiner
- Risiko for "false escalations" hvis thresholds er feil satt
---
### 2. Automated Remediation with Runbooks
**Bruk når:** Kjente failure modes med deterministiske fix-prosedyrer (scale-out, restart, rollback).
**Eksempel:** Azure OpenAI deployment nærmer seg TPM limit → Auto-scale til høyere tier.
**Runbook Template (PowerShell 7):**
```powershell
param(
[object] $WebhookData
)
# Parse Common Alert Schema
$alertData = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
$resourceId = $alertData.data.essentials.alertTargetIds[0]
$metricValue = $alertData.data.alertContext.condition.allOf[0].metricValue
# Extract OpenAI deployment info
$deployment = $resourceId -split '/' | Select-Object -Last 1
$rgName = ($resourceId -split '/')[4]
# Scale up to Standard tier if approaching limit
if ($metricValue -gt 8000) {
Update-AzCognitiveServicesAccount -ResourceGroupName $rgName `
-Name $deployment -Sku "S0" -Force
Write-Output "Scaled $deployment to S0 tier"
}
```
**Alert Rule Configuration:**
| Metric | Threshold | Action |
|--------|-----------|--------|
| `TokensPerMinute` | > 8000 (80% of 10K limit) | Invoke runbook `ScaleUpOpenAI` |
| `RequestLatency` | > 5000ms for 5 min | Invoke runbook `RestartDeployment` |
| `FailedRequests` | > 50 in 10 min | Send to Logic App for root cause analysis |
**Fordeler:**
- Reduserer Mean Time To Recovery (MTTR) dramatisk
- Fungerer 24/7 uten manuell inngripen
- Audit trail via Automation job logs
**Ulemper:**
- Runbooks må testes grundig (feil logic kan forverres situasjonen)
- Krever Automation Contributor role på ressursene
- Ikke egnet for komplekse diagnostiseringsscenarioer
---
### 3. Stateful vs. Stateless Alerting for AI Workloads
**Problem:** AI-requests kan generere tusenvis av failed requests ved samme rot-årsak (f.eks. model deployment down). Skal vi sende ett alert eller tusenvis?
**Stateful Alerting (anbefalt for AI):**
- **Enable:** `Automatically resolve alerts = true`
- **Behavior:** Ett alert fires når condition blir true, auto-resolves når condition blir false
- **Bruk når:** Infrastruktur-alerts (deployment down, API unavailable)
**Stateless Alerting:**
- **Enable:** `Automatically resolve alerts = false`
- **Behavior:** Nytt alert for hver evaluation cycle som matcher condition
- **Bruk når:** Per-request monitoring (track hver PII leak, hver toxic content response)
**Azure AI-spesifikk konfigurasjon:**
| Alert Rule | Type | Rationale |
|------------|------|-----------|
| `Azure OpenAI Deployment Unavailable` | Stateful | En deployment er enten oppe eller nede — send ett alert |
| `Prompt Injection Detected` | Stateless | Hver deteksjon skal logges individuelt (compliance) |
| `Content Safety Filter Triggered` | Stateless | Hver toxic response er en separat incident |
| `Token Rate Limit Approaching` | Stateful | Send warning når 80% nådd, resolve når < 70% |
---
## Beslutningsveiledning
### Severity Mapping for AI Incidents
| Severity | Definition | AI Examples | SLA | Escalation |
|----------|------------|-------------|-----|------------|
| **Sev 0** | Total service outage eller critical security breach | PII leak, AI Act violation, all models unavailable | < 15 min response | Tier 3 immediate |
| **Sev 1** | Major degradation affecting production workload | Primary model down, >50% error rate | < 30 min response | Tier 2 + manager notify |
| **Sev 2** | Partial degradation, workaround available | Secondary model down, latency >5s | < 2 hour response | Tier 2 |
| **Sev 3** | Minor issue, no user impact | Token costs 20% above budget | < 8 hour response | Tier 1 |
| **Sev 4** | Informational, proactive monitoring | Model drift detected, new version available | No SLA | Email only |
### Notification Channel Decision Tree
```
START: AI Alert Fired
│
├─ Is it Sev 0/1? ───YES──> SMS + Voice Call + Teams (immediate)
│ │
│ └─> Add webhook to PagerDuty/ServiceNow
│
└─ Is it Sev 2/3? ───YES──> Email + Teams channel
│
└─> Is it business hours? ───NO──> Add SMS for Sev 2
│
YES─> Email only
```
### Vanlige Feil (Red Flags)
| Anti-pattern | Problem | Anbefaling |
|--------------|---------|------------|
| **Sending all alerts to personal email** | Vacation/sickness = ingen response | Bruk distribution lists eller action groups per team |
| **No severity differentiation** | Alert fatigue — alt er "viktig" | Implementer 5-tier severity model |
| **No auto-escalation** | Critical alerts blir ignorert nattestid | Logic App med time-based escalation til manager |
| **Email-only for Sev 0** | Delays i critical situations | SMS + Voice Call for Sev 0/1 |
| **No actionable context** | Alerts sier "something is wrong" uten details | Custom properties med resource metadata, query results |
| **Alerting on every request failure** | Stateless alerts → spam | Bruk stateful alerts + aggregation windows (5-15 min) |
### Recommended Alert Rules for Azure AI Services
| Service | Metric/Log | Threshold | Action |
|---------|------------|-----------|--------|
| **Azure OpenAI** | `azure.openai.requests` (429 errors) | > 10 in 5 min | Scale up deployment tier |
| **Azure OpenAI** | `TokensPerMinute` | > 80% of quota | Email warning + runbook to request quota increase |
| **Azure AI Search** | `SearchLatency` | > 1000ms for 10 min | Check index size, scale up replicas |
| **Content Safety** | `ModeratedContent` (high severity) | Any occurrence | Stateless alert + SIEM integration |
| **Document Intelligence** | `FailedRequests` | > 20% error rate | Check API version compatibility, model availability |
---
## Integrasjon med Microsoft-stakken
### Azure Monitor ↔ Microsoft Sentinel
**Bruk:** Stream AI-alerts til Sentinel for correlation med security events.
**Konfigurasjon:**
1. **Action Group → Event Hub:**
```json
{
"eventHubReceiver": {
"name": "SentinelEventHub",
"subscriptionId": "...",
"eventHubNameSpace": "ai-monitoring",
"eventHubName": "alerts",
"useCommonAlertSchema": true
}
}
```
2. **Sentinel Data Connector:**
- Connect til Event Hub
- Parse Common Alert Schema
- Correlate med AuditLogs, SignInLogs for user context
**Fordeler:**
- Single pane of glass for security + operational monitoring
- Advanced threat detection (e.g., prompt injection patterns + user behavior anomalies)
- Compliance reporting (AI Act Article 72)
### Azure Monitor ↔ Logic Apps
**Bruk:** Enrich alerts med kontekstuell informasjon før notification.
**Eksempel Workflow:**
```
Alert: "Azure OpenAI High Error Rate" (Sev 2)
↓
Logic App receives webhook
↓
Query Log Analytics for last 100 error messages
↓
Group by error code (401, 429, 500)
↓
Fetch deployment tags (owner, cost center, environment)
↓
POST enriched alert til Teams:
"🔴 Sev 2: Azure OpenAI Errors
Deployment: gpt-4-prod
Owner: ai-platform-team@company.com
Top Errors: 429 (80%), 500 (15%), 401 (5%)
Cost Center: CC-12345
Environment: Production
Runbook: aka.ms/fix-429-errors"
```
**Template Actions:**
1. **HTTP (Get Model Metadata):** Call Azure OpenAI Management API for deployment details
2. **Log Analytics (Query Errors):** `AzureDiagnostics | where Category == "RequestResponse" and httpStatusCode_d >= 400`
3. **Teams (Post Adaptive Card):** Rich notification med buttons ("Acknowledge", "View Logs", "Run Remediation")
### Azure Monitor ↔ Azure Automation
**Bruk:** Auto-remediation for infrastruktur-alerts.
**Common Runbooks for AI:**
| Runbook | Trigger Alert | Action |
|---------|---------------|--------|
| `ScaleUpOpenAI` | TokensPerMinute > 80% | Update deployment tier (PTU → PTU-M) |
| `RestartFailedDeployment` | Health probe failed | Delete + redeploy model |
| `NotifyCompliance` | Content Safety violation | Email legal + log to compliance database |
| `RollbackModel` | Error rate > 50% after deployment | Swap to previous model version |
**Managed Identity Setup:**
```powershell
# Enable System-Assigned Managed Identity on Automation Account
Set-AzAutomationAccount -ResourceGroupName "rg-automation" `
-Name "ai-automation" -AssignSystemIdentity
# Assign Contributor role to Managed Identity
$automationAccount = Get-AzAutomationAccount -ResourceGroupName "rg-automation" -Name "ai-automation"
New-AzRoleAssignment -ObjectId $automationAccount.Identity.PrincipalId `
-RoleDefinitionName "Contributor" -Scope "/subscriptions/.../resourceGroups/rg-ai"
```
---
## Offentlig sektor (Norge)
### AI Act Article 72: Incident Reporting
EU AI Act krever at providers rapporterer "serious incidents" til nasjonale myndigheter innen **15 dager**. Azure Monitor alerts må derfor konfigureres med compliance logging.
**Serious Incident Definition (AI Act):**
- Death or serious injury
- Serious harm to health, property, or environment
- Serious violation of fundamental rights (e.g., discrimination)
**Implementering:**
1. **Tag Critical Alerts:**
```json
{
"customProperties": {
"aiActReportable": "true",
"incidentType": "discriminationRisk",
"affectedUsers": "approx. 500",
"dataProcessed": "PII (names, addresses)"
}
}
```
2. **Action Group → Event Hub → Archive Storage:**
- Stream til immutable blob storage (compliance retention)
- Hourly export til Sentinel for analysis
- Monthly report generation (Logic App)
3. **Notification til Compliance Officer:**
- Sev 0 alerts → immediate email til DPO + legal
- Include pre-filled incident report template
### Forvaltningsloven § 25 (Begrunnelsesplikt)
Vedtak fattet med AI-støtte må kunne forklares. Hvis AI-modellen feiler under saksbehandling, må dette logges og eskaleres.
**Alert Rule:** "AI Recommendation Unavailable During Case Processing"
**Action:**
1. **Immediate:** Email til saksbehandler (manual fallback)
2. **Within 1 hour:** Notify IT support
3. **Within 4 hours:** Incident report til seksjonsleder
4. **Audit log:** Store case ID, timestamp, error message (for later review)
### Schrems II / Data Residency
Alerts som inneholder PII må **ikke** sendes til tjenester utenfor EU/EØS. Dette gjelder spesielt webhooks til SaaS-løsninger (PagerDuty, Slack).
**Compliant Setup:**
| Notification Channel | Data Residency | Compliant? | Alternative |
|---------------------|----------------|------------|-------------|
| Email (Microsoft 365 EU tenant) | EU | ✅ Yes | — |
| Teams (EU datacenter) | EU | ✅ Yes | — |
| Event Hub → Sentinel (Norway East) | Norway | ✅ Yes | — |
| Webhook → PagerDuty (US) | USA | ❌ No | Bruk Logic App i Norway East som proxy, strip PII |
| SMS (Twilio US) | USA | ❌ No | Bruk Azure Communication Services (EU) |
**Best Practice:** Bruk `customProperties` til å skille mellom metadata (OK å sende ut) og PII (må holdes innenfor EU).
---
## Kostnad og lisensiering
### Azure Monitor Alerts Pricing (Norway East, Feb 2026)
| Alert Type | Price per Rule/Month | Price per Evaluation | Notes |
|------------|---------------------|---------------------|-------|
| **Metric Alert** (standard) | 0.10 USD | — | First 10 rules free per subscription |
| **Metric Alert** (multi-resource) | 0.10 USD | — | Can monitor 1000+ VMs with one rule |
| **Log Search Alert** | 0.10 USD | 0.20 USD per query execution | Frequency × time window = cost |
| **Activity Log Alert** | **FREE** | **FREE** | Use these whenever possible! |
| **Service Health Alert** | **FREE** | **FREE** | — |
| **Resource Health Alert** | **FREE** | **FREE** | — |
**Example Cost Calculation (Log Search Alert):**
```
Alert: "Azure OpenAI Error Rate > 10%"
Query frequency: Every 5 minutes
Time window: 15 minutes
Evaluations per month: (60/5) × 24 × 30 = 8640
Cost = 0.10 USD (rule) + (8640 × 0.20 USD) = 1728.10 USD/month
```
**Optimization Strategy:**
- Bruk **metric alerts** i stedet for log search alerts der mulig (gratis evaluations)
- Bruk **activity log alerts** for administrative events (gratis)
- Bruk **multi-resource alert rules** (én rule for mange ressurser)
- Øk query frequency til 15-30 min for non-critical alerts
### Action Group Pricing
| Action Type | Cost | Rate Limit |
|-------------|------|------------|
| **Email** | FREE | 1000 emails per hour per action group |
| **SMS** | 0.20 USD per SMS | 1 SMS per 5 min per phone number |
| **Voice Call** | 1.00 USD per call | 1 call per 5 min per phone number |
| **Webhook** | FREE | — |
| **Automation Runbook** | Automation job cost (0.002 USD per minute) | — |
| **Logic App** | Logic App execution cost (varies) | — |
| **Event Hub** | Event Hub ingress cost (0.028 USD per million events) | — |
**Best Practice:** Start med email + webhook (free), legg til SMS/voice call kun for Sev 0/1.
---
## For arkitekten (Cosmo)
### Spørsmål å stille kunden
1. **Severity Mapping:**
- "Hva definerer dere som en Sev 0 incident for deres AI-løsning? PII leak? Total nedetid? Noe annet?"
- "Hva er akseptabel Mean Time To Acknowledge (MTTA) per severity level?"
2. **Escalation Procedures:**
- "Har dere en on-call rotation? Hvem skal motta SMS/voice call ved nattestid for Sev 0/1?"
- "Skal management (seksjonsleder, DPO, juridisk) varsles automatisk ved visse typer alerts?"
3. **Compliance Requirements:**
- "Er løsningen omfattet av AI Act som high-risk system? Må dere rapportere serious incidents til myndighetene?"
- "Hvilke data residency-krav har dere? Er det OK å sende alerts til webhooks utenfor EU/EØS?"
4. **Automation vs. Manual Response:**
- "Er det failure modes hvor dere ønsker automatisk remediation (scale-up, restart)? Hva er risikoen ved feil automation?"
- "Hvilke alerts krever manuell triaging før action (f.eks. model rollback)?"
5. **Integration Points:**
- "Bruker dere ITSM-system (ServiceNow, Cherwell)? Skal alerts automatisk opprette incidents?"
- "Skal alerts streames til Sentinel for security correlation? Til Power BI for dashboards?"
6. **Alert Fatigue:**
- "Hvor mange alerts får dere per dag i dag? Hvor mange av dem er actionable?"
- "Er det alerts dere ignorerer fordi de 'alltid fyrer'? Hvordan kan vi redusere false positives?"
7. **Testing & Validation:**
- "Hvordan skal vi teste eskaleringsrutinene før go-live? Ønsker dere en tabletop exercise?"
- "Hva er akseptabel alert latency (tid fra incident → alert fires)? 1 min? 5 min?"
### Fallgruver å unngå
| Fallgruve | Konsekvens | Mitigering |
|-----------|------------|------------|
| **Alert spam (100+ alerts per dag)** | Team ignorer alle alerts | Bruk stateful alerts, øk aggregation windows, implementer alert processing rules |
| **No clear ownership** | Alerts går til "no-reply" inbox som ingen sjekker | Definer action groups per team/severity, bruk distribution lists |
| **Over-reliance på automation** | Runbook scaler opp feil ressurs → cost explosion | Start med manual approval workflows, test automation grundig |
| **PII i alert payload** | GDPR violation når sendt til external webhooks | Bruk `customProperties` for metadata only, strip PII i Logic App proxy |
| **No escalation for unacknowledged alerts** | Sev 0 alerts blir ikke sett nattestid | Implementer time-based escalation via Logic App |
| **Webhook endpoints without retry logic** | Alerts tapt hvis endpoint er midlertidig nede | Bruk Event Hub som buffer + reliable delivery |
| **Cost blindness** | Log search alerts med 1-min frequency → 1000+ USD/month | Bruk metric alerts der mulig, øk query frequency til 5-15 min |
### Anbefalinger per modenhetsnivå
**Level 1 (MVP / Pilot):**
- 1 action group med email til dev-team
- Metric alerts for kritiske metrics (availability, error rate)
- Stateful alerting for infrastruktur-events
- Severity: Kun Sev 1/2/3 (simplifisert)
**Level 2 (Production / Basic Maturity):**
- Multi-tier escalation (developer → platform team → manager)
- SMS + voice call for Sev 0/1
- Integration med Teams for collaborative triaging
- Alert processing rules for maintenance windows
- Automation runbooks for simple remediation (scale-up)
**Level 3 (High Maturity / Regulated):**
- Full 5-tier severity model med SLA per level
- ITSM integration (auto-create ServiceNow incidents)
- Sentinel integration for security correlation
- Compliance logging (AI Act incident reporting)
- Advanced automation (model rollback, canary deployments)
- Quarterly alert review + optimization (reduce alert fatigue)
**Level 4 (Best-in-Class / Autonomous):**
- AI-powered alert correlation (AIOps)
- Predictive alerting (model drift detected before user impact)
- Closed-loop remediation (auto-resolve 80%+ of alerts)
- Real-time cost optimization (auto-scale based on budget vs. demand)
- Continuous compliance monitoring (automated AI Act Article 72 reporting)
---
## Kilder og verifisering
### Verified (from Microsoft Learn MCP)
1. **Action Groups Overview**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups
*Confidence: High — Official documentation retrieved 2026-02, covers notification types, managed identity, rate limits.*
2. **Best Practices for Azure Monitor Alerts**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/best-practices-alerts
*Confidence: High — Well-Architected Framework guidance, includes reliability, cost optimization, operational excellence.*
3. **Webhook Retry Logic**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups#webhook
*Confidence: High — Documented retry intervals (5s, 20s, 5s, 40s, 5s) and 15-min cooldown.*
4. **Automation Runbook with Managed Identity**
https://learn.microsoft.com/en-us/azure/automation/automation-create-alert-triggered-runbook
*Confidence: High — Code sample for VM stop runbook using Common Alert Schema.*
5. **Alert Processing Rules**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-processing-rules
*Confidence: High — Covers suppression, action group override, scheduling.*
6. **Stateful vs. Stateless Alerts**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-troubleshoot-metric
*Confidence: High — "Automatically resolve alerts" checkbox behavior explained.*
7. **Service Limits for Notifications**
https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups#service-limits-for-notifications
*Confidence: High — Rate limits per notification type (SMS, voice, email).*
### Baseline (Model Knowledge)
8. **AI Act Article 72 Incident Reporting**
*Confidence: Medium — EU AI Act text available, but specific implementation guidance for Azure not yet published by Microsoft (as of Feb 2026).*
9. **Severity Mapping Best Practices**
*Confidence: Medium — Industry standard pattern (Sev 0-4), adapted for AI-specific scenarios based on architecture experience.*
10. **Multi-Tier Escalation Pattern**
*Confidence: High — Standard ITIL/SRE practice, Azure Monitor supports via action groups + Logic Apps.*
### Recommendations for Further Verification
- **Cost estimates:** Verify against Azure Pricing Calculator (pricing kan variere per region og currency fluctuations).
- **AI Act compliance:** Consult with legal team og Datatilsynet for norsk implementering av EU AI Act Article 72.
- **ITSM integration:** Test ITSM connector med deres spesifikke ServiceNow/Cherwell-versjon (API compatibility kan variere).