ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/alerting-strategies-escalation.md

# Alerting Strategies and Escalation Policies for AI Incidents

**Last updated:** 2026-05
**Status:** GA
**Category:** Monitoring & Observability

---

## Introduksjon

Effektive alerting-strategier og eskaleringsrutiner er kritiske for å sikre rask respons på AI-relaterte hendelser. I motsetning til tradisjonelle applikasjoner introduserer AI-systemer unike utfordringer: modeller kan degradere over tid (drift), prompt injection-angrep kan oppstå plutselig, og token-kostnader kan eksplodere uten varsel. En robust alerting-arkitektur må derfor kombinere reaktive varsler (noe gikk galt) med proaktive varsler (noe er i ferd med å gå galt).

Azure Monitor gir et omfattende rammeverk for alerting gjennom action groups, alert processing rules, og integrasjon med Azure Automation, Logic Apps, og ITSM-systemer. For AI-løsninger må denne infrastrukturen konfigureres med forståelse av både business impact og teknisk kompleksitet — en kritisk alert kan være en modell som returnerer bias-innhold, eller en Azure OpenAI-deployment som nærmer seg rate limit.

Eskaleringsrutiner må reflektere organisasjonens modenhetsnivå. En Minimum Viable Product (MVP)-løsning kan starte med e-postvarsler til et lite team, mens en produksjonsløsning for offentlig sektor krever Multi-tier escalation med clear ownership, automated incident response, og compliance logging for AI Act Article 72 (incident reporting).

---

## Kjernekomponenter

### Azure Monitor Alert Architecture

| Komponent | Beskrivelse | AI-relevans |
|-----------|-------------|-------------|
| **Alert Rules** | Definerer betingelser som trigger alerts (metrics, logs, activity log) | Token rate limits, model latency, failed requests |
| **Action Groups** | Samling av notifications og actions som kjøres når alert fires | Email, SMS, webhook, runbook, Logic App |
| **Alert Processing Rules** | Overstyr eller berik alert-oppførsel (suppression, tillegg av action groups, scheduling). Filtre støtter opptil 5 verdier per filter (logisk OR). Suppression har høyere prioritet enn Apply action groups. | Prevent alert fatigue under model redeployments; recurring maintenance windows *(Verified MCP 2026-04)* |
| **Common Alert Schema** | Uniform JSON payload på tvers av alle alert-typer | Forenkler webhook-integrasjoner og ITSM-connectors |
| **Severity Levels** | Sev 0 (Critical) til Sev 4 (Informational) | Map til business impact (Sev 0 = PII leak, Sev 4 = latency spike) |

### Notification Channels

| Channel | Bruksområde | Rate Limits | Global Resilience |
|---------|-------------|-------------|-------------------|
| **Email** | Standard notification (opptil 1000 mottakere per action group) | Best practices: Ikke send til personlige adresser, bruk distribution lists | ✅ Yes |
| **SMS** | Kritiske alerts (begrensede land) | 1 SMS per 5 min per nummer | ✅ Yes |
| **Voice Call** | Sev 0 incidents (natt/helg) | 1 call per 5 min per nummer | ✅ Yes |
| **Webhook** | Integration med eksterne systemer (PagerDuty, Slack) | Retry: 5 retries med 5-40s delay | ❌ Endpoint-dependent |
| **Azure App Push** | Mobile notifications til Azure-appen | Begrenset til Azure mobile app | ✅ Yes |
| **Event Hub** | Stream alerts til analytics/SIEM | Supports Private Link og NSP | ✅ Yes (regional) |

**Reusability:** Du kan legge til opptil 5 action groups per alert rule. Action groups kjøres concurrent (ingen bestemt rekkefølge). Multiple alert rules kan bruke samme action group. *(Verified MCP 2026-04)*

**Rate Limiting:** Azure Monitor rate-limiter notifications for å hindre spam. Hvis samme email/SMS/telefonnummer mottar for mange alerts, suspenderes notifications midlertidig. For AI-systemer som kan generere høy alert-volum (f.eks. per-request failures), bruk programmatic actions (Logic Apps, Automation Runbooks) i stedet.

### Action Types for AI Incidents

```json
{
  "actionType": "AutomationRunbook",
  "runbookName": "ScaleDownOpenAI",
  "webhookResourceId": "/subscriptions/.../runbooktest/webhooks/Alert...",
  "useCommonAlertSchema": true,
  "isGlobalRunbook": false
}
```

| Action Type | AI Use Case | Authentication | Managed Identity | Cross-tenant Support |
|-------------|-------------|----------------|-----------------|---------------------|
| **Automation Runbook** | Auto-scale Azure OpenAI TPM, restart failing deployments | Managed Identity (Automation Contributor, Role ID: f353d9bd) | ✅ Yes | ❌ No |
| **Logic App** | Enrich alert med model metadata, post til Teams/Slack | Managed Identity (Logic App Contributor, Role ID: 87a39d53) | ✅ Yes | ❌ No |
| **Azure Function** | Custom logic (e.g., invoke model rollback API) | HTTP trigger med access key | ❌ No | ❌ No |
| **Webhook** | Invoke external incident mgmt (PagerDuty, ServiceNow) | Basic auth via URI eller secure webhook (Entra ID) | ❌ No | ✅ Yes (limited) |
| **Event Hub** | Stream til SIEM (Microsoft Sentinel) for correlation | Managed Identity (Event Hubs Data Sender, Role ID: 2b629674) | ✅ Yes | ✅ Yes (up to API 2023-09-01-preview) |
| **Secure Webhook** | Invoke protected API med Entra ID-auth | Microsoft Entra app registration | ❌ No | ✅ Yes |
| **ITSM Connector** | Create incidents i ServiceNow, Cherwell | ITSM connection credentials | ❌ No | ❌ No |

*(Verified MCP 2026-04)*

**Managed Identity Best Practice (preview):** Managed Identity-støtte for Action Groups er nå tilgjengelig i **preview**. For Automation Runbooks, Logic Apps og Event Hubs, bruk managed identity i stedet for service principals. Azure Function, Webhook, Secure Webhook og ITSM støtter ikke managed identity — bruk HTTP trigger access key respektive secure webhook med Entra ID. Azure Portal legger automatisk til role assignments ved konfigurasjon. For PowerShell/CLI/SDK må du manuelt tildele roller (se tabell over). *(Verified MCP 2026-04)*

---

## Arkitekturmønstre

### 1. Multi-Tier Escalation for AI Incidents

**Bruk når:** Produksjonsløsninger med SLA-krav og 24/7 support.

**Implementering:**

```plaintext
Tier 1: On-Call Developer (Email + SMS)
  ├─ Sev 3-4 alerts → Respond within 4 hours
  └─ Auto-escalate to Tier 2 if no ACK within 30 min

Tier 2: AI Platform Team (Voice Call + PagerDuty)
  ├─ Sev 1-2 alerts → Respond within 30 min
  └─ Auto-escalate to Tier 3 if no resolution within 2 hours

Tier 3: Management + Legal (Email + Teams)
  └─ Sev 0 alerts → Data breach, AI Act violation, PII leak
```

**Azure Monitor Implementering:**

1. **Action Group per Tier:**
   - `AG-Tier1-Developers`: Email til dev-team distribution list
   - `AG-Tier2-Platform`: SMS + PagerDuty webhook
   - `AG-Tier3-Executive`: Voice call til on-call manager + Teams notification

2. **Alert Processing Rule for Auto-Escalation:**
   ```json
   {
     "rules": {
       "if": "alert.severity == 0 AND alert.state == 'New' FOR 30 minutes",
       "then": "add action group AG-Tier3-Executive"
     }
   }
   ```

3. **Time-Based Escalation (via Logic App):**
   - Webhook til Logic App som sjekker alert timestamp
   - Hvis ikke acknowledged innen threshold → invoke Tier 2/3 action groups

**Fordeler:**
- Clear ownership per severity level
- Reduserer alert fatigue for Tier 3
- Automatisk eskalering hindrer at kritiske alerts "faller mellom stolene"

**Ulemper:**
- Kompleks konfigurasjon (krever Logic Apps for time-based escalation)
- Krever testing og dokumentasjon av eskaleringsrutiner
- Risiko for "false escalations" hvis thresholds er feil satt

---

### 2. Automated Remediation with Runbooks

**Bruk når:** Kjente failure modes med deterministiske fix-prosedyrer (scale-out, restart, rollback).

**Eksempel:** Azure OpenAI deployment nærmer seg TPM limit → Auto-scale til høyere tier.

**Runbook Template (PowerShell 7):**

```powershell
param(
    [object] $WebhookData
)

# Parse Common Alert Schema
$alertData = (ConvertFrom-Json -InputObject $WebhookData.RequestBody)
$resourceId = $alertData.data.essentials.alertTargetIds[0]
$metricValue = $alertData.data.alertContext.condition.allOf[0].metricValue

# Extract OpenAI deployment info
$deployment = $resourceId -split '/' | Select-Object -Last 1
$rgName = ($resourceId -split '/')[4]

# Scale up to Standard tier if approaching limit
if ($metricValue -gt 8000) {
    Update-AzCognitiveServicesAccount -ResourceGroupName $rgName `
        -Name $deployment -Sku "S0" -Force
    Write-Output "Scaled $deployment to S0 tier"
}
```

**Alert Rule Configuration:**

| Metric | Threshold | Action |
|--------|-----------|--------|
| `TokensPerMinute` | > 8000 (80% of 10K limit) | Invoke runbook `ScaleUpOpenAI` |
| `RequestLatency` | > 5000ms for 5 min | Invoke runbook `RestartDeployment` |
| `FailedRequests` | > 50 in 10 min | Send to Logic App for root cause analysis |

**Fordeler:**
- Reduserer Mean Time To Recovery (MTTR) dramatisk
- Fungerer 24/7 uten manuell inngripen
- Audit trail via Automation job logs

**Ulemper:**
- Runbooks må testes grundig (feil logic kan forverres situasjonen)
- Krever Automation Contributor role på ressursene
- Ikke egnet for komplekse diagnostiseringsscenarioer

---

### 3. Stateful vs. Stateless Alerting for AI Workloads

**Problem:** AI-requests kan generere tusenvis av failed requests ved samme rot-årsak (f.eks. model deployment down). Skal vi sende ett alert eller tusenvis?

**Stateful Alerting (anbefalt for AI):**

- **Enable:** `Automatically resolve alerts = true`
- **Behavior:** Ett alert fires når condition blir true, auto-resolves når condition blir false
- **Bruk når:** Infrastruktur-alerts (deployment down, API unavailable)

**Stateless Alerting:**

- **Enable:** `Automatically resolve alerts = false`
- **Behavior:** Nytt alert for hver evaluation cycle som matcher condition
- **Bruk når:** Per-request monitoring (track hver PII leak, hver toxic content response)

**Azure AI-spesifikk konfigurasjon:**

| Alert Rule | Type | Rationale |
|------------|------|-----------|
| `Azure OpenAI Deployment Unavailable` | Stateful | En deployment er enten oppe eller nede — send ett alert |
| `Prompt Injection Detected` | Stateless | Hver deteksjon skal logges individuelt (compliance) |
| `Content Safety Filter Triggered` | Stateless | Hver toxic response er en separat incident |
| `Token Rate Limit Approaching` | Stateful | Send warning når 80% nådd, resolve når < 70% |

---

## Beslutningsveiledning

### Severity Mapping for AI Incidents

| Severity | Definition | AI Examples | SLA | Escalation |
|----------|------------|-------------|-----|------------|
| **Sev 0** | Total service outage eller critical security breach | PII leak, AI Act violation, all models unavailable | < 15 min response | Tier 3 immediate |
| **Sev 1** | Major degradation affecting production workload | Primary model down, >50% error rate | < 30 min response | Tier 2 + manager notify |
| **Sev 2** | Partial degradation, workaround available | Secondary model down, latency >5s | < 2 hour response | Tier 2 |
| **Sev 3** | Minor issue, no user impact | Token costs 20% above budget | < 8 hour response | Tier 1 |
| **Sev 4** | Informational, proactive monitoring | Model drift detected, new version available | No SLA | Email only |

### Notification Channel Decision Tree

```
START: AI Alert Fired
  │
  ├─ Is it Sev 0/1? ───YES──> SMS + Voice Call + Teams (immediate)
  │                     │
  │                     └─> Add webhook to PagerDuty/ServiceNow
  │
  └─ Is it Sev 2/3? ───YES──> Email + Teams channel
      │
      └─> Is it business hours? ───NO──> Add SMS for Sev 2
                                   │
                                   YES─> Email only
```

### Anbefalinger fra Well-Architected Framework

*(Verified MCP 2026-04)*

| Anbefaling | Fordel |
|------------|--------|
| **Bruk dynamic thresholds i metric alerts** | ML-baserte terskler tilpasser seg automatisk — unngår manuell threshold-tuning for AI-workloads med variabelt load |
| **Bruk managed identity for log search alert rules** | Credential-free auth; påkrevd hvis query aksesserer Azure Data Explorer (ADX) eller Azure Resource Graph (ARG) |
| **Bruk custom properties i alert rules** | Legg til metadata (owner, cost center, AI Act reportable) i alert payload — enklere triaging og compliance-logging |
| **Bruk Logic Apps for enrichment** | Customiser notification-format, hent tags fra ressurser, integrer med Teams/Slack/PagerDuty uten å endre alert rule |
| **Activity log alerts er gratis** | Bruk activity log alerts for administrative hendelser der mulig — ingen kostnad per evaluering |

### Vanlige Feil (Red Flags)

| Anti-pattern | Problem | Anbefaling |
|--------------|---------|------------|
| **Sending all alerts to personal email** | Vacation/sickness = ingen response | Bruk distribution lists eller action groups per team |
| **No severity differentiation** | Alert fatigue — alt er "viktig" | Implementer 5-tier severity model |
| **No auto-escalation** | Critical alerts blir ignorert nattestid | Logic App med time-based escalation til manager |
| **Email-only for Sev 0** | Delays i critical situations | SMS + Voice Call for Sev 0/1 |
| **No actionable context** | Alerts sier "something is wrong" uten details | Custom properties med resource metadata, query results |
| **Alerting on every request failure** | Stateless alerts → spam | Bruk stateful alerts + aggregation windows (5-15 min) |

### Recommended Alert Rules for Azure AI Services

| Service | Metric/Log | Threshold | Action |
|---------|------------|-----------|--------|
| **Azure OpenAI** | `azure.openai.requests` (429 errors) | > 10 in 5 min | Scale up deployment tier |
| **Azure OpenAI** | `TokensPerMinute` | > 80% of quota | Email warning + runbook to request quota increase |
| **Azure AI Search** | `SearchLatency` | > 1000ms for 10 min | Check index size, scale up replicas |
| **Content Safety** | `ModeratedContent` (high severity) | Any occurrence | Stateless alert + SIEM integration |
| **Document Intelligence** | `FailedRequests` | > 20% error rate | Check API version compatibility, model availability |

---

## Integrasjon med Microsoft-stakken

### Azure Monitor ↔ Microsoft Sentinel

**Bruk:** Stream AI-alerts til Sentinel for correlation med security events.

**Konfigurasjon:**

1. **Action Group → Event Hub:**
   ```json
   {
     "eventHubReceiver": {
       "name": "SentinelEventHub",
       "subscriptionId": "...",
       "eventHubNameSpace": "ai-monitoring",
       "eventHubName": "alerts",
       "useCommonAlertSchema": true
     }
   }
   ```

2. **Sentinel Data Connector:**
   - Connect til Event Hub
   - Parse Common Alert Schema
   - Correlate med AuditLogs, SignInLogs for user context

**Fordeler:**
- Single pane of glass for security + operational monitoring
- Advanced threat detection (e.g., prompt injection patterns + user behavior anomalies)
- Compliance reporting (AI Act Article 72)

### Azure Monitor ↔ Logic Apps

**Bruk:** Enrich alerts med kontekstuell informasjon før notification.

**Eksempel Workflow:**

```
Alert: "Azure OpenAI High Error Rate" (Sev 2)
  ↓
Logic App receives webhook
  ↓
Query Log Analytics for last 100 error messages
  ↓
Group by error code (401, 429, 500)
  ↓
Fetch deployment tags (owner, cost center, environment)
  ↓
POST enriched alert til Teams:
  "🔴 Sev 2: Azure OpenAI Errors
   Deployment: gpt-4-prod
   Owner: ai-platform-team@company.com
   Top Errors: 429 (80%), 500 (15%), 401 (5%)
   Cost Center: CC-12345
   Environment: Production
   Runbook: aka.ms/fix-429-errors"
```

**Template Actions:**

1. **HTTP (Get Model Metadata):** Call Azure OpenAI Management API for deployment details
2. **Log Analytics (Query Errors):** `AzureDiagnostics | where Category == "RequestResponse" and httpStatusCode_d >= 400`
3. **Teams (Post Adaptive Card):** Rich notification med buttons ("Acknowledge", "View Logs", "Run Remediation")

### Azure Monitor ↔ Azure Automation

**Bruk:** Auto-remediation for infrastruktur-alerts.

**Common Runbooks for AI:**

| Runbook | Trigger Alert | Action |
|---------|---------------|--------|
| `ScaleUpOpenAI` | TokensPerMinute > 80% | Update deployment tier (PTU → PTU-M) |
| `RestartFailedDeployment` | Health probe failed | Delete + redeploy model |
| `NotifyCompliance` | Content Safety violation | Email legal + log to compliance database |
| `RollbackModel` | Error rate > 50% after deployment | Swap to previous model version |

**Managed Identity Setup:**

```powershell
# Enable System-Assigned Managed Identity on Automation Account
Set-AzAutomationAccount -ResourceGroupName "rg-automation" `
    -Name "ai-automation" -AssignSystemIdentity

# Assign Contributor role to Managed Identity
$automationAccount = Get-AzAutomationAccount -ResourceGroupName "rg-automation" -Name "ai-automation"
New-AzRoleAssignment -ObjectId $automationAccount.Identity.PrincipalId `
    -RoleDefinitionName "Contributor" -Scope "/subscriptions/.../resourceGroups/rg-ai"
```

---

## Offentlig sektor (Norge)

### AI Act Article 72: Incident Reporting

EU AI Act krever at providers rapporterer "serious incidents" til nasjonale myndigheter innen **15 dager**. Azure Monitor alerts må derfor konfigureres med compliance logging.

**Serious Incident Definition (AI Act):**
- Death or serious injury
- Serious harm to health, property, or environment
- Serious violation of fundamental rights (e.g., discrimination)

**Implementering:**

1. **Tag Critical Alerts:**
   ```json
   {
     "customProperties": {
       "aiActReportable": "true",
       "incidentType": "discriminationRisk",
       "affectedUsers": "approx. 500",
       "dataProcessed": "PII (names, addresses)"
     }
   }
   ```

2. **Action Group → Event Hub → Archive Storage:**
   - Stream til immutable blob storage (compliance retention)
   - Hourly export til Sentinel for analysis
   - Monthly report generation (Logic App)

3. **Notification til Compliance Officer:**
   - Sev 0 alerts → immediate email til DPO + legal
   - Include pre-filled incident report template

### Forvaltningsloven § 25 (Begrunnelsesplikt)

Vedtak fattet med AI-støtte må kunne forklares. Hvis AI-modellen feiler under saksbehandling, må dette logges og eskaleres.

**Alert Rule:** "AI Recommendation Unavailable During Case Processing"

**Action:**
1. **Immediate:** Email til saksbehandler (manual fallback)
2. **Within 1 hour:** Notify IT support
3. **Within 4 hours:** Incident report til seksjonsleder
4. **Audit log:** Store case ID, timestamp, error message (for later review)

### Schrems II / Data Residency

Alerts som inneholder PII må **ikke** sendes til tjenester utenfor EU/EØS. Dette gjelder spesielt webhooks til SaaS-løsninger (PagerDuty, Slack).

**Compliant Setup:**

| Notification Channel | Data Residency | Compliant? | Alternative |
|---------------------|----------------|------------|-------------|
| Email (Microsoft 365 EU tenant) | EU | ✅ Yes | — |
| Teams (EU datacenter) | EU | ✅ Yes | — |
| Event Hub → Sentinel (Norway East) | Norway | ✅ Yes | — |
| Webhook → PagerDuty (US) | USA | ❌ No | Bruk Logic App i Norway East som proxy, strip PII |
| SMS (Twilio US) | USA | ❌ No | Bruk Azure Communication Services (EU) |

**Best Practice:** Bruk `customProperties` til å skille mellom metadata (OK å sende ut) og PII (må holdes innenfor EU).

---

## Kostnad og lisensiering

### Azure Monitor Alerts Pricing (Norway East, Feb 2026)

| Alert Type | Price per Rule/Month | Price per Evaluation | Notes |
|------------|---------------------|---------------------|-------|
| **Metric Alert** (standard) | 0.10 USD | — | First 10 rules free per subscription |
| **Metric Alert** (multi-resource) | 0.10 USD | — | Can monitor 1000+ VMs with one rule |
| **Log Search Alert** | 0.10 USD | 0.20 USD per query execution | Frequency × time window = cost |
| **Activity Log Alert** | **FREE** | **FREE** | Use these whenever possible! |
| **Service Health Alert** | **FREE** | **FREE** | — |
| **Resource Health Alert** | **FREE** | **FREE** | — |

**Example Cost Calculation (Log Search Alert):**

```
Alert: "Azure OpenAI Error Rate > 10%"
Query frequency: Every 5 minutes
Time window: 15 minutes
Evaluations per month: (60/5) × 24 × 30 = 8640

Cost = 0.10 USD (rule) + (8640 × 0.20 USD) = 1728.10 USD/month
```

**Optimization Strategy:**
- Bruk **metric alerts** i stedet for log search alerts der mulig (gratis evaluations)
- Bruk **activity log alerts** for administrative events (gratis)
- Bruk **multi-resource alert rules** (én rule for mange ressurser)
- Øk query frequency til 15-30 min for non-critical alerts

### Action Group Pricing

| Action Type | Cost | Rate Limit |
|-------------|------|------------|
| **Email** | FREE | 1000 emails per hour per action group |
| **SMS** | 0.20 USD per SMS | 1 SMS per 5 min per phone number |
| **Voice Call** | 1.00 USD per call | 1 call per 5 min per phone number |
| **Webhook** | FREE | — |
| **Automation Runbook** | Automation job cost (0.002 USD per minute) | — |
| **Logic App** | Logic App execution cost (varies) | — |
| **Event Hub** | Event Hub ingress cost (0.028 USD per million events) | — |

**Best Practice:** Start med email + webhook (free), legg til SMS/voice call kun for Sev 0/1.

---

## For arkitekten (Cosmo)

### Spørsmål å stille kunden

1. **Severity Mapping:**
   - "Hva definerer dere som en Sev 0 incident for deres AI-løsning? PII leak? Total nedetid? Noe annet?"
   - "Hva er akseptabel Mean Time To Acknowledge (MTTA) per severity level?"

2. **Escalation Procedures:**
   - "Har dere en on-call rotation? Hvem skal motta SMS/voice call ved nattestid for Sev 0/1?"
   - "Skal management (seksjonsleder, DPO, juridisk) varsles automatisk ved visse typer alerts?"

3. **Compliance Requirements:**
   - "Er løsningen omfattet av AI Act som high-risk system? Må dere rapportere serious incidents til myndighetene?"
   - "Hvilke data residency-krav har dere? Er det OK å sende alerts til webhooks utenfor EU/EØS?"

4. **Automation vs. Manual Response:**
   - "Er det failure modes hvor dere ønsker automatisk remediation (scale-up, restart)? Hva er risikoen ved feil automation?"
   - "Hvilke alerts krever manuell triaging før action (f.eks. model rollback)?"

5. **Integration Points:**
   - "Bruker dere ITSM-system (ServiceNow, Cherwell)? Skal alerts automatisk opprette incidents?"
   - "Skal alerts streames til Sentinel for security correlation? Til Power BI for dashboards?"

6. **Alert Fatigue:**
   - "Hvor mange alerts får dere per dag i dag? Hvor mange av dem er actionable?"
   - "Er det alerts dere ignorerer fordi de 'alltid fyrer'? Hvordan kan vi redusere false positives?"

7. **Testing & Validation:**
   - "Hvordan skal vi teste eskaleringsrutinene før go-live? Ønsker dere en tabletop exercise?"
   - "Hva er akseptabel alert latency (tid fra incident → alert fires)? 1 min? 5 min?"

### Fallgruver å unngå

| Fallgruve | Konsekvens | Mitigering |
|-----------|------------|------------|
| **Alert spam (100+ alerts per dag)** | Team ignorer alle alerts | Bruk stateful alerts, øk aggregation windows, implementer alert processing rules |
| **No clear ownership** | Alerts går til "no-reply" inbox som ingen sjekker | Definer action groups per team/severity, bruk distribution lists |
| **Over-reliance på automation** | Runbook scaler opp feil ressurs → cost explosion | Start med manual approval workflows, test automation grundig |
| **PII i alert payload** | GDPR violation når sendt til external webhooks | Bruk `customProperties` for metadata only, strip PII i Logic App proxy |
| **No escalation for unacknowledged alerts** | Sev 0 alerts blir ikke sett nattestid | Implementer time-based escalation via Logic App |
| **Webhook endpoints without retry logic** | Alerts tapt hvis endpoint er midlertidig nede | Bruk Event Hub som buffer + reliable delivery |
| **Cost blindness** | Log search alerts med 1-min frequency → 1000+ USD/month | Bruk metric alerts der mulig, øk query frequency til 5-15 min |

### Anbefalinger per modenhetsnivå

**Level 1 (MVP / Pilot):**
- 1 action group med email til dev-team
- Metric alerts for kritiske metrics (availability, error rate)
- Stateful alerting for infrastruktur-events
- Severity: Kun Sev 1/2/3 (simplifisert)

**Level 2 (Production / Basic Maturity):**
- Multi-tier escalation (developer → platform team → manager)
- SMS + voice call for Sev 0/1
- Integration med Teams for collaborative triaging
- Alert processing rules for maintenance windows
- Automation runbooks for simple remediation (scale-up)

**Level 3 (High Maturity / Regulated):**
- Full 5-tier severity model med SLA per level
- ITSM integration (auto-create ServiceNow incidents)
- Sentinel integration for security correlation
- Compliance logging (AI Act incident reporting)
- Advanced automation (model rollback, canary deployments)
- Quarterly alert review + optimization (reduce alert fatigue)

**Level 4 (Best-in-Class / Autonomous):**
- AI-powered alert correlation (AIOps)
- Predictive alerting (model drift detected before user impact)
- Closed-loop remediation (auto-resolve 80%+ of alerts)
- Real-time cost optimization (auto-scale based on budget vs. demand)
- Continuous compliance monitoring (automated AI Act Article 72 reporting)

---

## Kilder og verifisering

### Verified (from Microsoft Learn MCP)

1. **Action Groups Overview**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups
   *Confidence: High — Official documentation retrieved 2026-02, covers notification types, managed identity, rate limits.*

2. **Best Practices for Azure Monitor Alerts**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/best-practices-alerts
   *Confidence: High — Well-Architected Framework guidance, includes reliability, cost optimization, operational excellence.*

3. **Webhook Retry Logic**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups#webhook
   *Confidence: High — Documented retry intervals (5s, 20s, 5s, 40s, 5s) and 15-min cooldown.*

4. **Automation Runbook with Managed Identity**
   https://learn.microsoft.com/en-us/azure/automation/automation-create-alert-triggered-runbook
   *Confidence: High — Code sample for VM stop runbook using Common Alert Schema.*

5. **Alert Processing Rules**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-processing-rules
   *Confidence: High — Covers suppression, action group override, scheduling.*

6. **Stateful vs. Stateless Alerts**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-troubleshoot-metric
   *Confidence: High — "Automatically resolve alerts" checkbox behavior explained.*

7. **Service Limits for Notifications**
   https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/action-groups#service-limits-for-notifications
   *Confidence: High — Rate limits per notification type (SMS, voice, email).*

### Baseline (Model Knowledge)

8. **AI Act Article 72 Incident Reporting**
   *Confidence: Medium — EU AI Act text available, but specific implementation guidance for Azure not yet published by Microsoft (as of Feb 2026).*

9. **Severity Mapping Best Practices**
   *Confidence: Medium — Industry standard pattern (Sev 0-4), adapted for AI-specific scenarios based on architecture experience.*

10. **Multi-Tier Escalation Pattern**
    *Confidence: High — Standard ITIL/SRE practice, Azure Monitor supports via action groups + Logic Apps.*

### Recommendations for Further Verification

- **Cost estimates:** Verify against Azure Pricing Calculator (pricing kan variere per region og currency fluctuations).
- **AI Act compliance:** Consult with legal team og Datatilsynet for norsk implementering av EU AI Act Article 72.
- **ITSM integration:** Test ITSM connector med deres spesifikke ServiceNow/Cherwell-versjon (API compatibility kan variere).