# Monitoring and Alerting for Failover Detection

**Last updated:** 2026-02
**Status:** GA
**Category:** Business Continuity & Disaster Recovery

---

## Introduksjon

Rask og pålitelig deteksjon av feil er avgjørende for å minimere nedetid i AI-systemer. Failover-deteksjon handler om å oppdage at en tjeneste eller region har feilet, og å initiere gjenopprettingsprosessen så raskt som mulig. For AI-workloads er dette spesielt viktig fordi forsinkede svar eller manglende tilgjengelighet direkte påvirker brukeropplevelsen.

Azure Monitor, Application Insights og Azure Service Health gir et robust rammeverk for overvåking og alerting. For AI-spesifikke metrikker som token-forbruk, modellkvalitet og search-indeksvaliditet kreves tilpasset monitoring med custom metrics og KQL-spørringer.

For norsk offentlig sektor som følger ITIL-baserte prosesser, må monitoring integreres med eksisterende incident management-systemer. NSMs grunnprinsipper krever "planlegging for å håndtere hendelser" (prinsipp 4.3), som inkluderer automatisk deteksjon og varsling.

## Health check-endepunkter og heartbeats

### Health check arkitektur

```
┌──────────────────┐
│  Azure Monitor   │
│  (Availability   │
│   Tests)         │
└────────┬─────────┘
         │ HTTPS GET /health
         ▼
┌──────────────────┐     ┌───────────────────┐
│  App Service     │────▶│ Deep Health Check  │
│  /health         │     │ ├─ OpenAI ✓/✗     │
│  (Shallow)       │     │ ├─ AI Search ✓/✗  │
│                  │     │ ├─ Cosmos DB ✓/✗   │
│  /health/deep    │     │ ├─ Redis ✓/✗      │
│  (Deep)          │     │ └─ Key Vault ✓/✗  │
└──────────────────┘     └───────────────────┘
```

### Health check implementering

```python
# FastAPI health check endpoints for AI service
from fastapi import FastAPI, Response
from datetime import datetime
import asyncio

app = FastAPI()

class HealthStatus:
    def __init__(self):
        self.checks = {}
        self.overall = "unknown"

async def check_openai():
    """Check Azure OpenAI availability."""
    try:
        response = await openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=1,
            timeout=5
        )
        return {"status": "healthy", "latency_ms": response.usage.total_tokens}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

async def check_search():
    """Check Azure AI Search availability."""
    try:
        results = search_client.search(search_text="*", top=1)
        count = 0
        async for _ in results:
            count += 1
        return {"status": "healthy", "documents_accessible": True}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

async def check_cosmos():
    """Check Cosmos DB availability."""
    try:
        await cosmos_container.read_item(
            item="health-check", partition_key="system"
        )
        return {"status": "healthy"}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

@app.get("/health")
async def shallow_health():
    """Shallow health check — is the app running?"""
    return {"status": "healthy", "timestamp": datetime.utcnow().isoformat()}

@app.get("/health/deep")
async def deep_health(response: Response):
    """Deep health check — are all dependencies healthy?"""
    checks = await asyncio.gather(
        check_openai(),
        check_search(),
        check_cosmos(),
        return_exceptions=True
    )

    result = {
        "timestamp": datetime.utcnow().isoformat(),
        "checks": {
            "openai": checks[0] if not isinstance(checks[0], Exception) else {"status": "error"},
            "search": checks[1] if not isinstance(checks[1], Exception) else {"status": "error"},
            "cosmos": checks[2] if not isinstance(checks[2], Exception) else {"status": "error"},
        }
    }

    # Bestem overall status
    unhealthy = [k for k, v in result["checks"].items()
                  if v.get("status") != "healthy"]

    if not unhealthy:
        result["status"] = "healthy"
    elif len(unhealthy) == len(result["checks"]):
        result["status"] = "unhealthy"
        response.status_code = 503
    else:
        result["status"] = "degraded"
        result["degraded_services"] = unhealthy
        response.status_code = 200  # Degraded men funksjonell

    return result
```

### Azure Monitor Availability Tests

```bash
# Opprett availability test for shallow health check
az monitor app-insights web-test create \
  --resource-group "rg-ai-prod" \
  --app-insights "ai-app-insights-prod" \
  --web-test-name "health-check-shallow" \
  --location "norwayeast" \
  --defined-web-test-name "ShallowHealthCheck" \
  --url "https://ai-app-prod.azurewebsites.net/health" \
  --expected-status-code 200 \
  --frequency 300 \
  --timeout 30 \
  --enabled true

# Opprett availability test for deep health check
az monitor app-insights web-test create \
  --resource-group "rg-ai-prod" \
  --app-insights "ai-app-insights-prod" \
  --web-test-name "health-check-deep" \
  --location "norwayeast" \
  --defined-web-test-name "DeepHealthCheck" \
  --url "https://ai-app-prod.azurewebsites.net/health/deep" \
  --expected-status-code 200 \
  --frequency 300 \
  --timeout 60 \
  --enabled true
```

## Latens og feilrate-overvåking

### KQL-spørringer for AI-metrikker

```kusto
// Azure OpenAI — Latency tracking per deployment
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| where TimeGenerated > ago(1h)
| extend
    deploymentName = tostring(properties_s.modelDeploymentName),
    latencyMs = duration_s * 1000,
    statusCode = resultCode_d
| summarize
    P50 = percentile(latencyMs, 50),
    P95 = percentile(latencyMs, 95),
    P99 = percentile(latencyMs, 99),
    SuccessRate = round(countif(statusCode < 400) * 100.0 / count(), 2),
    TotalRequests = count()
    by bin(TimeGenerated, 5m), deploymentName
| order by TimeGenerated desc
```

```kusto
// Azure AI Search — Query performance
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where OperationName == "Query.Search"
| where TimeGenerated > ago(1h)
| extend
    queryLatencyMs = DurationMs,
    resultCount = toint(Properties.ResultCount)
| summarize
    AvgLatency = avg(queryLatencyMs),
    P95Latency = percentile(queryLatencyMs, 95),
    AvgResults = avg(resultCount),
    TotalQueries = count(),
    ErrorRate = round(countif(resultSignature_d >= 400) * 100.0 / count(), 2)
    by bin(TimeGenerated, 5m)
| order by TimeGenerated desc
```

```kusto
// End-to-end RAG pipeline latency
customMetrics
| where name == "rag_pipeline_duration_ms"
| where timestamp > ago(1h)
| extend
    phase = tostring(customDimensions.phase),
    region = tostring(customDimensions.region)
| summarize
    P50 = percentile(value, 50),
    P95 = percentile(value, 95),
    P99 = percentile(value, 99)
    by bin(timestamp, 5m), phase, region
| order by timestamp desc, phase asc
```

## Custom metrics for AI-tjenestehelse

### Application Insights custom metrics

```python
# Custom metrics for AI service health monitoring
from opencensus.ext.azure.log_exporter import AzureLogHandler
from applicationinsights import TelemetryClient
import time

tc = TelemetryClient(instrumentation_key="<key>")

class AIMetricsCollector:
    """Collect and emit custom AI metrics."""

    def track_openai_call(self, deployment, latency_ms, tokens_used, success):
        """Track Azure OpenAI API call metrics."""
        tc.track_metric("openai_latency_ms", latency_ms, properties={
            "deployment": deployment,
            "success": str(success)
        })
        tc.track_metric("openai_tokens_used", tokens_used, properties={
            "deployment": deployment
        })
        if not success:
            tc.track_metric("openai_error_count", 1, properties={
                "deployment": deployment
            })

    def track_search_call(self, index_name, latency_ms, result_count, success):
        """Track Azure AI Search call metrics."""
        tc.track_metric("search_latency_ms", latency_ms, properties={
            "index": index_name,
            "success": str(success)
        })
        tc.track_metric("search_result_count", result_count, properties={
            "index": index_name
        })

    def track_rag_pipeline(self, total_ms, search_ms, llm_ms, success):
        """Track end-to-end RAG pipeline metrics."""
        tc.track_metric("rag_total_latency_ms", total_ms)
        tc.track_metric("rag_search_latency_ms", search_ms)
        tc.track_metric("rag_llm_latency_ms", llm_ms)
        tc.track_metric("rag_pipeline_success", 1 if success else 0)

    def track_health_check(self, service_name, is_healthy, latency_ms):
        """Track health check results for dashboards."""
        tc.track_metric(f"health_{service_name}", 1 if is_healthy else 0)
        tc.track_metric(f"health_{service_name}_latency", latency_ms)

    def flush(self):
        tc.flush()
```

## Alert-regler og eskaleringspolicyer

### Alerting-strategi

| Metrikk | Warning | Critical | Aksjon |
|---------|---------|----------|--------|
| OpenAI error rate | > 5% i 5 min | > 20% i 5 min | Notify → Auto-failover |
| OpenAI P95 latency | > 5s | > 15s | Notify team |
| Search error rate | > 2% i 5 min | > 10% i 5 min | Notify → Auto-failover |
| Health check failure | 2 consecutive | 3 consecutive | Initiate DR |
| Token consumption | > 80% quota | > 95% quota | Scale/notify |
| Cosmos DB latency | > 50ms P95 | > 200ms P95 | Investigate |

### Alert rules i Azure Monitor

```bash
# Critical: AI service health check failures
az monitor metrics alert create \
  --name "ai-health-critical" \
  --resource-group "rg-ai-prod" \
  --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
  --condition "count availabilityResults/failed > 3" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 0 \
  --action-group "ag-ai-oncall" \
  --description "3+ health check failures in 5 min — initiate DR assessment"

# Warning: Elevated OpenAI latency
az monitor scheduled-query create \
  --name "aoai-latency-warning" \
  --resource-group "rg-ai-prod" \
  --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
  --condition "count > 0" \
  --condition-query "
    customMetrics
    | where name == 'openai_latency_ms'
    | where timestamp > ago(5m)
    | summarize P95 = percentile(value, 95)
    | where P95 > 5000
  " \
  --evaluation-frequency 1m \
  --window-size 5m \
  --severity 2 \
  --action-group "ag-ai-team"
```

## Integrasjon med incident management-systemer

### Azure Logic App for eskalering

```json
{
  "definition": {
    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json",
    "triggers": {
      "alert_webhook": {
        "type": "Request",
        "kind": "Http",
        "inputs": {
          "schema": {
            "type": "object",
            "properties": {
              "alertName": {"type": "string"},
              "severity": {"type": "integer"},
              "affectedResource": {"type": "string"}
            }
          }
        }
      }
    },
    "actions": {
      "create_incident": {
        "type": "ApiConnection",
        "inputs": {
          "method": "POST",
          "host": "servicenow-connection",
          "path": "/api/now/table/incident",
          "body": {
            "short_description": "@{triggerBody().alertName}",
            "urgency": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
            "impact": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
            "assignment_group": "AI Platform Team",
            "category": "AI Service"
          }
        }
      },
      "send_teams_notification": {
        "type": "ApiConnection",
        "inputs": {
          "method": "POST",
          "host": "teams-connection",
          "path": "/v3/conversations/@{variables('teamChannelId')}/activities",
          "body": {
            "type": "message",
            "text": "AI Service Alert: @{triggerBody().alertName} (Sev @{triggerBody().severity})"
          }
        },
        "runAfter": { "create_incident": ["Succeeded"] }
      }
    }
  }
}
```

### Automatisk failover-trigger

```python
# Azure Function triggered by Alert webhook — initiate automated failover
import azure.functions as func
from azure.mgmt.trafficmanager import TrafficManagerManagementClient
from azure.identity import DefaultAzureCredential

def main(req: func.HttpRequest) -> func.HttpResponse:
    """Handle Azure Monitor alert and trigger failover if needed."""
    alert_data = req.get_json()

    severity = alert_data.get("data", {}).get("essentials", {}).get("severity")
    alert_name = alert_data.get("data", {}).get("essentials", {}).get("alertRule")

    if severity in ["Sev0", "Sev1"] and "health-critical" in alert_name:
        # Initier automatisk failover
        credential = DefaultAzureCredential()
        tm_client = TrafficManagerManagementClient(credential, subscription_id)

        # Oppdater Traffic Manager til å bruke sekundær region
        profile = tm_client.profiles.get("rg-networking", "tm-ai-failover")
        for endpoint in profile.endpoints:
            if "secondary" in endpoint.name:
                endpoint.priority = 1
            else:
                endpoint.priority = 2

        tm_client.profiles.create_or_update("rg-networking", "tm-ai-failover", profile)

        return func.HttpResponse(
            f"Failover initiated for alert: {alert_name}", status_code=200
        )

    return func.HttpResponse("Alert received, no failover needed", status_code=200)
```

## Referanser

- [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai) — OpenAI monitoring og alerting
- [Monitor Azure AI Search](https://learn.microsoft.com/en-us/azure/search/monitor-azure-cognitive-search) — AI Search monitoring
- [Azure Monitor alerts overview](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview) — Alert-rammeverk
- [Health modeling and observability of mission-critical workloads](https://learn.microsoft.com/en-us/azure/well-architected/mission-critical/mission-critical-health-modeling) — Health modeling
- [Application Insights overview](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) — APM for applikasjoner
- [Azure Service Health](https://learn.microsoft.com/en-us/azure/service-health/overview) — Azure-tjenestestatus

## For Cosmo

- **Bruk denne referansen** når kunden setter opp monitoring og alerting for failover-deteksjon i AI-systemer.
- Implementer alltid to nivåer av health checks: shallow (er appen oppe?) og deep (er alle avhengigheter friske?).
- Alert-terskler bør baseres på baseline-metrikker — bruk minst 2 ukers normaldata før du setter statiske terskler.
- For automatisk failover: Krev minimum 3 påfølgende health check-feil før failover trigges for å unngå false positives.
- Integrer med eksisterende ITSM-systemer (ServiceNow, Jira Service Management) via Azure Logic Apps eller Azure Functions.