ms-ai-architect/skills/ms-ai-infrastructure/references/bcdr/monitoring-alerting-failover-detection.md

Monitoring and Alerting for Failover Detection

Last updated: 2026-02 Status: GA Category: Business Continuity & Disaster Recovery

---

Introduksjon

Rask og pålitelig deteksjon av feil er avgjørende for å minimere nedetid i AI-systemer. Failover-deteksjon handler om å oppdage at en tjeneste eller region har feilet, og å initiere gjenopprettingsprosessen så raskt som mulig. For AI-workloads er dette spesielt viktig fordi forsinkede svar eller manglende tilgjengelighet direkte påvirker brukeropplevelsen.

Azure Monitor, Application Insights og Azure Service Health gir et robust rammeverk for overvåking og alerting. For AI-spesifikke metrikker som token-forbruk, modellkvalitet og search-indeksvaliditet kreves tilpasset monitoring med custom metrics og KQL-spørringer.

For norsk offentlig sektor som følger ITIL-baserte prosesser, må monitoring integreres med eksisterende incident management-systemer. NSMs grunnprinsipper krever "planlegging for å håndtere hendelser" (prinsipp 4.3), som inkluderer automatisk deteksjon og varsling.

Health check-endepunkter og heartbeats

Health check arkitektur

┌──────────────────┐
│  Azure Monitor   │
│  (Availability   │
│   Tests)         │
└────────┬─────────┘
         │ HTTPS GET /health
         ▼
┌──────────────────┐     ┌───────────────────┐
│  App Service     │────▶│ Deep Health Check  │
│  /health         │     │ ├─ OpenAI ✓/✗     │
│  (Shallow)       │     │ ├─ AI Search ✓/✗  │
│                  │     │ ├─ Cosmos DB ✓/✗   │
│  /health/deep    │     │ ├─ Redis ✓/✗      │
│  (Deep)          │     │ └─ Key Vault ✓/✗  │
└──────────────────┘     └───────────────────┘

Health check implementering

# FastAPI health check endpoints for AI service
from fastapi import FastAPI, Response
from datetime import datetime
import asyncio

app = FastAPI()

class HealthStatus:
    def __init__(self):
        self.checks = {}
        self.overall = "unknown"

async def check_openai():
    """Check Azure OpenAI availability."""
    try:
        response = await openai_client.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": "ping"}],
            max_tokens=1,
            timeout=5
        )
        return {"status": "healthy", "latency_ms": response.usage.total_tokens}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

async def check_search():
    """Check Azure AI Search availability."""
    try:
        results = search_client.search(search_text="*", top=1)
        count = 0
        async for _ in results:
            count += 1
        return {"status": "healthy", "documents_accessible": True}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

async def check_cosmos():
    """Check Cosmos DB availability."""
    try:
        await cosmos_container.read_item(
            item="health-check", partition_key="system"
        )
        return {"status": "healthy"}
    except Exception as e:
        return {"status": "unhealthy", "error": str(e)}

@app.get("/health")
async def shallow_health():
    """Shallow health check — is the app running?"""
    return {"status": "healthy", "timestamp": datetime.utcnow().isoformat()}

@app.get("/health/deep")
async def deep_health(response: Response):
    """Deep health check — are all dependencies healthy?"""
    checks = await asyncio.gather(
        check_openai(),
        check_search(),
        check_cosmos(),
        return_exceptions=True
    )

    result = {
        "timestamp": datetime.utcnow().isoformat(),
        "checks": {
            "openai": checks[0] if not isinstance(checks[0], Exception) else {"status": "error"},
            "search": checks[1] if not isinstance(checks[1], Exception) else {"status": "error"},
            "cosmos": checks[2] if not isinstance(checks[2], Exception) else {"status": "error"},
        }
    }

    # Bestem overall status
    unhealthy = [k for k, v in result["checks"].items()
                  if v.get("status") != "healthy"]

    if not unhealthy:
        result["status"] = "healthy"
    elif len(unhealthy) == len(result["checks"]):
        result["status"] = "unhealthy"
        response.status_code = 503
    else:
        result["status"] = "degraded"
        result["degraded_services"] = unhealthy
        response.status_code = 200  # Degraded men funksjonell

    return result

Azure Monitor Availability Tests

# Opprett availability test for shallow health check
az monitor app-insights web-test create \
  --resource-group "rg-ai-prod" \
  --app-insights "ai-app-insights-prod" \
  --web-test-name "health-check-shallow" \
  --location "norwayeast" \
  --defined-web-test-name "ShallowHealthCheck" \
  --url "https://ai-app-prod.azurewebsites.net/health" \
  --expected-status-code 200 \
  --frequency 300 \
  --timeout 30 \
  --enabled true

# Opprett availability test for deep health check
az monitor app-insights web-test create \
  --resource-group "rg-ai-prod" \
  --app-insights "ai-app-insights-prod" \
  --web-test-name "health-check-deep" \
  --location "norwayeast" \
  --defined-web-test-name "DeepHealthCheck" \
  --url "https://ai-app-prod.azurewebsites.net/health/deep" \
  --expected-status-code 200 \
  --frequency 300 \
  --timeout 60 \
  --enabled true

Latens og feilrate-overvåking

KQL-spørringer for AI-metrikker

// Azure OpenAI — Latency tracking per deployment
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| where TimeGenerated > ago(1h)
| extend
    deploymentName = tostring(properties_s.modelDeploymentName),
    latencyMs = duration_s * 1000,
    statusCode = resultCode_d
| summarize
    P50 = percentile(latencyMs, 50),
    P95 = percentile(latencyMs, 95),
    P99 = percentile(latencyMs, 99),
    SuccessRate = round(countif(statusCode < 400) * 100.0 / count(), 2),
    TotalRequests = count()
    by bin(TimeGenerated, 5m), deploymentName
| order by TimeGenerated desc

// Azure AI Search — Query performance
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.SEARCH"
| where OperationName == "Query.Search"
| where TimeGenerated > ago(1h)
| extend
    queryLatencyMs = DurationMs,
    resultCount = toint(Properties.ResultCount)
| summarize
    AvgLatency = avg(queryLatencyMs),
    P95Latency = percentile(queryLatencyMs, 95),
    AvgResults = avg(resultCount),
    TotalQueries = count(),
    ErrorRate = round(countif(resultSignature_d >= 400) * 100.0 / count(), 2)
    by bin(TimeGenerated, 5m)
| order by TimeGenerated desc

// End-to-end RAG pipeline latency
customMetrics
| where name == "rag_pipeline_duration_ms"
| where timestamp > ago(1h)
| extend
    phase = tostring(customDimensions.phase),
    region = tostring(customDimensions.region)
| summarize
    P50 = percentile(value, 50),
    P95 = percentile(value, 95),
    P99 = percentile(value, 99)
    by bin(timestamp, 5m), phase, region
| order by timestamp desc, phase asc

Custom metrics for AI-tjenestehelse

Application Insights custom metrics

# Custom metrics for AI service health monitoring
from opencensus.ext.azure.log_exporter import AzureLogHandler
from applicationinsights import TelemetryClient
import time

tc = TelemetryClient(instrumentation_key="<key>")

class AIMetricsCollector:
    """Collect and emit custom AI metrics."""

    def track_openai_call(self, deployment, latency_ms, tokens_used, success):
        """Track Azure OpenAI API call metrics."""
        tc.track_metric("openai_latency_ms", latency_ms, properties={
            "deployment": deployment,
            "success": str(success)
        })
        tc.track_metric("openai_tokens_used", tokens_used, properties={
            "deployment": deployment
        })
        if not success:
            tc.track_metric("openai_error_count", 1, properties={
                "deployment": deployment
            })

    def track_search_call(self, index_name, latency_ms, result_count, success):
        """Track Azure AI Search call metrics."""
        tc.track_metric("search_latency_ms", latency_ms, properties={
            "index": index_name,
            "success": str(success)
        })
        tc.track_metric("search_result_count", result_count, properties={
            "index": index_name
        })

    def track_rag_pipeline(self, total_ms, search_ms, llm_ms, success):
        """Track end-to-end RAG pipeline metrics."""
        tc.track_metric("rag_total_latency_ms", total_ms)
        tc.track_metric("rag_search_latency_ms", search_ms)
        tc.track_metric("rag_llm_latency_ms", llm_ms)
        tc.track_metric("rag_pipeline_success", 1 if success else 0)

    def track_health_check(self, service_name, is_healthy, latency_ms):
        """Track health check results for dashboards."""
        tc.track_metric(f"health_{service_name}", 1 if is_healthy else 0)
        tc.track_metric(f"health_{service_name}_latency", latency_ms)

    def flush(self):
        tc.flush()

Alert-regler og eskaleringspolicyer

Alerting-strategi

Metrikk	Warning	Critical	Aksjon
OpenAI error rate	> 5% i 5 min	> 20% i 5 min	Notify → Auto-failover
OpenAI P95 latency	> 5s	> 15s	Notify team
Search error rate	> 2% i 5 min	> 10% i 5 min	Notify → Auto-failover
Health check failure	2 consecutive	3 consecutive	Initiate DR
Token consumption	> 80% quota	> 95% quota	Scale/notify
Cosmos DB latency	> 50ms P95	> 200ms P95	Investigate

Alert rules i Azure Monitor

# Critical: AI service health check failures
az monitor metrics alert create \
  --name "ai-health-critical" \
  --resource-group "rg-ai-prod" \
  --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
  --condition "count availabilityResults/failed > 3" \
  --window-size 5m \
  --evaluation-frequency 1m \
  --severity 0 \
  --action-group "ag-ai-oncall" \
  --description "3+ health check failures in 5 min — initiate DR assessment"

# Warning: Elevated OpenAI latency
az monitor scheduled-query create \
  --name "aoai-latency-warning" \
  --resource-group "rg-ai-prod" \
  --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
  --condition "count > 0" \
  --condition-query "
    customMetrics
    | where name == 'openai_latency_ms'
    | where timestamp > ago(5m)
    | summarize P95 = percentile(value, 95)
    | where P95 > 5000
  " \
  --evaluation-frequency 1m \
  --window-size 5m \
  --severity 2 \
  --action-group "ag-ai-team"

Integrasjon med incident management-systemer

Azure Logic App for eskalering

{
  "definition": {
    "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json",
    "triggers": {
      "alert_webhook": {
        "type": "Request",
        "kind": "Http",
        "inputs": {
          "schema": {
            "type": "object",
            "properties": {
              "alertName": {"type": "string"},
              "severity": {"type": "integer"},
              "affectedResource": {"type": "string"}
            }
          }
        }
      }
    },
    "actions": {
      "create_incident": {
        "type": "ApiConnection",
        "inputs": {
          "method": "POST",
          "host": "servicenow-connection",
          "path": "/api/now/table/incident",
          "body": {
            "short_description": "@{triggerBody().alertName}",
            "urgency": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
            "impact": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
            "assignment_group": "AI Platform Team",
            "category": "AI Service"
          }
        }
      },
      "send_teams_notification": {
        "type": "ApiConnection",
        "inputs": {
          "method": "POST",
          "host": "teams-connection",
          "path": "/v3/conversations/@{variables('teamChannelId')}/activities",
          "body": {
            "type": "message",
            "text": "AI Service Alert: @{triggerBody().alertName} (Sev @{triggerBody().severity})"
          }
        },
        "runAfter": { "create_incident": ["Succeeded"] }
      }
    }
  }
}

Automatisk failover-trigger

# Azure Function triggered by Alert webhook — initiate automated failover
import azure.functions as func
from azure.mgmt.trafficmanager import TrafficManagerManagementClient
from azure.identity import DefaultAzureCredential

def main(req: func.HttpRequest) -> func.HttpResponse:
    """Handle Azure Monitor alert and trigger failover if needed."""
    alert_data = req.get_json()

    severity = alert_data.get("data", {}).get("essentials", {}).get("severity")
    alert_name = alert_data.get("data", {}).get("essentials", {}).get("alertRule")

    if severity in ["Sev0", "Sev1"] and "health-critical" in alert_name:
        # Initier automatisk failover
        credential = DefaultAzureCredential()
        tm_client = TrafficManagerManagementClient(credential, subscription_id)

        # Oppdater Traffic Manager til å bruke sekundær region
        profile = tm_client.profiles.get("rg-networking", "tm-ai-failover")
        for endpoint in profile.endpoints:
            if "secondary" in endpoint.name:
                endpoint.priority = 1
            else:
                endpoint.priority = 2

        tm_client.profiles.create_or_update("rg-networking", "tm-ai-failover", profile)

        return func.HttpResponse(
            f"Failover initiated for alert: {alert_name}", status_code=200
        )

    return func.HttpResponse("Alert received, no failover needed", status_code=200)

Application Insights for AI-agenter i BCDR-kontekst (Verified MCP 2026-04)

Azure Monitor Application Insights tilbyr nå dedikert støtte for AI-agenter via Agent details view, som er kritisk for failover-deteksjon i agent-baserte AI-systemer.

Agent details view — BCDR-relevans

Funksjon	BCDR-bruk
Unified agent view	Monitorer agenter fra Foundry, Copilot Studio og tredjeparts i én visning
End-to-end transaction details	Spor samtaler (prompts, systemInstructions, tool usage) ved incident-analyse
Live metrics	Sanntids health under failover-scenarier
Availability tests	Automatisk helsesjekk av agent-endepunkter

Instrumenteringsveiledning per agent-plattform

• Azure AI Foundry-agenter: Koble Application Insights til Foundry-prosjektet for automatisk tracing
• Copilot Studio-agenter: Konfigurer built-in telemetri-eksport til App Insights
• Microsoft Agent Framework (self-hosted): Bruk Azure Monitor OpenTelemetry Distro
• LangChain/LangGraph og OpenAI Agents SDK: Bruk Azure AI OpenTelemetry Tracer

Anbefaling: Gi hver agent et unikt navn for å skille dem i Agent details view. Bruk samme App Insights-ressurs for agenter som er del av et større system.

Referanser

• Monitor Azure OpenAI — OpenAI monitoring og alerting
• Monitor Azure AI Search — AI Search monitoring
• Azure Monitor alerts overview — Alert-rammeverk (Verified MCP 2026-04) — Stateful vs. stateless alerts, Simple Log Search Alerts (preview) for per-row evaluering, Query-based metric alerts for Prometheus/OTel (public preview). Alert processing rules for suppression ved planlagt vedlikehold. Opptil 5 action groups per alert rule.
• Health modeling and observability of mission-critical workloads — Health modeling
• Application Insights overview — APM for applikasjoner (Verified MCP 2026-04) — Nå OpenTelemetry-basert (OTel) som primær instrumentering. Nye features: Agent details view for AI-agenter fra Foundry, Copilot Studio og tredjeparts agenter. Støtter: Azure AI Foundry (via Foundry SDK tracing), Copilot Studio (built-in telemetri → App Insights), Microsoft Agent Framework (self-hosted), LangChain/LangGraph og OpenAI Agents SDK. Batch og continuous evaluations for produksjonstraffic. Live Metrics for sanntids observabilitet under failover-scenarier.
• Azure Service Health — Azure-tjenestestatus

For Cosmo

• Bruk denne referansen når kunden setter opp monitoring og alerting for failover-deteksjon i AI-systemer.
• Implementer alltid to nivåer av health checks: shallow (er appen oppe?) og deep (er alle avhengigheter friske?).
• Alert-terskler bør baseres på baseline-metrikker — bruk minst 2 ukers normaldata før du setter statiske terskler.
• For automatisk failover: Krev minimum 3 påfølgende health check-feil før failover trigges for å unngå false positives.
• Integrer med eksisterende ITSM-systemer (ServiceNow, Jira Service Management) via Azure Logic Apps eller Azure Functions.