# Monitoring and Alerting for Failover Detection **Last updated:** 2026-04 **Status:** GA **Category:** Business Continuity & Disaster Recovery --- ## Introduksjon Rask og pålitelig deteksjon av feil er avgjørende for å minimere nedetid i AI-systemer. Failover-deteksjon handler om å oppdage at en tjeneste eller region har feilet, og å initiere gjenopprettingsprosessen så raskt som mulig. For AI-workloads er dette spesielt viktig fordi forsinkede svar eller manglende tilgjengelighet direkte påvirker brukeropplevelsen. Azure Monitor, Application Insights og Azure Service Health gir et robust rammeverk for overvåking og alerting. For AI-spesifikke metrikker som token-forbruk, modellkvalitet og search-indeksvaliditet kreves tilpasset monitoring med custom metrics og KQL-spørringer. For norsk offentlig sektor som følger ITIL-baserte prosesser, må monitoring integreres med eksisterende incident management-systemer. NSMs grunnprinsipper krever "planlegging for å håndtere hendelser" (prinsipp 4.3), som inkluderer automatisk deteksjon og varsling. ## Health check-endepunkter og heartbeats ### Health check arkitektur ``` ┌──────────────────┐ │ Azure Monitor │ │ (Availability │ │ Tests) │ └────────┬─────────┘ │ HTTPS GET /health ▼ ┌──────────────────┐ ┌───────────────────┐ │ App Service │────▶│ Deep Health Check │ │ /health │ │ ├─ OpenAI ✓/✗ │ │ (Shallow) │ │ ├─ AI Search ✓/✗ │ │ │ │ ├─ Cosmos DB ✓/✗ │ │ /health/deep │ │ ├─ Redis ✓/✗ │ │ (Deep) │ │ └─ Key Vault ✓/✗ │ └──────────────────┘ └───────────────────┘ ``` ### Health check implementering ```python # FastAPI health check endpoints for AI service from fastapi import FastAPI, Response from datetime import datetime import asyncio app = FastAPI() class HealthStatus: def __init__(self): self.checks = {} self.overall = "unknown" async def check_openai(): """Check Azure OpenAI availability.""" try: response = await openai_client.chat.completions.create( model="gpt-4o", messages=[{"role": "user", "content": "ping"}], max_tokens=1, timeout=5 ) return {"status": "healthy", "latency_ms": response.usage.total_tokens} except Exception as e: return {"status": "unhealthy", "error": str(e)} async def check_search(): """Check Azure AI Search availability.""" try: results = search_client.search(search_text="*", top=1) count = 0 async for _ in results: count += 1 return {"status": "healthy", "documents_accessible": True} except Exception as e: return {"status": "unhealthy", "error": str(e)} async def check_cosmos(): """Check Cosmos DB availability.""" try: await cosmos_container.read_item( item="health-check", partition_key="system" ) return {"status": "healthy"} except Exception as e: return {"status": "unhealthy", "error": str(e)} @app.get("/health") async def shallow_health(): """Shallow health check — is the app running?""" return {"status": "healthy", "timestamp": datetime.utcnow().isoformat()} @app.get("/health/deep") async def deep_health(response: Response): """Deep health check — are all dependencies healthy?""" checks = await asyncio.gather( check_openai(), check_search(), check_cosmos(), return_exceptions=True ) result = { "timestamp": datetime.utcnow().isoformat(), "checks": { "openai": checks[0] if not isinstance(checks[0], Exception) else {"status": "error"}, "search": checks[1] if not isinstance(checks[1], Exception) else {"status": "error"}, "cosmos": checks[2] if not isinstance(checks[2], Exception) else {"status": "error"}, } } # Bestem overall status unhealthy = [k for k, v in result["checks"].items() if v.get("status") != "healthy"] if not unhealthy: result["status"] = "healthy" elif len(unhealthy) == len(result["checks"]): result["status"] = "unhealthy" response.status_code = 503 else: result["status"] = "degraded" result["degraded_services"] = unhealthy response.status_code = 200 # Degraded men funksjonell return result ``` ### Azure Monitor Availability Tests ```bash # Opprett availability test for shallow health check az monitor app-insights web-test create \ --resource-group "rg-ai-prod" \ --app-insights "ai-app-insights-prod" \ --web-test-name "health-check-shallow" \ --location "norwayeast" \ --defined-web-test-name "ShallowHealthCheck" \ --url "https://ai-app-prod.azurewebsites.net/health" \ --expected-status-code 200 \ --frequency 300 \ --timeout 30 \ --enabled true # Opprett availability test for deep health check az monitor app-insights web-test create \ --resource-group "rg-ai-prod" \ --app-insights "ai-app-insights-prod" \ --web-test-name "health-check-deep" \ --location "norwayeast" \ --defined-web-test-name "DeepHealthCheck" \ --url "https://ai-app-prod.azurewebsites.net/health/deep" \ --expected-status-code 200 \ --frequency 300 \ --timeout 60 \ --enabled true ``` ## Latens og feilrate-overvåking ### KQL-spørringer for AI-metrikker ```kusto // Azure OpenAI — Latency tracking per deployment AzureDiagnostics | where ResourceProvider == "MICROSOFT.COGNITIVESERVICES" | where Category == "RequestResponse" | where TimeGenerated > ago(1h) | extend deploymentName = tostring(properties_s.modelDeploymentName), latencyMs = duration_s * 1000, statusCode = resultCode_d | summarize P50 = percentile(latencyMs, 50), P95 = percentile(latencyMs, 95), P99 = percentile(latencyMs, 99), SuccessRate = round(countif(statusCode < 400) * 100.0 / count(), 2), TotalRequests = count() by bin(TimeGenerated, 5m), deploymentName | order by TimeGenerated desc ``` ```kusto // Azure AI Search — Query performance AzureDiagnostics | where ResourceProvider == "MICROSOFT.SEARCH" | where OperationName == "Query.Search" | where TimeGenerated > ago(1h) | extend queryLatencyMs = DurationMs, resultCount = toint(Properties.ResultCount) | summarize AvgLatency = avg(queryLatencyMs), P95Latency = percentile(queryLatencyMs, 95), AvgResults = avg(resultCount), TotalQueries = count(), ErrorRate = round(countif(resultSignature_d >= 400) * 100.0 / count(), 2) by bin(TimeGenerated, 5m) | order by TimeGenerated desc ``` ```kusto // End-to-end RAG pipeline latency customMetrics | where name == "rag_pipeline_duration_ms" | where timestamp > ago(1h) | extend phase = tostring(customDimensions.phase), region = tostring(customDimensions.region) | summarize P50 = percentile(value, 50), P95 = percentile(value, 95), P99 = percentile(value, 99) by bin(timestamp, 5m), phase, region | order by timestamp desc, phase asc ``` ## Custom metrics for AI-tjenestehelse ### Application Insights custom metrics ```python # Custom metrics for AI service health monitoring from opencensus.ext.azure.log_exporter import AzureLogHandler from applicationinsights import TelemetryClient import time tc = TelemetryClient(instrumentation_key="") class AIMetricsCollector: """Collect and emit custom AI metrics.""" def track_openai_call(self, deployment, latency_ms, tokens_used, success): """Track Azure OpenAI API call metrics.""" tc.track_metric("openai_latency_ms", latency_ms, properties={ "deployment": deployment, "success": str(success) }) tc.track_metric("openai_tokens_used", tokens_used, properties={ "deployment": deployment }) if not success: tc.track_metric("openai_error_count", 1, properties={ "deployment": deployment }) def track_search_call(self, index_name, latency_ms, result_count, success): """Track Azure AI Search call metrics.""" tc.track_metric("search_latency_ms", latency_ms, properties={ "index": index_name, "success": str(success) }) tc.track_metric("search_result_count", result_count, properties={ "index": index_name }) def track_rag_pipeline(self, total_ms, search_ms, llm_ms, success): """Track end-to-end RAG pipeline metrics.""" tc.track_metric("rag_total_latency_ms", total_ms) tc.track_metric("rag_search_latency_ms", search_ms) tc.track_metric("rag_llm_latency_ms", llm_ms) tc.track_metric("rag_pipeline_success", 1 if success else 0) def track_health_check(self, service_name, is_healthy, latency_ms): """Track health check results for dashboards.""" tc.track_metric(f"health_{service_name}", 1 if is_healthy else 0) tc.track_metric(f"health_{service_name}_latency", latency_ms) def flush(self): tc.flush() ``` ## Alert-regler og eskaleringspolicyer ### Alerting-strategi | Metrikk | Warning | Critical | Aksjon | |---------|---------|----------|--------| | OpenAI error rate | > 5% i 5 min | > 20% i 5 min | Notify → Auto-failover | | OpenAI P95 latency | > 5s | > 15s | Notify team | | Search error rate | > 2% i 5 min | > 10% i 5 min | Notify → Auto-failover | | Health check failure | 2 consecutive | 3 consecutive | Initiate DR | | Token consumption | > 80% quota | > 95% quota | Scale/notify | | Cosmos DB latency | > 50ms P95 | > 200ms P95 | Investigate | ### Alert rules i Azure Monitor ```bash # Critical: AI service health check failures az monitor metrics alert create \ --name "ai-health-critical" \ --resource-group "rg-ai-prod" \ --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \ --condition "count availabilityResults/failed > 3" \ --window-size 5m \ --evaluation-frequency 1m \ --severity 0 \ --action-group "ag-ai-oncall" \ --description "3+ health check failures in 5 min — initiate DR assessment" # Warning: Elevated OpenAI latency az monitor scheduled-query create \ --name "aoai-latency-warning" \ --resource-group "rg-ai-prod" \ --scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \ --condition "count > 0" \ --condition-query " customMetrics | where name == 'openai_latency_ms' | where timestamp > ago(5m) | summarize P95 = percentile(value, 95) | where P95 > 5000 " \ --evaluation-frequency 1m \ --window-size 5m \ --severity 2 \ --action-group "ag-ai-team" ``` ## Integrasjon med incident management-systemer ### Azure Logic App for eskalering ```json { "definition": { "$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json", "triggers": { "alert_webhook": { "type": "Request", "kind": "Http", "inputs": { "schema": { "type": "object", "properties": { "alertName": {"type": "string"}, "severity": {"type": "integer"}, "affectedResource": {"type": "string"} } } } } }, "actions": { "create_incident": { "type": "ApiConnection", "inputs": { "method": "POST", "host": "servicenow-connection", "path": "/api/now/table/incident", "body": { "short_description": "@{triggerBody().alertName}", "urgency": "@{if(equals(triggerBody().severity, 0), '1', '2')}", "impact": "@{if(equals(triggerBody().severity, 0), '1', '2')}", "assignment_group": "AI Platform Team", "category": "AI Service" } } }, "send_teams_notification": { "type": "ApiConnection", "inputs": { "method": "POST", "host": "teams-connection", "path": "/v3/conversations/@{variables('teamChannelId')}/activities", "body": { "type": "message", "text": "AI Service Alert: @{triggerBody().alertName} (Sev @{triggerBody().severity})" } }, "runAfter": { "create_incident": ["Succeeded"] } } } } } ``` ### Automatisk failover-trigger ```python # Azure Function triggered by Alert webhook — initiate automated failover import azure.functions as func from azure.mgmt.trafficmanager import TrafficManagerManagementClient from azure.identity import DefaultAzureCredential def main(req: func.HttpRequest) -> func.HttpResponse: """Handle Azure Monitor alert and trigger failover if needed.""" alert_data = req.get_json() severity = alert_data.get("data", {}).get("essentials", {}).get("severity") alert_name = alert_data.get("data", {}).get("essentials", {}).get("alertRule") if severity in ["Sev0", "Sev1"] and "health-critical" in alert_name: # Initier automatisk failover credential = DefaultAzureCredential() tm_client = TrafficManagerManagementClient(credential, subscription_id) # Oppdater Traffic Manager til å bruke sekundær region profile = tm_client.profiles.get("rg-networking", "tm-ai-failover") for endpoint in profile.endpoints: if "secondary" in endpoint.name: endpoint.priority = 1 else: endpoint.priority = 2 tm_client.profiles.create_or_update("rg-networking", "tm-ai-failover", profile) return func.HttpResponse( f"Failover initiated for alert: {alert_name}", status_code=200 ) return func.HttpResponse("Alert received, no failover needed", status_code=200) ``` ## Application Insights for AI-agenter i BCDR-kontekst *(Verified MCP 2026-04)* Azure Monitor Application Insights tilbyr nå dedikert støtte for AI-agenter via **Agent details view**, som er kritisk for failover-deteksjon i agent-baserte AI-systemer. ### Agent details view — BCDR-relevans | Funksjon | BCDR-bruk | |----------|-----------| | **Unified agent view** | Monitorer agenter fra Foundry, Copilot Studio og tredjeparts i én visning | | **End-to-end transaction details** | Spor samtaler (prompts, systemInstructions, tool usage) ved incident-analyse | | **Live metrics** | Sanntids health under failover-scenarier | | **Availability tests** | Automatisk helsesjekk av agent-endepunkter | ### Instrumenteringsveiledning per agent-plattform (Verified MCP 2026-04) - **Azure AI Foundry-agenter:** Start med [tracing setup i Foundry](https://learn.microsoft.com/azure/foundry/observability/how-to/trace-agent-setup). Koble Application Insights til Foundry-prosjektet for automatisk tracing. Kan også bruke Azure Monitor OpenTelemetry Distro med Foundry SDK. - **Copilot Studio-agenter:** Konfigurer built-in telemetri-eksport til App Insights via innstillinger i Copilot Studio. - **Microsoft Agent Framework (self-hosted):** Bruk Azure Monitor OpenTelemetry Distro for telemetri til Azure Monitor. - **LangChain/LangGraph og OpenAI Agents SDK:** Bruk Azure AI OpenTelemetry Tracer. Framework-spesifikk veiledning tilgjengelig i Foundry docs. **Anbefaling:** Gi hver agent et unikt navn for å skille dem i Agent details view. Bruk samme App Insights-ressurs for agenter som er del av et større system. Vil du se agenter i Azure AI Foundry i tillegg til Azure Monitor, [koble App Insights-ressursen til Foundry-prosjektet](https://learn.microsoft.com/azure/foundry/observability/how-to/trace-agent-setup#connect-application-insights-to-your-foundry-project). ## Referanser - [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai) — OpenAI monitoring og alerting - [Monitor Azure AI Search](https://learn.microsoft.com/en-us/azure/search/monitor-azure-cognitive-search) — AI Search monitoring - [Azure Monitor alerts overview](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview) — Alert-rammeverk *(Verified MCP 2026-04)* — Stateful vs. stateless alerts. **Simple Log Search Alerts** (GA) for per-row KQL evaluering — raskere varsling enn tradisjonelle log alerts. **Query-based metric alerts** for Prometheus/OTel (public preview). Alerts stored 30 dager. Fired instances er read-only. Alert processing rules for suppression ved planlagt vedlikehold. **Azure Monitor Baseline Alerts** (`aka.ms/amba`) for policy-basert alerting i skala via Azure Policy. - [Health modeling and observability of mission-critical workloads](https://learn.microsoft.com/en-us/azure/well-architected/mission-critical/mission-critical-health-modeling) — Health modeling - [Application Insights overview](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) — APM for applikasjoner *(Verified MCP 2026-04)* — OpenTelemetry (OTel) er primær instrumentering. AI-agenter støttes via Agents-tab i getting started. Azure Functions støtter OTel via `"telemetryMode": "OpenTelemetry"` i `host.json`. Nye views: **Agent details view** (Foundry, Copilot Studio, tredjeparts), **SDK Stats** (exporter success/drop metrics), **Dashboards with Grafana** (direkte i Azure portal). Evaluations: batch (local/cloud/portal) og continuous (produksjonstraffic). Classic API SDKs migreres til OTel — se migrasjonsveiledning. Fired alert instances er nå read-only (kan ikke editeres etter at de er trigget). - [Azure Service Health](https://learn.microsoft.com/en-us/azure/service-health/overview) — Azure-tjenestestatus ## For Cosmo - **Bruk denne referansen** når kunden setter opp monitoring og alerting for failover-deteksjon i AI-systemer. - Implementer alltid to nivåer av health checks: shallow (er appen oppe?) og deep (er alle avhengigheter friske?). - Alert-terskler bør baseres på baseline-metrikker — bruk minst 2 ukers normaldata før du setter statiske terskler. - For automatisk failover: Krev minimum 3 påfølgende health check-feil før failover trigges for å unngå false positives. - Integrer med eksisterende ITSM-systemer (ServiceNow, Jira Service Management) via Azure Logic Apps eller Azure Functions.