feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,430 @@
|
|||
# Monitoring and Alerting for Failover Detection
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Status:** GA
|
||||
**Category:** Business Continuity & Disaster Recovery
|
||||
|
||||
---
|
||||
|
||||
## Introduksjon
|
||||
|
||||
Rask og pålitelig deteksjon av feil er avgjørende for å minimere nedetid i AI-systemer. Failover-deteksjon handler om å oppdage at en tjeneste eller region har feilet, og å initiere gjenopprettingsprosessen så raskt som mulig. For AI-workloads er dette spesielt viktig fordi forsinkede svar eller manglende tilgjengelighet direkte påvirker brukeropplevelsen.
|
||||
|
||||
Azure Monitor, Application Insights og Azure Service Health gir et robust rammeverk for overvåking og alerting. For AI-spesifikke metrikker som token-forbruk, modellkvalitet og search-indeksvaliditet kreves tilpasset monitoring med custom metrics og KQL-spørringer.
|
||||
|
||||
For norsk offentlig sektor som følger ITIL-baserte prosesser, må monitoring integreres med eksisterende incident management-systemer. NSMs grunnprinsipper krever "planlegging for å håndtere hendelser" (prinsipp 4.3), som inkluderer automatisk deteksjon og varsling.
|
||||
|
||||
## Health check-endepunkter og heartbeats
|
||||
|
||||
### Health check arkitektur
|
||||
|
||||
```
|
||||
┌──────────────────┐
|
||||
│ Azure Monitor │
|
||||
│ (Availability │
|
||||
│ Tests) │
|
||||
└────────┬─────────┘
|
||||
│ HTTPS GET /health
|
||||
▼
|
||||
┌──────────────────┐ ┌───────────────────┐
|
||||
│ App Service │────▶│ Deep Health Check │
|
||||
│ /health │ │ ├─ OpenAI ✓/✗ │
|
||||
│ (Shallow) │ │ ├─ AI Search ✓/✗ │
|
||||
│ │ │ ├─ Cosmos DB ✓/✗ │
|
||||
│ /health/deep │ │ ├─ Redis ✓/✗ │
|
||||
│ (Deep) │ │ └─ Key Vault ✓/✗ │
|
||||
└──────────────────┘ └───────────────────┘
|
||||
```
|
||||
|
||||
### Health check implementering
|
||||
|
||||
```python
|
||||
# FastAPI health check endpoints for AI service
|
||||
from fastapi import FastAPI, Response
|
||||
from datetime import datetime
|
||||
import asyncio
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
class HealthStatus:
|
||||
def __init__(self):
|
||||
self.checks = {}
|
||||
self.overall = "unknown"
|
||||
|
||||
async def check_openai():
|
||||
"""Check Azure OpenAI availability."""
|
||||
try:
|
||||
response = await openai_client.chat.completions.create(
|
||||
model="gpt-4o",
|
||||
messages=[{"role": "user", "content": "ping"}],
|
||||
max_tokens=1,
|
||||
timeout=5
|
||||
)
|
||||
return {"status": "healthy", "latency_ms": response.usage.total_tokens}
|
||||
except Exception as e:
|
||||
return {"status": "unhealthy", "error": str(e)}
|
||||
|
||||
async def check_search():
|
||||
"""Check Azure AI Search availability."""
|
||||
try:
|
||||
results = search_client.search(search_text="*", top=1)
|
||||
count = 0
|
||||
async for _ in results:
|
||||
count += 1
|
||||
return {"status": "healthy", "documents_accessible": True}
|
||||
except Exception as e:
|
||||
return {"status": "unhealthy", "error": str(e)}
|
||||
|
||||
async def check_cosmos():
|
||||
"""Check Cosmos DB availability."""
|
||||
try:
|
||||
await cosmos_container.read_item(
|
||||
item="health-check", partition_key="system"
|
||||
)
|
||||
return {"status": "healthy"}
|
||||
except Exception as e:
|
||||
return {"status": "unhealthy", "error": str(e)}
|
||||
|
||||
@app.get("/health")
|
||||
async def shallow_health():
|
||||
"""Shallow health check — is the app running?"""
|
||||
return {"status": "healthy", "timestamp": datetime.utcnow().isoformat()}
|
||||
|
||||
@app.get("/health/deep")
|
||||
async def deep_health(response: Response):
|
||||
"""Deep health check — are all dependencies healthy?"""
|
||||
checks = await asyncio.gather(
|
||||
check_openai(),
|
||||
check_search(),
|
||||
check_cosmos(),
|
||||
return_exceptions=True
|
||||
)
|
||||
|
||||
result = {
|
||||
"timestamp": datetime.utcnow().isoformat(),
|
||||
"checks": {
|
||||
"openai": checks[0] if not isinstance(checks[0], Exception) else {"status": "error"},
|
||||
"search": checks[1] if not isinstance(checks[1], Exception) else {"status": "error"},
|
||||
"cosmos": checks[2] if not isinstance(checks[2], Exception) else {"status": "error"},
|
||||
}
|
||||
}
|
||||
|
||||
# Bestem overall status
|
||||
unhealthy = [k for k, v in result["checks"].items()
|
||||
if v.get("status") != "healthy"]
|
||||
|
||||
if not unhealthy:
|
||||
result["status"] = "healthy"
|
||||
elif len(unhealthy) == len(result["checks"]):
|
||||
result["status"] = "unhealthy"
|
||||
response.status_code = 503
|
||||
else:
|
||||
result["status"] = "degraded"
|
||||
result["degraded_services"] = unhealthy
|
||||
response.status_code = 200 # Degraded men funksjonell
|
||||
|
||||
return result
|
||||
```
|
||||
|
||||
### Azure Monitor Availability Tests
|
||||
|
||||
```bash
|
||||
# Opprett availability test for shallow health check
|
||||
az monitor app-insights web-test create \
|
||||
--resource-group "rg-ai-prod" \
|
||||
--app-insights "ai-app-insights-prod" \
|
||||
--web-test-name "health-check-shallow" \
|
||||
--location "norwayeast" \
|
||||
--defined-web-test-name "ShallowHealthCheck" \
|
||||
--url "https://ai-app-prod.azurewebsites.net/health" \
|
||||
--expected-status-code 200 \
|
||||
--frequency 300 \
|
||||
--timeout 30 \
|
||||
--enabled true
|
||||
|
||||
# Opprett availability test for deep health check
|
||||
az monitor app-insights web-test create \
|
||||
--resource-group "rg-ai-prod" \
|
||||
--app-insights "ai-app-insights-prod" \
|
||||
--web-test-name "health-check-deep" \
|
||||
--location "norwayeast" \
|
||||
--defined-web-test-name "DeepHealthCheck" \
|
||||
--url "https://ai-app-prod.azurewebsites.net/health/deep" \
|
||||
--expected-status-code 200 \
|
||||
--frequency 300 \
|
||||
--timeout 60 \
|
||||
--enabled true
|
||||
```
|
||||
|
||||
## Latens og feilrate-overvåking
|
||||
|
||||
### KQL-spørringer for AI-metrikker
|
||||
|
||||
```kusto
|
||||
// Azure OpenAI — Latency tracking per deployment
|
||||
AzureDiagnostics
|
||||
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
|
||||
| where Category == "RequestResponse"
|
||||
| where TimeGenerated > ago(1h)
|
||||
| extend
|
||||
deploymentName = tostring(properties_s.modelDeploymentName),
|
||||
latencyMs = duration_s * 1000,
|
||||
statusCode = resultCode_d
|
||||
| summarize
|
||||
P50 = percentile(latencyMs, 50),
|
||||
P95 = percentile(latencyMs, 95),
|
||||
P99 = percentile(latencyMs, 99),
|
||||
SuccessRate = round(countif(statusCode < 400) * 100.0 / count(), 2),
|
||||
TotalRequests = count()
|
||||
by bin(TimeGenerated, 5m), deploymentName
|
||||
| order by TimeGenerated desc
|
||||
```
|
||||
|
||||
```kusto
|
||||
// Azure AI Search — Query performance
|
||||
AzureDiagnostics
|
||||
| where ResourceProvider == "MICROSOFT.SEARCH"
|
||||
| where OperationName == "Query.Search"
|
||||
| where TimeGenerated > ago(1h)
|
||||
| extend
|
||||
queryLatencyMs = DurationMs,
|
||||
resultCount = toint(Properties.ResultCount)
|
||||
| summarize
|
||||
AvgLatency = avg(queryLatencyMs),
|
||||
P95Latency = percentile(queryLatencyMs, 95),
|
||||
AvgResults = avg(resultCount),
|
||||
TotalQueries = count(),
|
||||
ErrorRate = round(countif(resultSignature_d >= 400) * 100.0 / count(), 2)
|
||||
by bin(TimeGenerated, 5m)
|
||||
| order by TimeGenerated desc
|
||||
```
|
||||
|
||||
```kusto
|
||||
// End-to-end RAG pipeline latency
|
||||
customMetrics
|
||||
| where name == "rag_pipeline_duration_ms"
|
||||
| where timestamp > ago(1h)
|
||||
| extend
|
||||
phase = tostring(customDimensions.phase),
|
||||
region = tostring(customDimensions.region)
|
||||
| summarize
|
||||
P50 = percentile(value, 50),
|
||||
P95 = percentile(value, 95),
|
||||
P99 = percentile(value, 99)
|
||||
by bin(timestamp, 5m), phase, region
|
||||
| order by timestamp desc, phase asc
|
||||
```
|
||||
|
||||
## Custom metrics for AI-tjenestehelse
|
||||
|
||||
### Application Insights custom metrics
|
||||
|
||||
```python
|
||||
# Custom metrics for AI service health monitoring
|
||||
from opencensus.ext.azure.log_exporter import AzureLogHandler
|
||||
from applicationinsights import TelemetryClient
|
||||
import time
|
||||
|
||||
tc = TelemetryClient(instrumentation_key="<key>")
|
||||
|
||||
class AIMetricsCollector:
|
||||
"""Collect and emit custom AI metrics."""
|
||||
|
||||
def track_openai_call(self, deployment, latency_ms, tokens_used, success):
|
||||
"""Track Azure OpenAI API call metrics."""
|
||||
tc.track_metric("openai_latency_ms", latency_ms, properties={
|
||||
"deployment": deployment,
|
||||
"success": str(success)
|
||||
})
|
||||
tc.track_metric("openai_tokens_used", tokens_used, properties={
|
||||
"deployment": deployment
|
||||
})
|
||||
if not success:
|
||||
tc.track_metric("openai_error_count", 1, properties={
|
||||
"deployment": deployment
|
||||
})
|
||||
|
||||
def track_search_call(self, index_name, latency_ms, result_count, success):
|
||||
"""Track Azure AI Search call metrics."""
|
||||
tc.track_metric("search_latency_ms", latency_ms, properties={
|
||||
"index": index_name,
|
||||
"success": str(success)
|
||||
})
|
||||
tc.track_metric("search_result_count", result_count, properties={
|
||||
"index": index_name
|
||||
})
|
||||
|
||||
def track_rag_pipeline(self, total_ms, search_ms, llm_ms, success):
|
||||
"""Track end-to-end RAG pipeline metrics."""
|
||||
tc.track_metric("rag_total_latency_ms", total_ms)
|
||||
tc.track_metric("rag_search_latency_ms", search_ms)
|
||||
tc.track_metric("rag_llm_latency_ms", llm_ms)
|
||||
tc.track_metric("rag_pipeline_success", 1 if success else 0)
|
||||
|
||||
def track_health_check(self, service_name, is_healthy, latency_ms):
|
||||
"""Track health check results for dashboards."""
|
||||
tc.track_metric(f"health_{service_name}", 1 if is_healthy else 0)
|
||||
tc.track_metric(f"health_{service_name}_latency", latency_ms)
|
||||
|
||||
def flush(self):
|
||||
tc.flush()
|
||||
```
|
||||
|
||||
## Alert-regler og eskaleringspolicyer
|
||||
|
||||
### Alerting-strategi
|
||||
|
||||
| Metrikk | Warning | Critical | Aksjon |
|
||||
|---------|---------|----------|--------|
|
||||
| OpenAI error rate | > 5% i 5 min | > 20% i 5 min | Notify → Auto-failover |
|
||||
| OpenAI P95 latency | > 5s | > 15s | Notify team |
|
||||
| Search error rate | > 2% i 5 min | > 10% i 5 min | Notify → Auto-failover |
|
||||
| Health check failure | 2 consecutive | 3 consecutive | Initiate DR |
|
||||
| Token consumption | > 80% quota | > 95% quota | Scale/notify |
|
||||
| Cosmos DB latency | > 50ms P95 | > 200ms P95 | Investigate |
|
||||
|
||||
### Alert rules i Azure Monitor
|
||||
|
||||
```bash
|
||||
# Critical: AI service health check failures
|
||||
az monitor metrics alert create \
|
||||
--name "ai-health-critical" \
|
||||
--resource-group "rg-ai-prod" \
|
||||
--scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
|
||||
--condition "count availabilityResults/failed > 3" \
|
||||
--window-size 5m \
|
||||
--evaluation-frequency 1m \
|
||||
--severity 0 \
|
||||
--action-group "ag-ai-oncall" \
|
||||
--description "3+ health check failures in 5 min — initiate DR assessment"
|
||||
|
||||
# Warning: Elevated OpenAI latency
|
||||
az monitor scheduled-query create \
|
||||
--name "aoai-latency-warning" \
|
||||
--resource-group "rg-ai-prod" \
|
||||
--scopes "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Insights/components/ai-app-insights-prod" \
|
||||
--condition "count > 0" \
|
||||
--condition-query "
|
||||
customMetrics
|
||||
| where name == 'openai_latency_ms'
|
||||
| where timestamp > ago(5m)
|
||||
| summarize P95 = percentile(value, 95)
|
||||
| where P95 > 5000
|
||||
" \
|
||||
--evaluation-frequency 1m \
|
||||
--window-size 5m \
|
||||
--severity 2 \
|
||||
--action-group "ag-ai-team"
|
||||
```
|
||||
|
||||
## Integrasjon med incident management-systemer
|
||||
|
||||
### Azure Logic App for eskalering
|
||||
|
||||
```json
|
||||
{
|
||||
"definition": {
|
||||
"$schema": "https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json",
|
||||
"triggers": {
|
||||
"alert_webhook": {
|
||||
"type": "Request",
|
||||
"kind": "Http",
|
||||
"inputs": {
|
||||
"schema": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"alertName": {"type": "string"},
|
||||
"severity": {"type": "integer"},
|
||||
"affectedResource": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"actions": {
|
||||
"create_incident": {
|
||||
"type": "ApiConnection",
|
||||
"inputs": {
|
||||
"method": "POST",
|
||||
"host": "servicenow-connection",
|
||||
"path": "/api/now/table/incident",
|
||||
"body": {
|
||||
"short_description": "@{triggerBody().alertName}",
|
||||
"urgency": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
|
||||
"impact": "@{if(equals(triggerBody().severity, 0), '1', '2')}",
|
||||
"assignment_group": "AI Platform Team",
|
||||
"category": "AI Service"
|
||||
}
|
||||
}
|
||||
},
|
||||
"send_teams_notification": {
|
||||
"type": "ApiConnection",
|
||||
"inputs": {
|
||||
"method": "POST",
|
||||
"host": "teams-connection",
|
||||
"path": "/v3/conversations/@{variables('teamChannelId')}/activities",
|
||||
"body": {
|
||||
"type": "message",
|
||||
"text": "AI Service Alert: @{triggerBody().alertName} (Sev @{triggerBody().severity})"
|
||||
}
|
||||
},
|
||||
"runAfter": { "create_incident": ["Succeeded"] }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Automatisk failover-trigger
|
||||
|
||||
```python
|
||||
# Azure Function triggered by Alert webhook — initiate automated failover
|
||||
import azure.functions as func
|
||||
from azure.mgmt.trafficmanager import TrafficManagerManagementClient
|
||||
from azure.identity import DefaultAzureCredential
|
||||
|
||||
def main(req: func.HttpRequest) -> func.HttpResponse:
|
||||
"""Handle Azure Monitor alert and trigger failover if needed."""
|
||||
alert_data = req.get_json()
|
||||
|
||||
severity = alert_data.get("data", {}).get("essentials", {}).get("severity")
|
||||
alert_name = alert_data.get("data", {}).get("essentials", {}).get("alertRule")
|
||||
|
||||
if severity in ["Sev0", "Sev1"] and "health-critical" in alert_name:
|
||||
# Initier automatisk failover
|
||||
credential = DefaultAzureCredential()
|
||||
tm_client = TrafficManagerManagementClient(credential, subscription_id)
|
||||
|
||||
# Oppdater Traffic Manager til å bruke sekundær region
|
||||
profile = tm_client.profiles.get("rg-networking", "tm-ai-failover")
|
||||
for endpoint in profile.endpoints:
|
||||
if "secondary" in endpoint.name:
|
||||
endpoint.priority = 1
|
||||
else:
|
||||
endpoint.priority = 2
|
||||
|
||||
tm_client.profiles.create_or_update("rg-networking", "tm-ai-failover", profile)
|
||||
|
||||
return func.HttpResponse(
|
||||
f"Failover initiated for alert: {alert_name}", status_code=200
|
||||
)
|
||||
|
||||
return func.HttpResponse("Alert received, no failover needed", status_code=200)
|
||||
```
|
||||
|
||||
## Referanser
|
||||
|
||||
- [Monitor Azure OpenAI](https://learn.microsoft.com/en-us/azure/ai-foundry/openai/how-to/monitor-openai) — OpenAI monitoring og alerting
|
||||
- [Monitor Azure AI Search](https://learn.microsoft.com/en-us/azure/search/monitor-azure-cognitive-search) — AI Search monitoring
|
||||
- [Azure Monitor alerts overview](https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-overview) — Alert-rammeverk
|
||||
- [Health modeling and observability of mission-critical workloads](https://learn.microsoft.com/en-us/azure/well-architected/mission-critical/mission-critical-health-modeling) — Health modeling
|
||||
- [Application Insights overview](https://learn.microsoft.com/en-us/azure/azure-monitor/app/app-insights-overview) — APM for applikasjoner
|
||||
- [Azure Service Health](https://learn.microsoft.com/en-us/azure/service-health/overview) — Azure-tjenestestatus
|
||||
|
||||
## For Cosmo
|
||||
|
||||
- **Bruk denne referansen** når kunden setter opp monitoring og alerting for failover-deteksjon i AI-systemer.
|
||||
- Implementer alltid to nivåer av health checks: shallow (er appen oppe?) og deep (er alle avhengigheter friske?).
|
||||
- Alert-terskler bør baseres på baseline-metrikker — bruk minst 2 ukers normaldata før du setter statiske terskler.
|
||||
- For automatisk failover: Krev minimum 3 påfølgende health check-feil før failover trigges for å unngå false positives.
|
||||
- Integrer med eksisterende ITSM-systemer (ServiceNow, Jira Service Management) via Azure Logic Apps eller Azure Functions.
|
||||
Loading…
Add table
Add a link
Reference in a new issue