open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity
ms-ai-architect/skills/ms-ai-infrastructure/references/bcdr/chaos-engineering-ai-systems.md
Kjell Tore Guttormsen ddce43d8b2 feat(ms-ai-architect): Spor 1 — Port-1-substrat migrert på 4 ikke-advisor-skills (243 Source + 327 Type + 325 TOC + stale-verified poison fjernet) [skip-docs] 
Steg 9 (R4): unified migrate-corpus.mjs --write over engineering/governance/
infrastructure/security. 327 filer mutert, verified=null, prosa byte-identisk
(fra første ## seksjon), advisor urørt (0 endringer).

To applier-fixes oppdaget under kjøring (TDD, RED→GREEN):
- insertHeaderFields: anker faller nå tilbake når en meta-linje selv passerer
  500B (2 filer pakket et avsnitt i **Status:** → Type/Source landet utenfor
  scan-vinduet, applierens post-write-assertion fanget + restaurerte).
- normalizeStaleVerified: fjerner nå ALLE stale non-date **Verified:** i
  500B-vinduet, inkl. stray body-dup rett under --- (9 mlops-genaiops-filer var
  ellers falskt "verified"/fresh, droppet fra worklist). Operatør-godkjent
  utvidelse av carve-out; kun stray metadata-linjer, aldri prosa.

test-transform-criterion: precondition oppdatert til post-migrasjons-sannhet
(fila bærer nå Source). Suite 728/728 grønn.
2026-07-04 10:19:11 +02:00

450 lines
17 KiB
Markdown
Raw Blame History

# Chaos Engineering for AI Systems
**Last updated:** 2026-02
**Status:** GA
**Category:** Business Continuity & Disaster Recovery
**Type:** reference
**Source:** https://learn.microsoft.com/azure/chaos-studio/chaos-studio-overview
---
## Innhold
- [Introduksjon](#introduksjon)
- [Feilinjeksjonsstrategier for AI-tjenester](#feilinjeksjonsstrategier-for-ai-tjenester)
- [Nettverkspartisjonssimulering](#nettverkspartisjonssimulering)
- [Last- og stresstesting](#last--og-stresstesting)
- [Recovery time-måling og validering](#recovery-time-måling-og-validering)
- [Verktøy og plattformer for chaos engineering](#verktøy-og-plattformer-for-chaos-engineering)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)
## Introduksjon
Chaos engineering er praksisen med å bevisst injisere feil i et system for å teste dets resiliens og avdekke svakheter før de forårsaker produksjonshendelser. For AI-systemer er dette spesielt verdifullt fordi AI-workloads har komplekse avhengighetskjeder (modell-endpoints, search-indekser, embedding-pipelines, datastores) der en feil i ett komponent kan kaskadere uforutsigbart.
Azure Chaos Studio er Azures native plattform for chaos engineering, og tilbyr både agentbasert og tjenestenivå feilinjeksjon. For AI-systemer kan Chaos Studio simulere alt fra nettverkspartisjonering til CPU-press og DNS-feil, noe som lar team validere at circuit breakers, retry-logikk og graceful degradation fungerer som forventet.
For norsk offentlig sektor er chaos engineering en viktig del av NSMs krav om regelmessig testing av sikkerhetstiltak (grunnprinsipp 4.3). Det anbefales at organisasjoner gjennomfører strukturerte feilinjeksjonstester minst kvartalsvis, og etter alle større endringer i AI-arkitekturen.
## Feilinjeksjonsstrategier for AI-tjenester
### Feilkatalog for AI-workloads
| Feiltype | Simulering | Påvirket komponent | Forventet respons |
|----------|-----------|-------------------|-------------------|
| Regional outage | DNS-feil eller nettverksblokk | Azure OpenAI | Failover til sekundær region |
| API throttling | Kunstig 429-respons | Azure OpenAI | Retry med backoff, graceful degradation |
| Search unavailable | Nettverksblokk til search | AI Search | Fallback til keyword search |
| High latency | Nettverksforsinkelse | Alle API-kall | Timeout → circuit breaker |
| Data corruption | Feil embedding-verdier | Cosmos DB / Search | Validering og rebuild |
| Memory pressure | VM memory stress | App Service | Auto-restart, scaling |
| Dependency failure | DNS poisoning | Key Vault, App Config | Cached config, graceful degradation |
### Azure Chaos Studio eksperimenter
```bash
# Aktiver Chaos Studio for ressurser
# Steg 1: Registrer target
az rest --method PUT \
--url "https://management.azure.com/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Web/sites/ai-app-prod/providers/Microsoft.Chaos/targets/Microsoft-AppService?api-version=2024-01-01" \
--body '{"properties":{}}'
# Steg 2: Aktiver capability (App Service Stop)
az rest --method PUT \
--url "https://management.azure.com/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Web/sites/ai-app-prod/providers/Microsoft.Chaos/targets/Microsoft-AppService/capabilities/Stop-1.0?api-version=2024-01-01" \
--body '{"properties":{}}'
```
### Chaos Experiment: Simuler Azure OpenAI Regional Outage
```json
{
"identity": {
"type": "SystemAssigned"
},
"location": "norwayeast",
"properties": {
"selectors": [
{
"id": "selector-nsg-block-openai",
"type": "List",
"targets": [
{
"id": "/subscriptions/{sub}/resourceGroups/rg-ai-prod/providers/Microsoft.Network/networkSecurityGroups/nsg-ai-app/providers/Microsoft.Chaos/targets/Microsoft-NetworkSecurityGroup",
"type": "ChaosTarget"
}
]
}
],
"steps": [
{
"name": "Block-OpenAI-Traffic",
"branches": [
{
"name": "branch-1",
"actions": [
{
"name": "urn:csci:microsoft:networkSecurityGroup:securityRule/1.1",
"type": "continuous",
"selectorId": "selector-nsg-block-openai",
"duration": "PT10M",
"parameters": [
{ "key": "name", "value": "chaos-block-openai" },
{ "key": "protocol", "value": "*" },
{ "key": "sourceAddresses", "value": "[\"*\"]" },
{ "key": "destinationAddresses", "value": "[\"CognitiveServicesManagement\"]" },
{ "key": "destinationPortRanges", "value": "[\"443\"]" },
{ "key": "access", "value": "Deny" },
{ "key": "priority", "value": "100" },
{ "key": "direction", "value": "Outbound" }
]
}
]
}
]
}
]
}
}
```
## Nettverkspartisjonssimulering
### Simuler cross-region nettverkspartisjon
```bash
# Chaos experiment: Simuler nettverkspartisjon mellom regioner
# Blokkerer VNet peering-trafikk for å teste failover
# Metode 1: NSG-basert blokkering
az network nsg rule create \
--resource-group "rg-networking" \
--nsg-name "nsg-ai-app" \
--name "chaos-block-cross-region" \
--priority 50 \
--direction Outbound \
--access Deny \
--protocol "*" \
--destination-address-prefixes "10.2.0.0/16" \
--description "CHAOS TEST: Block cross-region traffic"
# Vent og observer (10 minutter)
sleep 600
# Fjern blokkeringen
az network nsg rule delete \
--resource-group "rg-networking" \
--nsg-name "nsg-ai-app" \
--name "chaos-block-cross-region"
```
### DNS-feil simulering
```python
# Python: Simuler DNS-feil for testing
# Bruk Azure Private DNS zone override for å simulere DNS-feil
import subprocess
def simulate_dns_failure(target_fqdn: str, duration_minutes: int = 10):
"""Simulate DNS failure by overriding DNS resolution."""
print(f"Simulating DNS failure for {target_fqdn} for {duration_minutes} min")
# Opprett en DNS record som peker til en ikke-eksisterende IP
subprocess.run([
"az", "network", "private-dns", "record-set", "a", "add-record",
"--resource-group", "rg-networking",
"--zone-name", "privatelink.openai.azure.com",
"--record-set-name", "chaos-test",
"--ipv4-address", "10.255.255.255" # Ikke-ruterbar IP
])
print(f"DNS poisoned. Observing for {duration_minutes} minutes...")
import time
time.sleep(duration_minutes * 60)
# Rydd opp
subprocess.run([
"az", "network", "private-dns", "record-set", "a", "remove-record",
"--resource-group", "rg-networking",
"--zone-name", "privatelink.openai.azure.com",
"--record-set-name", "chaos-test",
"--ipv4-address", "10.255.255.255"
])
print("DNS restored.")
```
## Last- og stresstesting
### Load testing med Azure Load Testing
```yaml
# JMeter test plan for AI API stress testing
# azure-load-test-config.yaml
version: v0.1
testId: ai-stress-test
testPlan: ai-load-test.jmx
engineInstances: 5
configurationFiles:
- ai-load-test.jmx
failureCriteria:
- avg(response_time_ms) > 5000
- percentage(error) > 5
env:
- name: AOAI_ENDPOINT
value: https://aoai-prod.openai.azure.com
- name: SEARCH_ENDPOINT
value: https://search-prod.search.windows.net
```
```bash
# Opprett og kjør load test
az load test create \
--name "ai-stress-test" \
--resource-group "rg-ai-test" \
--load-test-resource "lt-ai-prod" \
--test-plan "ai-load-test.jmx" \
--engine-instances 5
# Kjør test med failover-scenario
az load test-run create \
--name "failover-stress-run" \
--resource-group "rg-ai-test" \
--load-test-resource "lt-ai-prod" \
--test-id "ai-stress-test" \
--description "Stress test during simulated failover"
```
### Gradvis belastningsøkning
```python
# Gradvis belastningsøkning for å finne breaking point
import asyncio
import aiohttp
import time
async def ramp_up_test(
endpoint: str,
start_rps: int = 10,
end_rps: int = 500,
step_rps: int = 10,
step_duration_seconds: int = 60
):
"""Gradually increase load to find service breaking point."""
current_rps = start_rps
results = []
while current_rps <= end_rps:
print(f"Testing at {current_rps} RPS for {step_duration_seconds}s...")
interval = 1.0 / current_rps
success_count = 0
error_count = 0
total_latency = 0
start = time.time()
while time.time() - start < step_duration_seconds:
try:
req_start = time.time()
async with aiohttp.ClientSession() as session:
async with session.post(endpoint, json={"query": "test"}) as resp:
if resp.status < 400:
success_count += 1
else:
error_count += 1
total_latency += (time.time() - req_start) * 1000
except Exception:
error_count += 1
await asyncio.sleep(interval)
total = success_count + error_count
error_rate = error_count / max(total, 1) * 100
avg_latency = total_latency / max(total, 1)
results.append({
"rps": current_rps,
"success": success_count,
"errors": error_count,
"error_rate": round(error_rate, 2),
"avg_latency_ms": round(avg_latency, 1)
})
print(f" Results: {error_rate:.1f}% errors, {avg_latency:.0f}ms avg latency")
# Stop hvis error rate er for høy
if error_rate > 20:
print(f"Breaking point found at {current_rps} RPS")
break
current_rps += step_rps
return results
```
## Recovery time-måling og validering
### RTO-måling under chaos testing
```python
# Automatisk RTO-måling under failover-test
import time
import requests
from datetime import datetime
class RTOMeasurement:
"""Measure actual RTO during failover tests."""
def __init__(self, health_endpoint: str, check_interval_seconds: float = 1.0):
self.health_endpoint = health_endpoint
self.check_interval = check_interval_seconds
self.measurements = []
def measure_rto(self, max_wait_seconds: int = 600) -> dict:
"""Continuously check health and measure recovery time."""
failure_detected = None
recovery_detected = None
was_healthy = True
checks = []
start = time.time()
while time.time() - start < max_wait_seconds:
try:
resp = requests.get(self.health_endpoint, timeout=5)
is_healthy = resp.status_code == 200
except Exception:
is_healthy = False
check = {
"timestamp": datetime.utcnow().isoformat(),
"elapsed_seconds": round(time.time() - start, 1),
"healthy": is_healthy
}
checks.append(check)
if was_healthy and not is_healthy and failure_detected is None:
failure_detected = time.time()
print(f"Failure detected at {check['elapsed_seconds']}s")
if not was_healthy and is_healthy and failure_detected and recovery_detected is None:
recovery_detected = time.time()
rto = recovery_detected - failure_detected
print(f"Recovery detected at {check['elapsed_seconds']}s — RTO: {rto:.1f}s")
was_healthy = is_healthy
time.sleep(self.check_interval)
result = {
"failure_detected": failure_detected is not None,
"recovery_detected": recovery_detected is not None,
"rto_seconds": round(recovery_detected - failure_detected, 1) if recovery_detected and failure_detected else None,
"total_checks": len(checks),
"healthy_checks": sum(1 for c in checks if c["healthy"]),
"unhealthy_checks": sum(1 for c in checks if not c["healthy"]),
"availability_pct": round(
sum(1 for c in checks if c["healthy"]) / max(len(checks), 1) * 100, 2
)
}
self.measurements.append(result)
return result
# Bruk
rto_meter = RTOMeasurement("https://ai-app-prod.azurewebsites.net/health")
result = rto_meter.measure_rto(max_wait_seconds=600)
print(f"Measured RTO: {result['rto_seconds']}s")
```
## Verktøy og plattformer for chaos engineering
### Azure Chaos Studio
| Funksjon | Beskrivelse | Støttede ressurser |
|----------|-------------|-------------------|
| Service-direct faults | Feil injisert via Azure API | App Service, AKS, Cosmos DB, NSG |
| Agent-based faults | Feil injisert via VM-agent | CPU/memory stress, network faults |
| Experiments | Strukturerte feilsekvenser | Alle støttede resurser |
| Permissions | RBAC-basert tilgangskontroll | Dedicated Chaos role |
### Komplementære verktøy
| Verktøy | Bruksområde | Integrasjon med Azure |
|---------|-------------|----------------------|
| Azure Chaos Studio | Native Azure fault injection | Innebygd |
| Azure Load Testing | Lasttesting | Innebygd, JMeter-basert |
| Litmus Chaos | Kubernetes chaos testing | AKS-kompatibel |
| Toxiproxy | Nettverksfeil for utvikling | Manuell oppsett |
| PYRIT | AI-spesifikk red teaming | Azure AI |
### Chaos Testing CI/CD-integrasjon
```yaml
# Azure DevOps Pipeline: Chaos testing som del av release
trigger: none
stages:
- stage: DeployToStaging
displayName: 'Deploy to Staging'
jobs:
- job: Deploy
steps:
- task: AzureWebApp@1
inputs:
appName: 'ai-app-staging'
- stage: ChaosTests
displayName: 'Run Chaos Experiments'
dependsOn: DeployToStaging
jobs:
- job: RunChaosExperiment
steps:
- task: AzureCLI@2
displayName: 'Start chaos experiment'
inputs:
azureSubscription: 'chaos-service-connection'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Start chaos experiment
EXPERIMENT_ID=$(az rest --method POST \
--url "https://management.azure.com/subscriptions/{sub}/resourceGroups/rg-ai-test/providers/Microsoft.Chaos/experiments/openai-failover-test/start?api-version=2024-01-01" \
--query "statusUrl" -o tsv)
echo "Experiment started: $EXPERIMENT_ID"
# Vent og mål RTO
python measure_rto.py \
--endpoint "https://ai-app-staging.azurewebsites.net/health" \
--max-wait 300
- task: AzureCLI@2
displayName: 'Validate results'
inputs:
azureSubscription: 'chaos-service-connection'
scriptType: 'bash'
scriptLocation: 'inlineScript'
inlineScript: |
# Sjekk at RTO er innenfor mål
RTO=$(python -c "import json; print(json.load(open('rto_result.json'))['rto_seconds'])")
if [ $(echo "$RTO > 900" | bc) -eq 1 ]; then
echo "##vso[task.logissue type=error]RTO exceeded 15 minutes: ${RTO}s"
exit 1
fi
echo "RTO within target: ${RTO}s"
```
## Referanser
- [What is Azure Chaos Studio?](https://learn.microsoft.com/en-us/azure/chaos-studio/chaos-studio-overview) — Chaos Studio oversikt
- [Understand chaos engineering and resilience](https://learn.microsoft.com/en-us/azure/chaos-studio/chaos-studio-chaos-engineering-overview) — Chaos engineering konsepter
- [Architecture strategies for designing a reliability testing strategy](https://learn.microsoft.com/en-us/azure/well-architected/reliability/testing-strategy) — WAF testing-strategi
- [Continuous validation with Azure Load Testing and Chaos Studio](https://learn.microsoft.com/en-us/azure/architecture/guide/testing/mission-critical-deployment-testing) — Kombinert testing
- [Shift right to test in production](https://learn.microsoft.com/en-us/devops/deliver/shift-right-test-production) — Fault injection i produksjon
- [Chaos Agent overview](https://learn.microsoft.com/en-us/azure/chaos-studio/chaos-agent-overview) — Agent-basert feilinjeksjon
## For Cosmo
- **Bruk denne referansen** når kunden ønsker å implementere chaos engineering for AI-systemer, eller når de trenger å validere sine DR-prosedyrer.
- Start med tabletop-øvelser før reelle feilinjeksjoner — forstå forventet oppførsel før du bryter ting.
- Bruk Azure Chaos Studio i staging-miljøer først, deretter gradvis i produksjon med begrenset blast radius.
- Integrer chaos testing i CI/CD — automatiserte failover-tester bør kjøres etter hver infrastrukturendring.
- RTO-måling er den viktigste outputen — dokumenter faktisk vs. planlagt RTO for å identifisere gap.
Reference in a new issue View git blame Copy permalink