ms-ai-architect/skills/ms-ai-security/references/performance-scalability/load-testing-ai-services.md

# Load Testing AI Services

**Last updated:** 2026-06-24
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference
**Source:** https://learn.microsoft.com/azure/load-testing/overview-what-is-azure-load-testing

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Load Test Design](#load-test-design)
- [Realistic Traffic Patterns](#realistic-traffic-patterns)
- [Bottleneck Analysis](#bottleneck-analysis)
- [Capacity Forecasting](#capacity-forecasting)
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
- [Beslutningsrammeverk](#beslutningsrammeverk)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

Load testing av Azure AI Services er fundamentalt annerledes enn tradisjonell web-applikasjons lasttesting. AI-tjenester har variabel responstid basert på input-størrelse og output-kompleksitet, token-baserte rate limits (TPM/RPM) som ikke korrelerer lineært med antall forespørsler, og kostnader som skalerer med bruk. En enkelt Azure OpenAI-forespørsel kan ta fra 200ms til 120 sekunder avhengig av modell, prompt-størrelse og generert output.

Microsoft tilbyr to offisielle verktøy for dette: Azure Load Testing (JMeter-basert managed service) og azure-openai-benchmark (CLI-verktøy spesifikt for Azure OpenAI). For Provisioned Throughput Units (PTU) er benchmarking spesielt viktig fordi den faktiske throughputen avhenger av workload shape — forholdet mellom input og output tokens, call rate og cache match rate.

For norsk offentlig sektor bør load testing gjennomføres før produksjonslansering av alle AI-tjenester som eksponeres mot sluttbrukere, og deretter regelmessig for å verifisere at ytelsen holder seg innenfor definerte SLAer.

## Kjernekomponenter

| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| Azure Load Testing | Managed lasttestings-tjeneste | Azure Load Testing (JMeter) |
| azure-openai-benchmark | Offisielt benchmarking-verktøy for Azure OpenAI | GitHub CLI |
| Azure Monitor | Metrikker under lasttest | Azure Monitor |
| Application Insights | End-to-end latens-sporing | App Insights |
| Performance Optimizer | Azure Functions ytelsesoptimalisering | Azure Load Testing |

## Load Test Design

### Test-scenarioer for Azure OpenAI

```yaml
# azure-load-test-config.yaml
# Konfigurasjon for Azure Load Testing
version: v0.1
testId: aoai-load-test-chat
testPlan: aoai-chat-test.jmx
engineInstances: 3
configurationFiles:
  - aoai-chat-test.jmx
  - test-prompts.csv
failureCriteria:
  - avg(response_time_ms) > 5000
  - percentage(error) > 5
  - p95(response_time_ms) > 15000
env:
  - name: AOAI_ENDPOINT
    value: https://aoai-prod.openai.azure.com
  - name: DEPLOYMENT_NAME
    value: gpt-4o
  - name: API_VERSION
    value: "2024-10-21"
secrets:
  - name: AOAI_API_KEY
    value: $(aoai-api-key)  # Referanse til Key Vault
```

### Python-basert lasttest

```python
import asyncio
import time
import statistics
from dataclasses import dataclass, field
from openai import AsyncAzureOpenAI

@dataclass
class LoadTestConfig:
    target_rps: float           # Forespørsler per sekund
    duration_seconds: int        # Testens varighet
    ramp_up_seconds: int = 30   # Tid til full belastning
    model: str = "gpt-4o"
    max_tokens: int = 500
    concurrent_limit: int = 50

@dataclass
class LoadTestResults:
    total_requests: int = 0
    successful: int = 0
    failed: int = 0
    throttled: int = 0
    latencies_ms: list[float] = field(default_factory=list)
    tokens_used: int = 0

    @property
    def p50(self) -> float:
        return statistics.median(self.latencies_ms) if self.latencies_ms else 0

    @property
    def p95(self) -> float:
        if not self.latencies_ms:
            return 0
        sorted_l = sorted(self.latencies_ms)
        idx = int(len(sorted_l) * 0.95)
        return sorted_l[idx]

    @property
    def p99(self) -> float:
        if not self.latencies_ms:
            return 0
        sorted_l = sorted(self.latencies_ms)
        idx = int(len(sorted_l) * 0.99)
        return sorted_l[idx]

    @property
    def error_rate(self) -> float:
        total = self.successful + self.failed
        return round(self.failed / max(total, 1) * 100, 2)

    @property
    def throttle_rate(self) -> float:
        total = self.successful + self.failed
        return round(self.throttled / max(total, 1) * 100, 2)


async def run_load_test(
    client: AsyncAzureOpenAI,
    config: LoadTestConfig,
    test_prompts: list[str]
) -> LoadTestResults:
    """Run load test against Azure OpenAI deployment."""
    results = LoadTestResults()
    semaphore = asyncio.Semaphore(config.concurrent_limit)
    prompt_idx = 0

    async def send_request():
        nonlocal prompt_idx
        async with semaphore:
            prompt = test_prompts[prompt_idx % len(test_prompts)]
            prompt_idx += 1

            start = time.time()
            try:
                response = await client.chat.completions.create(
                    model=config.model,
                    messages=[{"role": "user", "content": prompt}],
                    max_tokens=config.max_tokens
                )
                latency = (time.time() - start) * 1000
                results.latencies_ms.append(latency)
                results.successful += 1
                results.tokens_used += response.usage.total_tokens

            except Exception as e:
                results.failed += 1
                if hasattr(e, 'status_code') and e.status_code == 429:
                    results.throttled += 1

            results.total_requests += 1

    # Ramp-up og sustained load
    start_time = time.time()
    tasks = []

    while time.time() - start_time < config.duration_seconds:
        elapsed = time.time() - start_time

        # Ramp-up: gradvis øk RPS
        if elapsed < config.ramp_up_seconds:
            current_rps = config.target_rps * (
                elapsed / config.ramp_up_seconds)
        else:
            current_rps = config.target_rps

        if current_rps > 0:
            interval = 1.0 / current_rps
            tasks.append(asyncio.create_task(send_request()))
            await asyncio.sleep(interval)

    # Vent på at alle pågående forespørsler fullføres
    await asyncio.gather(*tasks, return_exceptions=True)

    return results


# Bruk
async def main():
    client = AsyncAzureOpenAI(
        azure_endpoint="https://my-aoai.openai.azure.com",
        api_key="...",
        api_version="2024-10-21"
    )

    config = LoadTestConfig(
        target_rps=5.0,
        duration_seconds=300,
        ramp_up_seconds=60,
        model="gpt-4o",
        max_tokens=500,
        concurrent_limit=20
    )

    prompts = [
        "Oppsummer denne teksten: ...",
        "Klassifiser dette dokumentet: ...",
        "Generer et svar på denne klagen: ..."
    ]

    results = await run_load_test(client, config, prompts)

    print(f"Total: {results.total_requests}")
    print(f"Success: {results.successful}")
    print(f"Error rate: {results.error_rate}%")
    print(f"Throttle rate: {results.throttle_rate}%")
    print(f"P50: {results.p50:.0f}ms")
    print(f"P95: {results.p95:.0f}ms")
    print(f"P99: {results.p99:.0f}ms")
```

## Realistic Traffic Patterns

### Workload Shape-profiler

```python
from enum import Enum

class WorkloadProfile(Enum):
    CHAT_BOT = "chat_bot"
    DOCUMENT_ANALYSIS = "document_analysis"
    RAG_SEARCH = "rag_search"
    CODE_GENERATION = "code_generation"
    BATCH_PROCESSING = "batch_processing"

WORKLOAD_SHAPES = {
    WorkloadProfile.CHAT_BOT: {
        "avg_input_tokens": 200,
        "avg_output_tokens": 300,
        "peak_rps": 10,
        "avg_rps": 3,
        "pattern": "bursty",
        "description": "Korte input, moderate svar, ujevn trafikk"
    },
    WorkloadProfile.DOCUMENT_ANALYSIS: {
        "avg_input_tokens": 4000,
        "avg_output_tokens": 800,
        "peak_rps": 2,
        "avg_rps": 0.5,
        "pattern": "batch",
        "description": "Store input, strukturert output, batch-mønster"
    },
    WorkloadProfile.RAG_SEARCH: {
        "avg_input_tokens": 3000,
        "avg_output_tokens": 500,
        "peak_rps": 20,
        "avg_rps": 8,
        "pattern": "steady_with_peaks",
        "description": "Store kontekster fra search, mange samtidige"
    },
    WorkloadProfile.CODE_GENERATION: {
        "avg_input_tokens": 1500,
        "avg_output_tokens": 2000,
        "peak_rps": 5,
        "avg_rps": 1,
        "pattern": "variable",
        "description": "Middels input, store output, variabel"
    },
    WorkloadProfile.BATCH_PROCESSING: {
        "avg_input_tokens": 2000,
        "avg_output_tokens": 500,
        "peak_rps": 50,
        "avg_rps": 30,
        "pattern": "sustained",
        "description": "Jevnt høy belastning i batch-vindu"
    }
}
```

## Bottleneck Analysis

### Flaskehals-identifisering under test

```python
def analyze_bottlenecks(results: LoadTestResults, config: LoadTestConfig) -> list[str]:
    """Identify bottlenecks from load test results."""
    findings = []

    # 1. Throttling-analyse
    if results.throttle_rate > 5:
        findings.append(
            f"HIGH_THROTTLING: {results.throttle_rate}% throttled. "
            f"Øk TPM-kvote eller distribuer over flere regioner.")

    # 2. Latens-analyse
    if results.p95 > 10000:
        findings.append(
            f"HIGH_LATENCY: P95={results.p95:.0f}ms. "
            f"Vurder PTU for forutsigbar latens, "
            f"eller reduser max_tokens/prompt-størrelse.")

    # 3. Latens-spredning
    if results.p99 > results.p50 * 5:
        findings.append(
            f"HIGH_VARIANCE: P99/P50 ratio={results.p99/results.p50:.1f}. "
            f"Tyder på kapasitetsproblemer ved peak — "
            f"vurder circuit breaker og retry-logikk.")

    # 4. Throughput vs target
    actual_rps = results.successful / (
        config.duration_seconds - config.ramp_up_seconds)
    if actual_rps < config.target_rps * 0.8:
        findings.append(
            f"LOW_THROUGHPUT: {actual_rps:.1f} RPS vs target "
            f"{config.target_rps} RPS. "
            f"Klient-side bottleneck — øk concurrent_limit.")

    # 5. Error rate
    if results.error_rate > 1:
        findings.append(
            f"HIGH_ERRORS: {results.error_rate}% errors. "
            f"Sjekk 5xx-feil i Azure Monitor.")

    return findings
```

## Capacity Forecasting

### PTU-dimensjonering fra lasttestresultater

```python
def forecast_ptu_requirements(
    test_results: LoadTestResults,
    target_rps: float,
    model: str = "gpt-4o",
    growth_factor: float = 1.3  # 30% vekstmargin
) -> dict:
    """Forecast PTU requirements based on load test data."""

    # TPM per PTU (fra Microsoft dokumentasjon)
    tpm_per_ptu = {
        "gpt-4o": 2500,
        "gpt-4o-mini": 37000,
        "gpt-4.1": 3000,
        "gpt-4.1-mini": 14900,
        "gpt-4.1-nano": 59400,
        "o3": 3000
    }

    if model not in tpm_per_ptu:
        raise ValueError(f"Unknown model: {model}")

    avg_tokens_per_request = (
        test_results.tokens_used / max(test_results.successful, 1))
    required_tpm = target_rps * 60 * avg_tokens_per_request
    required_tpm_with_growth = required_tpm * growth_factor

    ptus_needed = required_tpm_with_growth / tpm_per_ptu[model]

    # Round opp til nærmeste deployable enhet.
    # Global & Data Zone Provisioned (anbefalt default): minimum 15 PTU, increment 5 for alle GPT-/o-modeller.
    # Regional Provisioned: minimum 50 (gpt-4.1/o3/gpt-4o), 25 for mini/nano-modeller.
    min_deployment = 15
    increment = 5
    ptus_deployed = max(
        min_deployment,
        ((int(ptus_needed) // increment) + 1) * increment
    )

    return {
        "model": model,
        "avg_tokens_per_request": round(avg_tokens_per_request),
        "required_tpm": round(required_tpm),
        "required_tpm_with_growth": round(required_tpm_with_growth),
        "ptus_needed_exact": round(ptus_needed, 1),
        "ptus_deployed": ptus_deployed,
        "headroom_pct": round(
            (ptus_deployed * tpm_per_ptu[model] - required_tpm) /
            required_tpm * 100, 1)
    }
```

### Azure OpenAI Benchmark Tool

```bash
# Offisielt benchmarking-verktøy fra Microsoft
# https://github.com/Azure/azure-openai-benchmark

# Installer
pip install azure-openai-benchmark

# Kjør benchmark med standard workload shape
azure-openai-benchmark \
  --api-key $AOAI_API_KEY \
  --api-base-endpoint https://my-aoai.openai.azure.com \
  --deployment gpt-4o \
  --shape-profile balanced \
  --clients 20 \
  --duration 600 \
  --output-format json \
  --output results.json

# Custom workload shape
azure-openai-benchmark \
  --api-key $AOAI_API_KEY \
  --api-base-endpoint https://my-aoai.openai.azure.com \
  --deployment gpt-4o \
  --context-tokens 3000 \
  --max-tokens 500 \
  --clients 10 \
  --rate 5 \
  --duration 300
```

## Norsk offentlig sektor

- **Krav til testing**: NSMs grunnprinsipper krever ytelsestesting av kritiske tjenester. AI-tjenester som brukes i saksbehandling bør lasttestes kvartalsvis og etter større endringer.
- **Testmiljø**: Bruk separate Azure OpenAI-deployments for lasttesting — aldri test mot produksjons-kvoten. Global Standard deployments er kostnadseffektive for testing.
- **Data i tester**: Bruk syntetiske eller anonymiserte data i lasttester. Reelle personopplysninger skal ikke brukes i testmiljøer.
- **Dokumentasjon**: Lasttestresultater bør dokumenteres som del av driftsdokumentasjonen og refereres i SLA-avtaler med interne tjenesteeiere.
- **Kostnadsbevissthet**: Lasttester genererer reelle token-kostnader. Estimer kostnad på forhånd og sett budsjettgrenser.

## Beslutningsrammeverk

| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Ny produksjonsdeployment | Full lasttest med ramp-up | Baseline etablering |
| PTU-dimensjonering | azure-openai-benchmark + kapasitetskalkulator | Mest nøyaktige tall |
| Etter kvoteendring | Regression-test med baseline | Verifiser forbedring |
| Multi-region failover | Lasttest under simulert feil | Valider failover-ytelse |
| Periodisk verifisering | Månedlig smoke test | Fang degradering tidlig |

## Referanser

- [Run a benchmark](https://learn.microsoft.com/azure/foundry/openai/how-to/provisioned-get-started#run-a-benchmark) — Azure OpenAI benchmarking guide
- [Azure OpenAI Benchmark Tool](https://github.com/Azure/azure-openai-benchmark) — Offisielt CLI-verktøy
- [Azure Load Testing overview](https://learn.microsoft.com/azure/load-testing/overview-what-is-azure-load-testing) — Managed lasttesting
- [Performance and latency](https://learn.microsoft.com/azure/foundry/openai/how-to/latency) — Throughput vs latency forklaring
- [Capacity planning](https://learn.microsoft.com/azure/well-architected/performance-efficiency/capacity-planning) — WAF kapasitetsplanlegging

## For Cosmo

- **Bruk denne referansen** når kunden skal dimensjonere Azure OpenAI-deployment, validere ytelse før lansering, eller feilsøke ytelsesprobler i produksjon.
- Alltid bruk azure-openai-benchmark for PTU-dimensjonering — kapasitetskalkulatoren gir estimater, benchmarking gir reelle tall.
- Definer workload shape (input tokens, output tokens, call rate) FØR testing — resultatene er kun gyldige for den testede workloaden.
- Kjør lasttester i minimum 10 minutter for å oppnå steady state — korte tester gir misvisende resultater.
- For norsk offentlig sektor: dokumenter baseline-ytelse og bruk den som referansepunkt for SLA-avtaler.