ms-ai-architect/skills/ms-ai-security/references/performance-scalability/rate-limit-management.md

# Rate Limit Management

**Last updated:** 2026-02
**Status:** GA
**Category:** Performance & Scalability

---

## Introduksjon

Azure OpenAI bruker to rate limit-mekanismer: Tokens-per-Minute (TPM) og Requests-per-Minute (RPM). Når en av disse grensene overskrides, returnerer tjenesten HTTP 429 (Too Many Requests) med en `Retry-After` header som angir hvor mange sekunder klienten bør vente. For Standard deployments er rate limits direkte koblet til den tildelte kvoten, mens Provisioned Throughput (PTU) deployments returnerer 429 når utilization overstiger 100%.

Rate limit management er en av de mest kritiske aspektene ved produksjonsdrift av Azure OpenAI. Uten robust håndtering vil brukere oppleve sporadiske feil, og applikasjonen kan miste forespørsler under belastningstopper. Microsofts offisielle SDK-er (Python og JavaScript) har innebygd retry-logikk med eksponentiell backoff, men dette dekker kun grunnleggende scenarier. For produksjonsarkitekturer trengs mer sofistikerte strategier som multi-region failover, proaktiv throttling og quota monitoring.

For norsk offentlig sektor, der AI-tjenester kan være forretningskritiske for saksbehandling, er det avgjørende å ha en veldefinert strategi for rate limit management som sikrer at tjenesten er tilgjengelig selv under belastningstopper.

## Kjernekomponenter

| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| TPM/RPM Quota | Rate limiting per deployment | Azure OpenAI |
| Retry-After header | Server-side ventetid-instruksjon | HTTP 429 respons |
| Azure APIM | Gateway med rate limiting policies | Azure API Management |
| Circuit Breaker | Forhindre kaskade-feil | APIM / custom |
| Quota Management API | Programmatisk kvotejustering | Azure Management REST API |
| Azure Monitor | Rate limit-metrikker og alerting | Azure Monitor |

## Exponential Backoff Implementation

### Python SDK innebygd retry

```python
from openai import AzureOpenAI

# SDK har innebygd retry med exponential backoff
client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com",
    api_key="...",
    api_version="2024-10-21",
    max_retries=3,      # Default: 2
    timeout=120.0        # Total timeout i sekunder
)

# Per-request override
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"max_retries": "5"}  # Maks 5 forsøk for denne
)
```

### Custom retry med respekt for Retry-After

```python
import asyncio
import time
import random
from openai import AsyncAzureOpenAI, RateLimitError, APIError

class RateLimitHandler:
    """Advanced rate limit handling with exponential backoff."""

    def __init__(
        self,
        client: AsyncAzureOpenAI,
        max_retries: int = 5,
        base_delay: float = 1.0,
        max_delay: float = 60.0,
        jitter: bool = True
    ):
        self.client = client
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.jitter = jitter
        self._consecutive_429s = 0

    async def chat_completion(self, **kwargs) -> dict:
        """Execute chat completion with advanced retry logic."""
        last_exception = None

        for attempt in range(self.max_retries + 1):
            try:
                response = await self.client.chat.completions.create(**kwargs)
                self._consecutive_429s = 0  # Reset on success
                return response

            except RateLimitError as e:
                self._consecutive_429s += 1
                last_exception = e

                # Respekter Retry-After header
                retry_after = getattr(e, 'retry_after', None)
                if retry_after:
                    delay = float(retry_after)
                else:
                    # Exponential backoff: 1s, 2s, 4s, 8s, 16s...
                    delay = min(
                        self.base_delay * (2 ** attempt),
                        self.max_delay
                    )

                # Legg til jitter for å unngå thundering herd
                if self.jitter:
                    delay *= (0.5 + random.random())

                print(f"Rate limited (attempt {attempt + 1}/"
                      f"{self.max_retries}). "
                      f"Waiting {delay:.1f}s...")
                await asyncio.sleep(delay)

            except APIError as e:
                if e.status_code and e.status_code >= 500:
                    # Server error — retry
                    delay = self.base_delay * (2 ** attempt)
                    await asyncio.sleep(delay)
                    last_exception = e
                else:
                    raise  # Client error — ikke retry

        raise last_exception  # Alle forsøk brukt opp

    @property
    def is_throttled(self) -> bool:
        """Check if we're currently experiencing throttling."""
        return self._consecutive_429s >= 3
```

### .NET Polly-basert retry

```csharp
using Polly;
using Polly.Retry;

// Konfigurer retry policy med Polly
var retryPolicy = Policy
    .Handle<Azure.RequestFailedException>(ex => ex.Status == 429)
    .Or<Azure.RequestFailedException>(ex => ex.Status >= 500)
    .WaitAndRetryAsync(
        retryCount: 5,
        sleepDurationProvider: (retryAttempt, exception, context) =>
        {
            // Bruk Retry-After header hvis tilgjengelig
            if (exception is Azure.RequestFailedException rfEx)
            {
                var retryAfter = rfEx.GetRawResponse()?
                    .Headers.TryGetValue("Retry-After", out var value)
                    == true ? value : null;

                if (retryAfter != null &&
                    double.TryParse(retryAfter, out var seconds))
                {
                    return TimeSpan.FromSeconds(seconds);
                }
            }

            // Fallback: exponential backoff med jitter
            var baseDelay = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));
            var jitter = TimeSpan.FromMilliseconds(
                Random.Shared.Next(0, 1000));
            return baseDelay + jitter;
        },
        onRetryAsync: (exception, timespan, retryAttempt, context) =>
        {
            Console.WriteLine(
                $"Retry {retryAttempt} after {timespan.TotalSeconds:F1}s "
                + $"due to {exception.Message}");
            return Task.CompletedTask;
        }
    );
```

## Quota Request Process

### Overvåk og juster kvote programmatisk

```python
import requests

def get_quota_usage(
    subscription_id: str,
    resource_group: str,
    account_name: str,
    access_token: str
) -> dict:
    """Get current quota usage for Azure OpenAI deployments."""
    url = (
        f"https://management.azure.com/subscriptions/{subscription_id}"
        f"/resourceGroups/{resource_group}"
        f"/providers/Microsoft.CognitiveServices"
        f"/accounts/{account_name}"
        f"/deployments?api-version=2023-05-01"
    )

    headers = {"Authorization": f"Bearer {access_token}"}
    response = requests.get(url, headers=headers)
    deployments = response.json()["value"]

    usage = []
    for d in deployments:
        props = d["properties"]
        usage.append({
            "deployment": d["name"],
            "model": props["model"]["name"],
            "tpm_allocated": props.get("rateLimits", [{}])[0].get(
                "count", 0) if props.get("rateLimits") else 0,
            "sku": props.get("sku", {}).get("name", "unknown")
        })

    return usage

def update_deployment_quota(
    subscription_id: str,
    resource_group: str,
    account_name: str,
    deployment_name: str,
    new_tpm: int,
    access_token: str
):
    """Update TPM quota for a deployment."""
    url = (
        f"https://management.azure.com/subscriptions/{subscription_id}"
        f"/resourceGroups/{resource_group}"
        f"/providers/Microsoft.CognitiveServices"
        f"/accounts/{account_name}"
        f"/deployments/{deployment_name}?api-version=2023-05-01"
    )

    body = {
        "sku": {
            "name": "Standard",
            "capacity": new_tpm // 1000  # TPM i tusen-enheter
        }
    }

    headers = {
        "Authorization": f"Bearer {access_token}",
        "Content-Type": "application/json"
    }
    response = requests.patch(url, json=body, headers=headers)
    return response.json()
```

## Multi-Region Failover

### Automatisk failover ved rate limiting

```python
from dataclasses import dataclass, field
from typing import Optional
import time

@dataclass
class RegionalEndpoint:
    region: str
    endpoint: str
    api_key: str
    priority: int = 1
    is_healthy: bool = True
    throttled_until: float = 0
    consecutive_errors: int = 0

class MultiRegionRateLimitHandler:
    """Handle rate limits by failing over to other regions."""

    def __init__(self, endpoints: list[RegionalEndpoint]):
        self.endpoints = sorted(endpoints, key=lambda e: e.priority)

    def _get_available_endpoint(self) -> Optional[RegionalEndpoint]:
        """Get best available endpoint respecting throttle state."""
        now = time.time()

        for ep in self.endpoints:
            if ep.is_healthy and now > ep.throttled_until:
                return ep

        # Alle throttled — returner den som er tidligst klar
        available = sorted(
            self.endpoints,
            key=lambda e: e.throttled_until
        )
        return available[0] if available else None

    async def execute(self, **kwargs) -> dict:
        """Execute request with multi-region failover."""
        for attempt in range(len(self.endpoints) * 2):
            endpoint = self._get_available_endpoint()
            if not endpoint:
                raise Exception("No endpoints available")

            # Vent hvis throttled
            wait_time = max(0, endpoint.throttled_until - time.time())
            if wait_time > 0:
                await asyncio.sleep(wait_time)

            try:
                client = AsyncAzureOpenAI(
                    azure_endpoint=endpoint.endpoint,
                    api_key=endpoint.api_key,
                    api_version="2024-10-21",
                    max_retries=0  # Vi håndterer retry selv
                )
                response = await client.chat.completions.create(**kwargs)
                endpoint.consecutive_errors = 0
                endpoint.is_healthy = True
                return response

            except RateLimitError as e:
                retry_after = getattr(e, 'retry_after', 10)
                endpoint.throttled_until = time.time() + float(retry_after)
                endpoint.consecutive_errors += 1
                print(f"Region {endpoint.region} throttled for "
                      f"{retry_after}s. Trying next region...")
                continue

            except APIError as e:
                if e.status_code >= 500:
                    endpoint.consecutive_errors += 1
                    if endpoint.consecutive_errors >= 3:
                        endpoint.is_healthy = False
                    continue
                raise

        raise Exception("All regions exhausted")


# Konfigurasjon
handler = MultiRegionRateLimitHandler([
    RegionalEndpoint(
        region="norwayeast",
        endpoint="https://aoai-norway.openai.azure.com",
        api_key="...",
        priority=1
    ),
    RegionalEndpoint(
        region="swedencentral",
        endpoint="https://aoai-sweden.openai.azure.com",
        api_key="...",
        priority=2
    ),
    RegionalEndpoint(
        region="westeurope",
        endpoint="https://aoai-westeu.openai.azure.com",
        api_key="...",
        priority=3
    )
])
```

## Usage Monitoring

### KQL-spørringer for rate limit monitoring

```python
# Overvåk throttling i Azure Monitor

THROTTLE_MONITORING = """
AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
    TotalRequests = count(),
    Successful = countif(StatusCode == "200"),
    Throttled = countif(StatusCode == "429"),
    ServerErrors = countif(StatusCode startswith "5")
    by bin(TimeGenerated, 5m), Resource
| extend
    ThrottleRate = round(Throttled * 100.0 / TotalRequests, 2),
    ErrorRate = round(ServerErrors * 100.0 / TotalRequests, 2)
| where ThrottleRate > 0 or ErrorRate > 0
| order by TimeGenerated desc
"""

# Alert: Varsle når throttle rate overstiger terskel
THROTTLE_ALERT = """
AzureMetrics
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
    Total = count(),
    Throttled = countif(StatusCode == "429")
    by bin(TimeGenerated, 5m)
| extend ThrottleRate = Throttled * 100.0 / Total
| where ThrottleRate > 10
"""

# Quota utilization trend
QUOTA_UTILIZATION = """
AzureMetrics
| where MetricName in ("ProcessedPromptTokens", "GeneratedCompletionTokens")
| summarize
    PromptTPM = sumif(Total, MetricName == "ProcessedPromptTokens"),
    CompletionTPM = sumif(Total, MetricName == "GeneratedCompletionTokens")
    by bin(TimeGenerated, 1m)
| extend TotalTPM = PromptTPM + CompletionTPM
| order by TimeGenerated desc
"""
```

## Norsk offentlig sektor

- **SLA-implikasjoner**: Standard Azure OpenAI deployments har ingen latens-SLA — 429-feil er forventet atferd under høy belastning. Dokumenter dette i tjenesteavtaler med interne brukere.
- **Kvoteplanlegging**: Statlige organisasjoner bør planlegge TPM-kvote basert på forventet bruksmønster med 30-50% margin. Kvoteøkninger kan ta tid å behandle.
- **Multi-region compliance**: Ved failover til andre regioner, sørg for at databehandleravtale dekker alle regioner. For sensitivt innhold, bruk kun EU-baserte regioner.
- **Overvåking**: Sett opp Azure Monitor-alerts for throttle rate > 5% og utilization > 80% for proaktiv kvotejustering.
- **Beredskap**: Ha en eskaleringsplan for kvoteøkninger som inkluderer kontaktinformasjon for Microsoft-support.

## Beslutningsrammeverk

| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Sporadisk throttling (<5%) | Innebygd SDK retry | Tilstrekkelig for lav frekvens |
| Hyppig throttling (5-20%) | Øk kvote + multi-region failover | Kvoten er for lav for trafikken |
| Kritisk tjeneste, null toleranse | PTU deployment | Garantert kapasitet |
| Variabel trafikk med peaks | APIM med token rate limiting | Jevner ut trafikkmønstre |
| Multi-tenant applikasjon | Per-tenant rate limiting i APIM | Fair share mellom brukere |

## Referanser

- [Manage Azure OpenAI quota](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/quota) — Kvotehåndtering
- [Azure OpenAI quotas and limits](https://learn.microsoft.com/azure/ai-foundry/openai/quotas-limits) — Grenser per modell
- [Azure OpenAI SDK retry handling](https://learn.microsoft.com/azure/ai-foundry/openai/supported-languages) — SDK retry-konfigurasjon
- [Use a gateway in front of Azure OpenAI](https://learn.microsoft.com/azure/architecture/ai-ml/guide/azure-openai-gateway-multi-backend) — Multi-region gateway

## For Cosmo

- **Bruk denne referansen** når kunden opplever 429-feil, planlegger kvotestrategi, eller designer multi-region failover for Azure OpenAI.
- Alltid sjekk og respekter `Retry-After` headeren — SDK-ene gjør dette automatisk, men custom-klienter må implementere det.
- Multi-region failover er den mest robuste løsningen: prioriter Norway East → Sweden Central → West Europe for norske kunder.
- PTU eliminerer rate limiting helt (innenfor tildelt kapasitet) — anbefal for forretningskritiske workloads.
- Proaktiv kvotemonitorering er billigere enn reaktiv feilhåndtering — sett opp alerts FØR throttling oppstår.