ms-ai-architect/skills/ms-ai-security/references/performance-scalability/rate-limit-management.md

Rate Limit Management

Last updated: 2026-06-19 | Verified: MCP 2026-06-19 Status: GA Category: Performance & Scalability

---

Introduksjon

Azure OpenAI bruker to rate limit-mekanismer: Tokens-per-Minute (TPM) og Requests-per-Minute (RPM). Når en av disse grensene overskrides, returnerer tjenesten HTTP 429 (Too Many Requests) med en Retry-After header som angir hvor mange sekunder klienten bør vente. For Standard deployments er rate limits direkte koblet til den tildelte kvoten, mens Provisioned Throughput (PTU) deployments returnerer 429 når utilization overstiger 100%.

Rate limit management er en av de mest kritiske aspektene ved produksjonsdrift av Azure OpenAI. Uten robust håndtering vil brukere oppleve sporadiske feil, og applikasjonen kan miste forespørsler under belastningstopper. Microsofts offisielle SDK-er (Python og JavaScript) har innebygd retry-logikk med eksponentiell backoff, men dette dekker kun grunnleggende scenarier. For produksjonsarkitekturer trengs mer sofistikerte strategier som multi-region failover, proaktiv throttling og quota monitoring.

For norsk offentlig sektor, der AI-tjenester kan være forretningskritiske for saksbehandling, er det avgjørende å ha en veldefinert strategi for rate limit management som sikrer at tjenesten er tilgjengelig selv under belastningstopper.

Kjernekomponenter

Komponent	Formål	Teknologi
TPM/RPM Quota	Rate limiting per deployment	Azure OpenAI
Retry-After header	Server-side ventetid-instruksjon	HTTP 429 respons
Azure APIM	Gateway med rate limiting policies	Azure API Management
Circuit Breaker	Forhindre kaskade-feil	APIM / custom
Quota Management API	Programmatisk kvotejustering	Azure Management REST API
Azure Monitor	Rate limit-metrikker og alerting	Azure Monitor

Exponential Backoff Implementation

Python SDK innebygd retry

from openai import AzureOpenAI

# SDK har innebygd retry med exponential backoff
client = AzureOpenAI(
    azure_endpoint="https://my-aoai.openai.azure.com",
    api_key="...",
    api_version="2024-10-21",
    max_retries=3,      # Default: 2
    timeout=120.0        # Total timeout i sekunder
)

# Per-request override
response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello"}],
    extra_headers={"max_retries": "5"}  # Maks 5 forsøk for denne
)

Custom retry med respekt for Retry-After

import asyncio
import time
import random
from openai import AsyncAzureOpenAI, RateLimitError, APIError

class RateLimitHandler:
    """Advanced rate limit handling with exponential backoff."""

    def __init__(
        self,
        client: AsyncAzureOpenAI,
        max_retries: int = 5,
        base_delay: float = 1.0,
        max_delay: float = 60.0,
        jitter: bool = True
    ):
        self.client = client
        self.max_retries = max_retries
        self.base_delay = base_delay
        self.max_delay = max_delay
        self.jitter = jitter
        self._consecutive_429s = 0

    async def chat_completion(self, **kwargs) -> dict:
        """Execute chat completion with advanced retry logic."""
        last_exception = None

        for attempt in range(self.max_retries + 1):
            try:
                response = await self.client.chat.completions.create(**kwargs)
                self._consecutive_429s = 0  # Reset on success
                return response

            except RateLimitError as e:
                self._consecutive_429s += 1
                last_exception = e

                # Respekter Retry-After header
                retry_after = getattr(e, 'retry_after', None)
                if retry_after:
                    delay = float(retry_after)
                else:
                    # Exponential backoff: 1s, 2s, 4s, 8s, 16s...
                    delay = min(
                        self.base_delay * (2 ** attempt),
                        self.max_delay
                    )

                # Legg til jitter for å unngå thundering herd
                if self.jitter:
                    delay *= (0.5 + random.random())

                print(f"Rate limited (attempt {attempt + 1}/"
                      f"{self.max_retries}). "
                      f"Waiting {delay:.1f}s...")
                await asyncio.sleep(delay)

            except APIError as e:
                if e.status_code and e.status_code >= 500:
                    # Server error — retry
                    delay = self.base_delay * (2 ** attempt)
                    await asyncio.sleep(delay)
                    last_exception = e
                else:
                    raise  # Client error — ikke retry

        raise last_exception  # Alle forsøk brukt opp

    @property
    def is_throttled(self) -> bool:
        """Check if we're currently experiencing throttling."""
        return self._consecutive_429s >= 3

.NET Polly-basert retry

using Polly;
using Polly.Retry;

// Konfigurer retry policy med Polly
var retryPolicy = Policy
    .Handle<Azure.RequestFailedException>(ex => ex.Status == 429)
    .Or<Azure.RequestFailedException>(ex => ex.Status >= 500)
    .WaitAndRetryAsync(
        retryCount: 5,
        sleepDurationProvider: (retryAttempt, exception, context) =>
        {
            // Bruk Retry-After header hvis tilgjengelig
            if (exception is Azure.RequestFailedException rfEx)
            {
                var retryAfter = rfEx.GetRawResponse()?
                    .Headers.TryGetValue("Retry-After", out var value)
                    == true ? value : null;

                if (retryAfter != null &&
                    double.TryParse(retryAfter, out var seconds))
                {
                    return TimeSpan.FromSeconds(seconds);
                }
            }

            // Fallback: exponential backoff med jitter
            var baseDelay = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));
            var jitter = TimeSpan.FromMilliseconds(
                Random.Shared.Next(0, 1000));
            return baseDelay + jitter;
        },
        onRetryAsync: (exception, timespan, retryAttempt, context) =>
        {
            Console.WriteLine(
                $"Retry {retryAttempt} after {timespan.TotalSeconds:F1}s "
                + $"due to {exception.Message}");
            return Task.CompletedTask;
        }
    );

Quota Request Process

Overvåk og juster kvote programmatisk

import requests

def get_quota_usage(
    subscription_id: str,
    resource_group: str,
    account_name: str,
    access_token: str
) -> dict:
    """Get current quota usage for Azure OpenAI deployments."""
    url = (
        f"https://management.azure.com/subscriptions/{subscription_id}"
        f"/resourceGroups/{resource_group}"
        f"/providers/Microsoft.CognitiveServices"
        f"/accounts/{account_name}"
        f"/deployments?api-version=2023-05-01"
    )

    headers = {"Authorization": f"Bearer {access_token}"}
    response = requests.get(url, headers=headers)
    deployments = response.json()["value"]

    usage = []
    for d in deployments:
        props = d["properties"]
        usage.append({
            "deployment": d["name"],
            "model": props["model"]["name"],
            "tpm_allocated": props.get("rateLimits", [{}])[0].get(
                "count", 0) if props.get("rateLimits") else 0,
            "sku": props.get("sku", {}).get("name", "unknown")
        })

    return usage

def update_deployment_quota(
    subscription_id: str,
    resource_group: str,
    account_name: str,
    deployment_name: str,
    new_tpm: int,
    access_token: str
):
    """Update TPM quota for a deployment."""
    url = (
        f"https://management.azure.com/subscriptions/{subscription_id}"
        f"/resourceGroups/{resource_group}"
        f"/providers/Microsoft.CognitiveServices"
        f"/accounts/{account_name}"
        f"/deployments/{deployment_name}?api-version=2023-05-01"
    )

    body = {
        "sku": {
            "name": "Standard",
            "capacity": new_tpm // 1000  # TPM i tusen-enheter
        }
    }

    headers = {
        "Authorization": f"Bearer {access_token}",
        "Content-Type": "application/json"
    }
    response = requests.patch(url, json=body, headers=headers)
    return response.json()

Multi-Region Failover

Automatisk failover ved rate limiting

from dataclasses import dataclass, field
from typing import Optional
import time

@dataclass
class RegionalEndpoint:
    region: str
    endpoint: str
    api_key: str
    priority: int = 1
    is_healthy: bool = True
    throttled_until: float = 0
    consecutive_errors: int = 0

class MultiRegionRateLimitHandler:
    """Handle rate limits by failing over to other regions."""

    def __init__(self, endpoints: list[RegionalEndpoint]):
        self.endpoints = sorted(endpoints, key=lambda e: e.priority)

    def _get_available_endpoint(self) -> Optional[RegionalEndpoint]:
        """Get best available endpoint respecting throttle state."""
        now = time.time()

        for ep in self.endpoints:
            if ep.is_healthy and now > ep.throttled_until:
                return ep

        # Alle throttled — returner den som er tidligst klar
        available = sorted(
            self.endpoints,
            key=lambda e: e.throttled_until
        )
        return available[0] if available else None

    async def execute(self, **kwargs) -> dict:
        """Execute request with multi-region failover."""
        for attempt in range(len(self.endpoints) * 2):
            endpoint = self._get_available_endpoint()
            if not endpoint:
                raise Exception("No endpoints available")

            # Vent hvis throttled
            wait_time = max(0, endpoint.throttled_until - time.time())
            if wait_time > 0:
                await asyncio.sleep(wait_time)

            try:
                client = AsyncAzureOpenAI(
                    azure_endpoint=endpoint.endpoint,
                    api_key=endpoint.api_key,
                    api_version="2024-10-21",
                    max_retries=0  # Vi håndterer retry selv
                )
                response = await client.chat.completions.create(**kwargs)
                endpoint.consecutive_errors = 0
                endpoint.is_healthy = True
                return response

            except RateLimitError as e:
                retry_after = getattr(e, 'retry_after', 10)
                endpoint.throttled_until = time.time() + float(retry_after)
                endpoint.consecutive_errors += 1
                print(f"Region {endpoint.region} throttled for "
                      f"{retry_after}s. Trying next region...")
                continue

            except APIError as e:
                if e.status_code >= 500:
                    endpoint.consecutive_errors += 1
                    if endpoint.consecutive_errors >= 3:
                        endpoint.is_healthy = False
                    continue
                raise

        raise Exception("All regions exhausted")


# Konfigurasjon
handler = MultiRegionRateLimitHandler([
    RegionalEndpoint(
        region="norwayeast",
        endpoint="https://aoai-norway.openai.azure.com",
        api_key="...",
        priority=1
    ),
    RegionalEndpoint(
        region="swedencentral",
        endpoint="https://aoai-sweden.openai.azure.com",
        api_key="...",
        priority=2
    ),
    RegionalEndpoint(
        region="westeurope",
        endpoint="https://aoai-westeu.openai.azure.com",
        api_key="...",
        priority=3
    )
])

Usage Monitoring

KQL-spørringer for rate limit monitoring

# Overvåk throttling i Azure Monitor

THROTTLE_MONITORING = """
AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
    TotalRequests = count(),
    Successful = countif(StatusCode == "200"),
    Throttled = countif(StatusCode == "429"),
    ServerErrors = countif(StatusCode startswith "5")
    by bin(TimeGenerated, 5m), Resource
| extend
    ThrottleRate = round(Throttled * 100.0 / TotalRequests, 2),
    ErrorRate = round(ServerErrors * 100.0 / TotalRequests, 2)
| where ThrottleRate > 0 or ErrorRate > 0
| order by TimeGenerated desc
"""

# Alert: Varsle når throttle rate overstiger terskel
THROTTLE_ALERT = """
AzureMetrics
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
    Total = count(),
    Throttled = countif(StatusCode == "429")
    by bin(TimeGenerated, 5m)
| extend ThrottleRate = Throttled * 100.0 / Total
| where ThrottleRate > 10
"""

# Quota utilization trend
QUOTA_UTILIZATION = """
AzureMetrics
| where MetricName in ("ProcessedPromptTokens", "GeneratedCompletionTokens")
| summarize
    PromptTPM = sumif(Total, MetricName == "ProcessedPromptTokens"),
    CompletionTPM = sumif(Total, MetricName == "GeneratedCompletionTokens")
    by bin(TimeGenerated, 1m)
| extend TotalTPM = PromptTPM + CompletionTPM
| order by TimeGenerated desc
"""

Gateway Multi-Backend som Rate Limit-strategi (oppdatert 2026-06-19)

Microsoft dokumenterer multi-backend gateway som den anbefalte arkitekturmønsteret for rate limit management — primært via Azure API Management:

Anbefalte topologier for rate limit-distribusjon

Viktig: Standard-kvote er subscription-nivå, ikke Azure OpenAI-instansnivå. Load balancing mellom standard-instanser i samme subscription gir IKKE høyere gjennomstrømning — bruk separate subscriptions eller global/data zone deployments for reell kvoteutvidelse. — Verified (MCP 2026-06-19)

Topologi	Kvote-kapasitet	Kompleksitet	Anbefalt for
Single instance	Baseline TPM	Lav	Utvikling, lav trafikk
Multi-backend, single region	2-5x baseline	Medium	Produksjon, standard
Multi-subscription	5-20x baseline	Høy	Høy trafikk enterprise
Multi-region	Nær ubegrenset	Høy	Kritisk infrastruktur

APIM-basert rate limit distribusjon

<!-- APIM Policy: Distribuer rate limit på tvers av backends -->
<policies>
    <inbound>
        <base />
        
        <!-- Token-based rate limiting i APIM (avlaster Azure OpenAI) -->
        <azure-openai-token-limit
            counter-key="@(context.Request.Headers.GetValueOrDefault("x-client-id", "default"))"
            tokens-per-minute="10000"
            estimate-prompt-tokens="true"
            tokens-consumed-variable-name="consumed-tokens"
            remaining-tokens-variable-name="remaining-tokens" />
        
        <!-- Velg backend basert på tilgjengelighet -->
        <set-variable name="backend-url" value="@{
            // Prioritert liste: prøv Norway East, fallback til Sweden Central
            if (context.Variables.GetValueOrDefault<int>("norway-throttle") < DateTimeOffset.UtcNow.ToUnixTimeSeconds())
                return "https://aoai-norway.openai.azure.com";
            return "https://aoai-sweden.openai.azure.com";
        }" />
        
        <set-backend-service base-url="@(context.Variables.GetValueOrDefault<string>("backend-url"))" />
    </inbound>
    
    <backend>
        <retry condition="@(context.Response.StatusCode == 429)" count="2" interval="0">
            <set-variable name="norway-throttle" value="@(
                DateTimeOffset.UtcNow.AddSeconds(
                    double.Parse(context.Response.Headers.GetValueOrDefault("Retry-After", "10"))
                ).ToUnixTimeSeconds())" />
            <set-backend-service base-url="https://aoai-sweden.openai.azure.com" />
            <forward-request />
        </retry>
    </backend>
</policies>

Norsk offentlig sektor

• SLA-implikasjoner: Standard Azure OpenAI deployments har ingen latens-SLA — 429-feil er forventet atferd under høy belastning. Dokumenter dette i tjenesteavtaler med interne brukere.
• Kvoteplanlegging: Statlige organisasjoner bør planlegge TPM-kvote basert på forventet bruksmønster med 30-50% margin. Kvoteøkninger kan ta tid å behandle.
• Multi-region compliance: Ved failover til andre regioner, sørg for at databehandleravtale dekker alle regioner. For sensitivt innhold, bruk kun EU-baserte regioner.
• Overvåking: Sett opp Azure Monitor-alerts for throttle rate > 5% og utilization > 80% for proaktiv kvotejustering.
• Beredskap: Ha en eskaleringsplan for kvoteøkninger som inkluderer kontaktinformasjon for Microsoft-support.

Beslutningsrammeverk

Scenario	Anbefaling	Begrunnelse
Sporadisk throttling (<5%)	Innebygd SDK retry	Tilstrekkelig for lav frekvens
Hyppig throttling (5-20%)	Øk kvote + multi-region failover	Kvoten er for lav for trafikken
Kritisk tjeneste, null toleranse	PTU deployment	Garantert kapasitet
Variabel trafikk med peaks	APIM med token rate limiting	Jevner ut trafikkmønstre
Multi-tenant applikasjon	Per-tenant rate limiting i APIM	Fair share mellom brukere

Referanser

• Manage Azure OpenAI quota — Kvotehåndtering
• Azure OpenAI quotas and limits — Grenser per modell
• Azure OpenAI SDK retry handling — SDK retry-konfigurasjon
• Use a gateway in front of multiple Azure OpenAI deployments or instances — Multi-region gateway (Azure OpenAI i Foundry Models) — Verified (MCP 2026-06-19)

For Cosmo

• Bruk denne referansen når kunden opplever 429-feil, planlegger kvotestrategi, eller designer multi-region failover for Azure OpenAI.
• Alltid sjekk og respekter Retry-After headeren — SDK-ene gjør dette automatisk, men custom-klienter må implementere det.
• Multi-region failover er den mest robuste løsningen: prioriter Norway East → Sweden Central → West Europe for norske kunder.
• PTU eliminerer rate limiting helt (innenfor tildelt kapasitet) — anbefal for forretningskritiske workloads.
• Proaktiv kvotemonitorering er billigere enn reaktiv feilhåndtering — sett opp alerts FØR throttling oppstår.