open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity
ms-ai-architect/skills/ms-ai-security/references/performance-scalability/rate-limit-management.md
Kjell Tore Guttormsen ddce43d8b2 feat(ms-ai-architect): Spor 1 — Port-1-substrat migrert på 4 ikke-advisor-skills (243 Source + 327 Type + 325 TOC + stale-verified poison fjernet) [skip-docs] 
Steg 9 (R4): unified migrate-corpus.mjs --write over engineering/governance/
infrastructure/security. 327 filer mutert, verified=null, prosa byte-identisk
(fra første ## seksjon), advisor urørt (0 endringer).

To applier-fixes oppdaget under kjøring (TDD, RED→GREEN):
- insertHeaderFields: anker faller nå tilbake når en meta-linje selv passerer
  500B (2 filer pakket et avsnitt i **Status:** → Type/Source landet utenfor
  scan-vinduet, applierens post-write-assertion fanget + restaurerte).
- normalizeStaleVerified: fjerner nå ALLE stale non-date **Verified:** i
  500B-vinduet, inkl. stray body-dup rett under --- (9 mlops-genaiops-filer var
  ellers falskt "verified"/fresh, droppet fra worklist). Operatør-godkjent
  utvidelse av carve-out; kun stray metadata-linjer, aldri prosa.

test-transform-criterion: precondition oppdatert til post-migrasjons-sannhet
(fila bærer nå Source). Suite 728/728 grønn.
2026-07-04 10:19:11 +02:00

504 lines
19 KiB
Markdown
Raw Blame History

# Rate Limit Management
**Last updated:** 2026-06-19 | Verified: MCP 2026-06-19
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference
**Source:** https://learn.microsoft.com/azure/foundry/openai/quotas-limits
---
## Innhold
- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Exponential Backoff Implementation](#exponential-backoff-implementation)
- [Quota Request Process](#quota-request-process)
- [Multi-Region Failover](#multi-region-failover)
- [Usage Monitoring](#usage-monitoring)
- [Gateway Multi-Backend som Rate Limit-strategi (oppdatert 2026-06-19)](#gateway-multi-backend-som-rate-limit-strategi-oppdatert-2026-06-19)
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
- [Beslutningsrammeverk](#beslutningsrammeverk)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)
## Introduksjon
Azure OpenAI bruker to rate limit-mekanismer: Tokens-per-Minute (TPM) og Requests-per-Minute (RPM). Når en av disse grensene overskrides, returnerer tjenesten HTTP 429 (Too Many Requests) med en `Retry-After` header som angir hvor mange sekunder klienten bør vente. For Standard deployments er rate limits direkte koblet til den tildelte kvoten, mens Provisioned Throughput (PTU) deployments returnerer 429 når utilization overstiger 100%.
Rate limit management er en av de mest kritiske aspektene ved produksjonsdrift av Azure OpenAI. Uten robust håndtering vil brukere oppleve sporadiske feil, og applikasjonen kan miste forespørsler under belastningstopper. Microsofts offisielle SDK-er (Python og JavaScript) har innebygd retry-logikk med eksponentiell backoff, men dette dekker kun grunnleggende scenarier. For produksjonsarkitekturer trengs mer sofistikerte strategier som multi-region failover, proaktiv throttling og quota monitoring.
For norsk offentlig sektor, der AI-tjenester kan være forretningskritiske for saksbehandling, er det avgjørende å ha en veldefinert strategi for rate limit management som sikrer at tjenesten er tilgjengelig selv under belastningstopper.
## Kjernekomponenter
| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| TPM/RPM Quota | Rate limiting per deployment | Azure OpenAI |
| Retry-After header | Server-side ventetid-instruksjon | HTTP 429 respons |
| Azure APIM | Gateway med rate limiting policies | Azure API Management |
| Circuit Breaker | Forhindre kaskade-feil | APIM / custom |
| Quota Management API | Programmatisk kvotejustering | Azure Management REST API |
| Azure Monitor | Rate limit-metrikker og alerting | Azure Monitor |
## Exponential Backoff Implementation
### Python SDK innebygd retry
```python
from openai import AzureOpenAI
# SDK har innebygd retry med exponential backoff
client = AzureOpenAI(
azure_endpoint="https://my-aoai.openai.azure.com",
api_key="...",
api_version="2024-10-21",
max_retries=3, # Default: 2
timeout=120.0 # Total timeout i sekunder
)
# Per-request override
response = client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": "Hello"}],
extra_headers={"max_retries": "5"} # Maks 5 forsøk for denne
)
```
### Custom retry med respekt for Retry-After
```python
import asyncio
import time
import random
from openai import AsyncAzureOpenAI, RateLimitError, APIError
class RateLimitHandler:
"""Advanced rate limit handling with exponential backoff."""
def __init__(
self,
client: AsyncAzureOpenAI,
max_retries: int = 5,
base_delay: float = 1.0,
max_delay: float = 60.0,
jitter: bool = True
):
self.client = client
self.max_retries = max_retries
self.base_delay = base_delay
self.max_delay = max_delay
self.jitter = jitter
self._consecutive_429s = 0
async def chat_completion(self, **kwargs) -> dict:
"""Execute chat completion with advanced retry logic."""
last_exception = None
for attempt in range(self.max_retries + 1):
try:
response = await self.client.chat.completions.create(**kwargs)
self._consecutive_429s = 0 # Reset on success
return response
except RateLimitError as e:
self._consecutive_429s += 1
last_exception = e
# Respekter Retry-After header
retry_after = getattr(e, 'retry_after', None)
if retry_after:
delay = float(retry_after)
else:
# Exponential backoff: 1s, 2s, 4s, 8s, 16s...
delay = min(
self.base_delay * (2 ** attempt),
self.max_delay
)
# Legg til jitter for å unngå thundering herd
if self.jitter:
delay *= (0.5 + random.random())
print(f"Rate limited (attempt {attempt + 1}/"
f"{self.max_retries}). "
f"Waiting {delay:.1f}s...")
await asyncio.sleep(delay)
except APIError as e:
if e.status_code and e.status_code >= 500:
# Server error — retry
delay = self.base_delay * (2 ** attempt)
await asyncio.sleep(delay)
last_exception = e
else:
raise # Client error — ikke retry
raise last_exception # Alle forsøk brukt opp
@property
def is_throttled(self) -> bool:
"""Check if we're currently experiencing throttling."""
return self._consecutive_429s >= 3
```
### .NET Polly-basert retry
```csharp
using Polly;
using Polly.Retry;
// Konfigurer retry policy med Polly
var retryPolicy = Policy
.Handle<Azure.RequestFailedException>(ex => ex.Status == 429)
.Or<Azure.RequestFailedException>(ex => ex.Status >= 500)
.WaitAndRetryAsync(
retryCount: 5,
sleepDurationProvider: (retryAttempt, exception, context) =>
{
// Bruk Retry-After header hvis tilgjengelig
if (exception is Azure.RequestFailedException rfEx)
{
var retryAfter = rfEx.GetRawResponse()?
.Headers.TryGetValue("Retry-After", out var value)
== true ? value : null;
if (retryAfter != null &&
double.TryParse(retryAfter, out var seconds))
{
return TimeSpan.FromSeconds(seconds);
}
}
// Fallback: exponential backoff med jitter
var baseDelay = TimeSpan.FromSeconds(Math.Pow(2, retryAttempt));
var jitter = TimeSpan.FromMilliseconds(
Random.Shared.Next(0, 1000));
return baseDelay + jitter;
},
onRetryAsync: (exception, timespan, retryAttempt, context) =>
{
Console.WriteLine(
$"Retry {retryAttempt} after {timespan.TotalSeconds:F1}s "
+ $"due to {exception.Message}");
return Task.CompletedTask;
}
);
```
## Quota Request Process
### Overvåk og juster kvote programmatisk
```python
import requests
def get_quota_usage(
subscription_id: str,
resource_group: str,
account_name: str,
access_token: str
) -> dict:
"""Get current quota usage for Azure OpenAI deployments."""
url = (
f"https://management.azure.com/subscriptions/{subscription_id}"
f"/resourceGroups/{resource_group}"
f"/providers/Microsoft.CognitiveServices"
f"/accounts/{account_name}"
f"/deployments?api-version=2023-05-01"
)
headers = {"Authorization": f"Bearer {access_token}"}
response = requests.get(url, headers=headers)
deployments = response.json()["value"]
usage = []
for d in deployments:
props = d["properties"]
usage.append({
"deployment": d["name"],
"model": props["model"]["name"],
"tpm_allocated": props.get("rateLimits", [{}])[0].get(
"count", 0) if props.get("rateLimits") else 0,
"sku": props.get("sku", {}).get("name", "unknown")
})
return usage
def update_deployment_quota(
subscription_id: str,
resource_group: str,
account_name: str,
deployment_name: str,
new_tpm: int,
access_token: str
):
"""Update TPM quota for a deployment."""
url = (
f"https://management.azure.com/subscriptions/{subscription_id}"
f"/resourceGroups/{resource_group}"
f"/providers/Microsoft.CognitiveServices"
f"/accounts/{account_name}"
f"/deployments/{deployment_name}?api-version=2023-05-01"
)
body = {
"sku": {
"name": "Standard",
"capacity": new_tpm // 1000 # TPM i tusen-enheter
}
}
headers = {
"Authorization": f"Bearer {access_token}",
"Content-Type": "application/json"
}
response = requests.patch(url, json=body, headers=headers)
return response.json()
```
## Multi-Region Failover
### Automatisk failover ved rate limiting
```python
from dataclasses import dataclass, field
from typing import Optional
import time
@dataclass
class RegionalEndpoint:
region: str
endpoint: str
api_key: str
priority: int = 1
is_healthy: bool = True
throttled_until: float = 0
consecutive_errors: int = 0
class MultiRegionRateLimitHandler:
"""Handle rate limits by failing over to other regions."""
def __init__(self, endpoints: list[RegionalEndpoint]):
self.endpoints = sorted(endpoints, key=lambda e: e.priority)
def _get_available_endpoint(self) -> Optional[RegionalEndpoint]:
"""Get best available endpoint respecting throttle state."""
now = time.time()
for ep in self.endpoints:
if ep.is_healthy and now > ep.throttled_until:
return ep
# Alle throttled — returner den som er tidligst klar
available = sorted(
self.endpoints,
key=lambda e: e.throttled_until
)
return available[0] if available else None
async def execute(self, **kwargs) -> dict:
"""Execute request with multi-region failover."""
for attempt in range(len(self.endpoints) * 2):
endpoint = self._get_available_endpoint()
if not endpoint:
raise Exception("No endpoints available")
# Vent hvis throttled
wait_time = max(0, endpoint.throttled_until - time.time())
if wait_time > 0:
await asyncio.sleep(wait_time)
try:
client = AsyncAzureOpenAI(
azure_endpoint=endpoint.endpoint,
api_key=endpoint.api_key,
api_version="2024-10-21",
max_retries=0 # Vi håndterer retry selv
)
response = await client.chat.completions.create(**kwargs)
endpoint.consecutive_errors = 0
endpoint.is_healthy = True
return response
except RateLimitError as e:
retry_after = getattr(e, 'retry_after', 10)
endpoint.throttled_until = time.time() + float(retry_after)
endpoint.consecutive_errors += 1
print(f"Region {endpoint.region} throttled for "
f"{retry_after}s. Trying next region...")
continue
except APIError as e:
if e.status_code >= 500:
endpoint.consecutive_errors += 1
if endpoint.consecutive_errors >= 3:
endpoint.is_healthy = False
continue
raise
raise Exception("All regions exhausted")
# Konfigurasjon
handler = MultiRegionRateLimitHandler([
RegionalEndpoint(
region="norwayeast",
endpoint="https://aoai-norway.openai.azure.com",
api_key="...",
priority=1
),
RegionalEndpoint(
region="swedencentral",
endpoint="https://aoai-sweden.openai.azure.com",
api_key="...",
priority=2
),
RegionalEndpoint(
region="westeurope",
endpoint="https://aoai-westeu.openai.azure.com",
api_key="...",
priority=3
)
])
```
## Usage Monitoring
### KQL-spørringer for rate limit monitoring
```python
# Overvåk throttling i Azure Monitor
THROTTLE_MONITORING = """
AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
TotalRequests = count(),
Successful = countif(StatusCode == "200"),
Throttled = countif(StatusCode == "429"),
ServerErrors = countif(StatusCode startswith "5")
by bin(TimeGenerated, 5m), Resource
| extend
ThrottleRate = round(Throttled * 100.0 / TotalRequests, 2),
ErrorRate = round(ServerErrors * 100.0 / TotalRequests, 2)
| where ThrottleRate > 0 or ErrorRate > 0
| order by TimeGenerated desc
"""
# Alert: Varsle når throttle rate overstiger terskel
THROTTLE_ALERT = """
AzureMetrics
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
Total = count(),
Throttled = countif(StatusCode == "429")
by bin(TimeGenerated, 5m)
| extend ThrottleRate = Throttled * 100.0 / Total
| where ThrottleRate > 10
"""
# Quota utilization trend
QUOTA_UTILIZATION = """
AzureMetrics
| where MetricName in ("ProcessedPromptTokens", "GeneratedCompletionTokens")
| summarize
PromptTPM = sumif(Total, MetricName == "ProcessedPromptTokens"),
CompletionTPM = sumif(Total, MetricName == "GeneratedCompletionTokens")
by bin(TimeGenerated, 1m)
| extend TotalTPM = PromptTPM + CompletionTPM
| order by TimeGenerated desc
"""
```
## Gateway Multi-Backend som Rate Limit-strategi (oppdatert 2026-06-19)
Microsoft dokumenterer multi-backend gateway som den anbefalte arkitekturmønsteret for rate limit management — primært via Azure API Management:
### Anbefalte topologier for rate limit-distribusjon
> **Viktig:** Standard-kvote er subscription-nivå, ikke Azure OpenAI-instansnivå. Load balancing mellom standard-instanser i samme subscription gir IKKE høyere gjennomstrømning — bruk separate subscriptions eller global/data zone deployments for reell kvoteutvidelse. — Verified (MCP 2026-06-19)
| Topologi | Kvote-kapasitet | Kompleksitet | Anbefalt for |
|----------|----------------|--------------|------------|
| Single instance | Baseline TPM | Lav | Utvikling, lav trafikk |
| Multi-backend, single region | 2-5x baseline | Medium | Produksjon, standard |
| Multi-subscription | 5-20x baseline | Høy | Høy trafikk enterprise |
| Multi-region | Nær ubegrenset | Høy | Kritisk infrastruktur |
### APIM-basert rate limit distribusjon
```xml
<!-- APIM Policy: Distribuer rate limit på tvers av backends -->
<policies>
<inbound>
<base />
<!-- Token-based rate limiting i APIM (avlaster Azure OpenAI) -->
<azure-openai-token-limit
counter-key="@(context.Request.Headers.GetValueOrDefault("x-client-id", "default"))"
tokens-per-minute="10000"
estimate-prompt-tokens="true"
tokens-consumed-variable-name="consumed-tokens"
remaining-tokens-variable-name="remaining-tokens" />
<!-- Velg backend basert på tilgjengelighet -->
<set-variable name="backend-url" value="@{
// Prioritert liste: prøv Norway East, fallback til Sweden Central
if (context.Variables.GetValueOrDefault<int>("norway-throttle") < DateTimeOffset.UtcNow.ToUnixTimeSeconds())
return "https://aoai-norway.openai.azure.com";
return "https://aoai-sweden.openai.azure.com";
}" />
<set-backend-service base-url="@(context.Variables.GetValueOrDefault<string>("backend-url"))" />
</inbound>
<backend>
<retry condition="@(context.Response.StatusCode == 429)" count="2" interval="0">
<set-variable name="norway-throttle" value="@(
DateTimeOffset.UtcNow.AddSeconds(
double.Parse(context.Response.Headers.GetValueOrDefault("Retry-After", "10"))
).ToUnixTimeSeconds())" />
<set-backend-service base-url="https://aoai-sweden.openai.azure.com" />
<forward-request />
</retry>
</backend>
</policies>
```
## Norsk offentlig sektor
- **SLA-implikasjoner**: Standard Azure OpenAI deployments har ingen latens-SLA — 429-feil er forventet atferd under høy belastning. Dokumenter dette i tjenesteavtaler med interne brukere.
- **Kvoteplanlegging**: Statlige organisasjoner bør planlegge TPM-kvote basert på forventet bruksmønster med 30-50% margin. Kvoteøkninger kan ta tid å behandle.
- **Multi-region compliance**: Ved failover til andre regioner, sørg for at databehandleravtale dekker alle regioner. For sensitivt innhold, bruk kun EU-baserte regioner.
- **Overvåking**: Sett opp Azure Monitor-alerts for throttle rate > 5% og utilization > 80% for proaktiv kvotejustering.
- **Beredskap**: Ha en eskaleringsplan for kvoteøkninger som inkluderer kontaktinformasjon for Microsoft-support.
## Beslutningsrammeverk
| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Sporadisk throttling (<5%) | Innebygd SDK retry | Tilstrekkelig for lav frekvens |
| Hyppig throttling (5-20%) | Øk kvote + multi-region failover | Kvoten er for lav for trafikken |
| Kritisk tjeneste, null toleranse | PTU deployment | Garantert kapasitet |
| Variabel trafikk med peaks | APIM med token rate limiting | Jevner ut trafikkmønstre |
| Multi-tenant applikasjon | Per-tenant rate limiting i APIM | Fair share mellom brukere |
## Referanser
- [Manage Azure OpenAI quota](https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota) — Kvotehåndtering
- [Azure OpenAI quotas and limits](https://learn.microsoft.com/azure/foundry/openai/quotas-limits) — Grenser per modell
- [Azure OpenAI SDK retry handling](https://learn.microsoft.com/azure/foundry/openai/supported-languages) — SDK retry-konfigurasjon
- [Use a gateway in front of multiple Azure OpenAI deployments or instances](https://learn.microsoft.com/azure/architecture/ai-ml/guide/azure-openai-gateway-multi-backend) — Multi-region gateway (Azure OpenAI i Foundry Models) — Verified (MCP 2026-06-19)
## For Cosmo
- **Bruk denne referansen** når kunden opplever 429-feil, planlegger kvotestrategi, eller designer multi-region failover for Azure OpenAI.
- Alltid sjekk og respekter `Retry-After` headeren — SDK-ene gjør dette automatisk, men custom-klienter må implementere det.
- Multi-region failover er den mest robuste løsningen: prioriter Norway East → Sweden Central → West Europe for norske kunder.
- PTU eliminerer rate limiting helt (innenfor tildelt kapasitet) — anbefal for forretningskritiske workloads.
- Proaktiv kvotemonitorering er billigere enn reaktiv feilhåndtering — sett opp alerts FØR throttling oppstår.
Reference in a new issue View git blame Copy permalink