Kjell Tore Guttormsen
ddce43d8b2
Steg 9 (R4): unified migrate-corpus.mjs --write over engineering/governance/ infrastructure/security. 327 filer mutert, verified=null, prosa byte-identisk (fra første ## seksjon), advisor urørt (0 endringer). To applier-fixes oppdaget under kjøring (TDD, RED→GREEN): - insertHeaderFields: anker faller nå tilbake når en meta-linje selv passerer 500B (2 filer pakket et avsnitt i **Status:** → Type/Source landet utenfor scan-vinduet, applierens post-write-assertion fanget + restaurerte). - normalizeStaleVerified: fjerner nå ALLE stale non-date **Verified:** i 500B-vinduet, inkl. stray body-dup rett under --- (9 mlops-genaiops-filer var ellers falskt "verified"/fresh, droppet fra worklist). Operatør-godkjent utvidelse av carve-out; kun stray metadata-linjer, aldri prosa. test-transform-criterion: precondition oppdatert til post-migrasjons-sannhet (fila bærer nå Source). Suite 728/728 grønn.
446 lines
16 KiB
Markdown
446 lines
16 KiB
Markdown
# Concurrent Request Optimization
|
|
|
|
**Last updated:** 2026-06-24
|
|
**Status:** GA
|
|
**Category:** Performance & Scalability
|
|
**Type:** reference
|
|
**Source:** https://learn.microsoft.com/azure/foundry/openai/how-to/latency
|
|
|
|
---
|
|
|
|
## Innhold
|
|
|
|
- [Introduksjon](#introduksjon)
|
|
- [Kjernekomponenter](#kjernekomponenter)
|
|
- [Concurrency Level Tuning](#concurrency-level-tuning)
|
|
- [Request Queueing Strategies](#request-queueing-strategies)
|
|
- [Deadlock Prevention](#deadlock-prevention)
|
|
- [Resource Contention Resolution](#resource-contention-resolution)
|
|
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
|
|
- [Beslutningsrammeverk](#beslutningsrammeverk)
|
|
- [Referanser](#referanser)
|
|
- [For Cosmo](#for-cosmo)
|
|
|
|
## Introduksjon
|
|
|
|
Concurrent request optimization handler om å maksimere antall samtidige forespørsler mot Azure OpenAI uten å overbelaste tjenesten eller miste forespørsler. Den optimale graden av samtidighet avhenger av deployment-type (Standard vs. PTU), tildelt kvote (TPM/RPM), modellens responstid og klientens evne til å håndtere parallelle forbindelser.
|
|
|
|
For Standard deployments bestemmer RPM-kvoten den harde grensen for samtidige forespørsler per minutt, men den faktiske grensen er ofte lavere fordi lange forespørsler blokkerer kapasitet. For PTU deployments er grensen definert av utilization — når prosessert kapasitet nærmer seg 100% av tildelte PTUs, begynner 429-feil. I begge tilfeller er nøkkelen å finne sweet spot der throughput er maksimert uten overdreven throttling.
|
|
|
|
For norsk offentlig sektor, der AI-applikasjoner kan betjene hundrevis av samtidige saksbehandlere, er concurrent request optimization avgjørende for å sikre jevn brukeropplevelse uten at noen opplever timeout eller feil.
|
|
|
|
## Kjernekomponenter
|
|
|
|
| Komponent | Formål | Teknologi |
|
|
|-----------|--------|-----------|
|
|
| Semaphore | Begrens concurrent requests klient-side | asyncio / SemaphoreSlim |
|
|
| Token Bucket | Rate limiting med burst-støtte | Custom / APIM |
|
|
| Connection Pool | Gjenbruk HTTP-forbindelser | HttpClient / aiohttp |
|
|
| Circuit Breaker | Forhindre kaskade ved overbelastning | Polly / custom |
|
|
| Queue | Buffer forespørsler under peak | Service Bus / in-memory |
|
|
|
|
## Concurrency Level Tuning
|
|
|
|
### Finn optimal concurrency
|
|
|
|
```python
|
|
import asyncio
|
|
import time
|
|
from openai import AsyncAzureOpenAI, RateLimitError
|
|
|
|
async def find_optimal_concurrency(
|
|
client: AsyncAzureOpenAI,
|
|
model: str = "gpt-4o",
|
|
test_prompt: str = "Oppsummer dette kort.",
|
|
max_tokens: int = 200,
|
|
test_levels: list[int] = None,
|
|
requests_per_level: int = 50
|
|
) -> dict:
|
|
"""Find optimal concurrency level through progressive testing."""
|
|
if test_levels is None:
|
|
test_levels = [1, 5, 10, 20, 30, 50, 75, 100]
|
|
|
|
results = []
|
|
|
|
for concurrency in test_levels:
|
|
semaphore = asyncio.Semaphore(concurrency)
|
|
stats = {"success": 0, "throttled": 0, "errors": 0, "latencies": []}
|
|
|
|
async def send_one():
|
|
async with semaphore:
|
|
start = time.time()
|
|
try:
|
|
await client.chat.completions.create(
|
|
model=model,
|
|
messages=[{"role": "user", "content": test_prompt}],
|
|
max_tokens=max_tokens
|
|
)
|
|
stats["latencies"].append(
|
|
(time.time() - start) * 1000)
|
|
stats["success"] += 1
|
|
except RateLimitError:
|
|
stats["throttled"] += 1
|
|
except Exception:
|
|
stats["errors"] += 1
|
|
|
|
start = time.time()
|
|
await asyncio.gather(
|
|
*[send_one() for _ in range(requests_per_level)])
|
|
duration = time.time() - start
|
|
|
|
total = stats["success"] + stats["throttled"] + stats["errors"]
|
|
throttle_rate = stats["throttled"] / max(total, 1) * 100
|
|
|
|
result = {
|
|
"concurrency": concurrency,
|
|
"throughput_rps": round(stats["success"] / duration, 2),
|
|
"throttle_rate_pct": round(throttle_rate, 1),
|
|
"p50_ms": round(sorted(stats["latencies"])[
|
|
len(stats["latencies"]) // 2], 0)
|
|
if stats["latencies"] else 0,
|
|
"p95_ms": round(sorted(stats["latencies"])[
|
|
int(len(stats["latencies"]) * 0.95)], 0)
|
|
if stats["latencies"] else 0,
|
|
"success": stats["success"],
|
|
"throttled": stats["throttled"]
|
|
}
|
|
results.append(result)
|
|
|
|
print(f"Concurrency {concurrency}: "
|
|
f"{result['throughput_rps']} RPS, "
|
|
f"{throttle_rate:.1f}% throttled, "
|
|
f"P50={result['p50_ms']}ms")
|
|
|
|
# Stopp hvis throttle rate er for høy
|
|
if throttle_rate > 30:
|
|
print(f"Stopping: throttle rate too high at {concurrency}")
|
|
break
|
|
|
|
# Finn optimal: best throughput med <5% throttling
|
|
acceptable = [r for r in results if r["throttle_rate_pct"] < 5]
|
|
if acceptable:
|
|
optimal = max(acceptable, key=lambda r: r["throughput_rps"])
|
|
else:
|
|
optimal = results[0]
|
|
|
|
return {
|
|
"optimal_concurrency": optimal["concurrency"],
|
|
"optimal_throughput_rps": optimal["throughput_rps"],
|
|
"all_results": results
|
|
}
|
|
```
|
|
|
|
### Adaptive concurrency control
|
|
|
|
```python
|
|
class AdaptiveConcurrencyController:
|
|
"""Dynamically adjust concurrency based on response signals."""
|
|
|
|
def __init__(
|
|
self,
|
|
initial_concurrency: int = 10,
|
|
min_concurrency: int = 1,
|
|
max_concurrency: int = 100,
|
|
increase_threshold: float = 0.02, # Øk hvis <2% throttled
|
|
decrease_threshold: float = 0.10, # Reduser hvis >10% throttled
|
|
adjustment_interval: float = 10.0 # Juster hvert 10. sekund
|
|
):
|
|
self.current = initial_concurrency
|
|
self.min_concurrency = min_concurrency
|
|
self.max_concurrency = max_concurrency
|
|
self.increase_threshold = increase_threshold
|
|
self.decrease_threshold = decrease_threshold
|
|
self.adjustment_interval = adjustment_interval
|
|
self._semaphore = asyncio.Semaphore(initial_concurrency)
|
|
self._window_success = 0
|
|
self._window_throttled = 0
|
|
self._last_adjustment = time.time()
|
|
|
|
async def acquire(self):
|
|
"""Acquire a concurrency slot."""
|
|
await self._semaphore.acquire()
|
|
|
|
def release(self, was_throttled: bool = False):
|
|
"""Release slot and record outcome."""
|
|
self._semaphore.release()
|
|
if was_throttled:
|
|
self._window_throttled += 1
|
|
else:
|
|
self._window_success += 1
|
|
|
|
self._maybe_adjust()
|
|
|
|
def _maybe_adjust(self):
|
|
"""Periodically adjust concurrency."""
|
|
now = time.time()
|
|
if now - self._last_adjustment < self.adjustment_interval:
|
|
return
|
|
|
|
total = self._window_success + self._window_throttled
|
|
if total < 10: # Ikke nok data
|
|
return
|
|
|
|
throttle_rate = self._window_throttled / total
|
|
|
|
old = self.current
|
|
if throttle_rate < self.increase_threshold:
|
|
# Trygt å øke
|
|
self.current = min(
|
|
self.current + max(1, self.current // 10),
|
|
self.max_concurrency)
|
|
elif throttle_rate > self.decrease_threshold:
|
|
# Må redusere
|
|
self.current = max(
|
|
self.current - max(1, self.current // 5),
|
|
self.min_concurrency)
|
|
|
|
if self.current != old:
|
|
# Opprett ny semaphore med justert limit
|
|
self._semaphore = asyncio.Semaphore(self.current)
|
|
print(f"Concurrency adjusted: {old} → {self.current} "
|
|
f"(throttle rate: {throttle_rate:.1%})")
|
|
|
|
self._window_success = 0
|
|
self._window_throttled = 0
|
|
self._last_adjustment = now
|
|
```
|
|
|
|
## Request Queueing Strategies
|
|
|
|
### Priority queue for AI-forespørsler
|
|
|
|
```python
|
|
import asyncio
|
|
import heapq
|
|
from enum import IntEnum
|
|
from dataclasses import dataclass, field
|
|
from typing import Any
|
|
|
|
class Priority(IntEnum):
|
|
URGENT = 1
|
|
HIGH = 2
|
|
NORMAL = 3
|
|
LOW = 4
|
|
BACKGROUND = 5
|
|
|
|
@dataclass(order=True)
|
|
class PrioritizedRequest:
|
|
priority: int
|
|
timestamp: float = field(compare=True)
|
|
request: Any = field(compare=False)
|
|
future: asyncio.Future = field(compare=False, repr=False)
|
|
|
|
class PriorityRequestQueue:
|
|
"""Priority queue for AI requests with concurrency control."""
|
|
|
|
def __init__(self, max_concurrent: int = 20):
|
|
self._queue: list[PrioritizedRequest] = []
|
|
self._semaphore = asyncio.Semaphore(max_concurrent)
|
|
self._processing = True
|
|
|
|
async def submit(
|
|
self,
|
|
request: dict,
|
|
priority: Priority = Priority.NORMAL
|
|
) -> asyncio.Future:
|
|
"""Submit request with priority. Returns future."""
|
|
future = asyncio.get_event_loop().create_future()
|
|
item = PrioritizedRequest(
|
|
priority=priority.value,
|
|
timestamp=time.time(),
|
|
request=request,
|
|
future=future
|
|
)
|
|
heapq.heappush(self._queue, item)
|
|
return future
|
|
|
|
async def process_loop(self, process_fn):
|
|
"""Continuously process queued requests."""
|
|
while self._processing:
|
|
if not self._queue:
|
|
await asyncio.sleep(0.01)
|
|
continue
|
|
|
|
await self._semaphore.acquire()
|
|
item = heapq.heappop(self._queue)
|
|
|
|
asyncio.create_task(
|
|
self._process_item(item, process_fn))
|
|
|
|
async def _process_item(self, item, process_fn):
|
|
try:
|
|
result = await process_fn(item.request)
|
|
if not item.future.done():
|
|
item.future.set_result(result)
|
|
except Exception as e:
|
|
if not item.future.done():
|
|
item.future.set_exception(e)
|
|
finally:
|
|
self._semaphore.release()
|
|
```
|
|
|
|
## Deadlock Prevention
|
|
|
|
### Unngå resource starvation
|
|
|
|
```python
|
|
class DeadlockPreventionWrapper:
|
|
"""Prevent deadlocks in concurrent AI request processing."""
|
|
|
|
def __init__(
|
|
self,
|
|
client: AsyncAzureOpenAI,
|
|
max_concurrent: int = 20,
|
|
request_timeout: float = 120.0,
|
|
starvation_timeout: float = 300.0
|
|
):
|
|
self.client = client
|
|
self.semaphore = asyncio.Semaphore(max_concurrent)
|
|
self.request_timeout = request_timeout
|
|
self.starvation_timeout = starvation_timeout
|
|
self._active_requests: dict[str, float] = {}
|
|
|
|
async def execute(self, request_id: str, **kwargs):
|
|
"""Execute with timeout and starvation protection."""
|
|
# Timeout på semaphore acquire — forhindrer deadlock
|
|
try:
|
|
await asyncio.wait_for(
|
|
self.semaphore.acquire(),
|
|
timeout=self.starvation_timeout
|
|
)
|
|
except asyncio.TimeoutError:
|
|
raise TimeoutError(
|
|
f"Request {request_id} starved waiting for "
|
|
f"concurrency slot for {self.starvation_timeout}s. "
|
|
f"Consider increasing max_concurrent or reducing "
|
|
f"request volume.")
|
|
|
|
self._active_requests[request_id] = time.time()
|
|
|
|
try:
|
|
# Timeout på selve forespørselen
|
|
result = await asyncio.wait_for(
|
|
self.client.chat.completions.create(**kwargs),
|
|
timeout=self.request_timeout
|
|
)
|
|
return result
|
|
|
|
except asyncio.TimeoutError:
|
|
raise TimeoutError(
|
|
f"Request {request_id} timed out after "
|
|
f"{self.request_timeout}s")
|
|
|
|
finally:
|
|
self._active_requests.pop(request_id, None)
|
|
self.semaphore.release()
|
|
|
|
@property
|
|
def active_count(self) -> int:
|
|
return len(self._active_requests)
|
|
|
|
def get_stuck_requests(self, threshold_seconds: float = 60) -> list:
|
|
"""Identify requests that may be stuck."""
|
|
now = time.time()
|
|
return [
|
|
{"id": rid, "age_seconds": round(now - start, 1)}
|
|
for rid, start in self._active_requests.items()
|
|
if now - start > threshold_seconds
|
|
]
|
|
```
|
|
|
|
## Resource Contention Resolution
|
|
|
|
### Token bucket for fair scheduling
|
|
|
|
```python
|
|
import time
|
|
|
|
class TokenBucket:
|
|
"""Token bucket rate limiter for fair resource sharing."""
|
|
|
|
def __init__(
|
|
self,
|
|
tokens_per_second: float,
|
|
max_burst: int = 10
|
|
):
|
|
self.rate = tokens_per_second
|
|
self.max_burst = max_burst
|
|
self._tokens = max_burst
|
|
self._last_refill = time.time()
|
|
self._lock = asyncio.Lock()
|
|
|
|
async def acquire(self, tokens: int = 1) -> float:
|
|
"""Acquire tokens, waiting if necessary. Returns wait time."""
|
|
async with self._lock:
|
|
self._refill()
|
|
|
|
if self._tokens >= tokens:
|
|
self._tokens -= tokens
|
|
return 0
|
|
|
|
# Beregn ventetid
|
|
deficit = tokens - self._tokens
|
|
wait_time = deficit / self.rate
|
|
await asyncio.sleep(wait_time)
|
|
self._refill()
|
|
self._tokens -= tokens
|
|
return wait_time
|
|
|
|
def _refill(self):
|
|
now = time.time()
|
|
elapsed = now - self._last_refill
|
|
self._tokens = min(
|
|
self.max_burst,
|
|
self._tokens + elapsed * self.rate
|
|
)
|
|
self._last_refill = now
|
|
|
|
|
|
class FairScheduler:
|
|
"""Fair scheduling across multiple tenants/users."""
|
|
|
|
def __init__(self, total_rps: float, num_tenants: int):
|
|
self.total_rps = total_rps
|
|
per_tenant_rps = total_rps / num_tenants
|
|
self.buckets: dict[str, TokenBucket] = {}
|
|
self._default_rps = per_tenant_rps
|
|
|
|
def get_bucket(self, tenant_id: str) -> TokenBucket:
|
|
if tenant_id not in self.buckets:
|
|
self.buckets[tenant_id] = TokenBucket(
|
|
tokens_per_second=self._default_rps,
|
|
max_burst=int(self._default_rps * 2)
|
|
)
|
|
return self.buckets[tenant_id]
|
|
```
|
|
|
|
## Norsk offentlig sektor
|
|
|
|
- **Fair use**: I multi-tenant løsninger der flere enheter deler samme Azure OpenAI-deployment, bruk per-tenant rate limiting for å sikre rettferdig fordeling.
|
|
- **Brukeropplevelse**: Sett starvation timeout til maks ventetid brukere aksepterer (typisk 30-60 sekunder). Returner informativ feilmelding ved timeout.
|
|
- **Overvåking**: Logg concurrent request-nivå, kø-dybde og starvation-hendelser i Application Insights for kapasitetsplanlegging.
|
|
- **Skalering**: Planlegg for peak-perioder (morgen 08-10, etter lunsj 12-13) med høyere concurrent limits eller ekstra kvote.
|
|
|
|
## Beslutningsrammeverk
|
|
|
|
| Scenario | Anbefaling | Begrunnelse |
|
|
|----------|------------|-------------|
|
|
| Ukjent workload | Start med 10 concurrent, juster | Unngå initial throttling |
|
|
| Forutsigbar, jevn trafikk | Statisk semaphore på optimal nivå | Enklest å implementere |
|
|
| Variable peaks | Adaptive concurrency controller | Automatisk tilpasning |
|
|
| Multi-tenant | Priority queue + per-tenant bucket | Fair resource sharing |
|
|
| Kritisk latens | Lav concurrency + PTU | Forutsigbar responstid |
|
|
|
|
## Referanser
|
|
|
|
- [Manage Azure OpenAI quota](https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota) — RPM/TPM grenser
|
|
- [Performance and latency](https://learn.microsoft.com/azure/foundry/openai/how-to/latency) — Concurrent requests og throughput
|
|
- [Provisioned throughput](https://learn.microsoft.com/azure/foundry/openai/how-to/provisioned-get-started) — PTU utilization
|
|
|
|
## For Cosmo
|
|
|
|
- **Bruk denne referansen** når kunden opplever timeout, starvation eller ujevn ytelse i AI-applikasjoner med mange samtidige brukere.
|
|
- Start konservativt (10-20 concurrent) og øk gradvis mens du monitorerer throttle rate — aldri gå rett til 100 concurrent.
|
|
- Adaptive concurrency control er anbefalt for produksjon — statiske verdier fungerer dårlig når trafikkmønstre endres.
|
|
- Prioritetskøer er viktige for multi-tenant: sørg for at kritiske oppgaver (saksbehandler-beslutninger) ikke blokkeres av bakgrunnsjobber.
|
|
- Deadlock prevention med timeouts er obligatorisk — uten det kan en hengende forespørsel blokkere alle slots permanent.
|