# Concurrent Request Optimization

**Last updated:** 2026-06-24
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference
**Source:** https://learn.microsoft.com/azure/foundry/openai/how-to/latency

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Concurrency Level Tuning](#concurrency-level-tuning)
- [Request Queueing Strategies](#request-queueing-strategies)
- [Deadlock Prevention](#deadlock-prevention)
- [Resource Contention Resolution](#resource-contention-resolution)
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
- [Beslutningsrammeverk](#beslutningsrammeverk)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

Concurrent request optimization handler om å maksimere antall samtidige forespørsler mot Azure OpenAI uten å overbelaste tjenesten eller miste forespørsler. Den optimale graden av samtidighet avhenger av deployment-type (Standard vs. PTU), tildelt kvote (TPM/RPM), modellens responstid og klientens evne til å håndtere parallelle forbindelser.

For Standard deployments bestemmer RPM-kvoten den harde grensen for samtidige forespørsler per minutt, men den faktiske grensen er ofte lavere fordi lange forespørsler blokkerer kapasitet. For PTU deployments er grensen definert av utilization — når prosessert kapasitet nærmer seg 100% av tildelte PTUs, begynner 429-feil. I begge tilfeller er nøkkelen å finne sweet spot der throughput er maksimert uten overdreven throttling.

For norsk offentlig sektor, der AI-applikasjoner kan betjene hundrevis av samtidige saksbehandlere, er concurrent request optimization avgjørende for å sikre jevn brukeropplevelse uten at noen opplever timeout eller feil.

## Kjernekomponenter

| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| Semaphore | Begrens concurrent requests klient-side | asyncio / SemaphoreSlim |
| Token Bucket | Rate limiting med burst-støtte | Custom / APIM |
| Connection Pool | Gjenbruk HTTP-forbindelser | HttpClient / aiohttp |
| Circuit Breaker | Forhindre kaskade ved overbelastning | Polly / custom |
| Queue | Buffer forespørsler under peak | Service Bus / in-memory |

## Concurrency Level Tuning

### Finn optimal concurrency

```python
import asyncio
import time
from openai import AsyncAzureOpenAI, RateLimitError

async def find_optimal_concurrency(
    client: AsyncAzureOpenAI,
    model: str = "gpt-4o",
    test_prompt: str = "Oppsummer dette kort.",
    max_tokens: int = 200,
    test_levels: list[int] = None,
    requests_per_level: int = 50
) -> dict:
    """Find optimal concurrency level through progressive testing."""
    if test_levels is None:
        test_levels = [1, 5, 10, 20, 30, 50, 75, 100]

    results = []

    for concurrency in test_levels:
        semaphore = asyncio.Semaphore(concurrency)
        stats = {"success": 0, "throttled": 0, "errors": 0, "latencies": []}

        async def send_one():
            async with semaphore:
                start = time.time()
                try:
                    await client.chat.completions.create(
                        model=model,
                        messages=[{"role": "user", "content": test_prompt}],
                        max_tokens=max_tokens
                    )
                    stats["latencies"].append(
                        (time.time() - start) * 1000)
                    stats["success"] += 1
                except RateLimitError:
                    stats["throttled"] += 1
                except Exception:
                    stats["errors"] += 1

        start = time.time()
        await asyncio.gather(
            *[send_one() for _ in range(requests_per_level)])
        duration = time.time() - start

        total = stats["success"] + stats["throttled"] + stats["errors"]
        throttle_rate = stats["throttled"] / max(total, 1) * 100

        result = {
            "concurrency": concurrency,
            "throughput_rps": round(stats["success"] / duration, 2),
            "throttle_rate_pct": round(throttle_rate, 1),
            "p50_ms": round(sorted(stats["latencies"])[
                len(stats["latencies"]) // 2], 0)
                if stats["latencies"] else 0,
            "p95_ms": round(sorted(stats["latencies"])[
                int(len(stats["latencies"]) * 0.95)], 0)
                if stats["latencies"] else 0,
            "success": stats["success"],
            "throttled": stats["throttled"]
        }
        results.append(result)

        print(f"Concurrency {concurrency}: "
              f"{result['throughput_rps']} RPS, "
              f"{throttle_rate:.1f}% throttled, "
              f"P50={result['p50_ms']}ms")

        # Stopp hvis throttle rate er for høy
        if throttle_rate > 30:
            print(f"Stopping: throttle rate too high at {concurrency}")
            break

    # Finn optimal: best throughput med <5% throttling
    acceptable = [r for r in results if r["throttle_rate_pct"] < 5]
    if acceptable:
        optimal = max(acceptable, key=lambda r: r["throughput_rps"])
    else:
        optimal = results[0]

    return {
        "optimal_concurrency": optimal["concurrency"],
        "optimal_throughput_rps": optimal["throughput_rps"],
        "all_results": results
    }
```

### Adaptive concurrency control

```python
class AdaptiveConcurrencyController:
    """Dynamically adjust concurrency based on response signals."""

    def __init__(
        self,
        initial_concurrency: int = 10,
        min_concurrency: int = 1,
        max_concurrency: int = 100,
        increase_threshold: float = 0.02,  # Øk hvis <2% throttled
        decrease_threshold: float = 0.10,  # Reduser hvis >10% throttled
        adjustment_interval: float = 10.0   # Juster hvert 10. sekund
    ):
        self.current = initial_concurrency
        self.min_concurrency = min_concurrency
        self.max_concurrency = max_concurrency
        self.increase_threshold = increase_threshold
        self.decrease_threshold = decrease_threshold
        self.adjustment_interval = adjustment_interval
        self._semaphore = asyncio.Semaphore(initial_concurrency)
        self._window_success = 0
        self._window_throttled = 0
        self._last_adjustment = time.time()

    async def acquire(self):
        """Acquire a concurrency slot."""
        await self._semaphore.acquire()

    def release(self, was_throttled: bool = False):
        """Release slot and record outcome."""
        self._semaphore.release()
        if was_throttled:
            self._window_throttled += 1
        else:
            self._window_success += 1

        self._maybe_adjust()

    def _maybe_adjust(self):
        """Periodically adjust concurrency."""
        now = time.time()
        if now - self._last_adjustment < self.adjustment_interval:
            return

        total = self._window_success + self._window_throttled
        if total < 10:  # Ikke nok data
            return

        throttle_rate = self._window_throttled / total

        old = self.current
        if throttle_rate < self.increase_threshold:
            # Trygt å øke
            self.current = min(
                self.current + max(1, self.current // 10),
                self.max_concurrency)
        elif throttle_rate > self.decrease_threshold:
            # Må redusere
            self.current = max(
                self.current - max(1, self.current // 5),
                self.min_concurrency)

        if self.current != old:
            # Opprett ny semaphore med justert limit
            self._semaphore = asyncio.Semaphore(self.current)
            print(f"Concurrency adjusted: {old} → {self.current} "
                  f"(throttle rate: {throttle_rate:.1%})")

        self._window_success = 0
        self._window_throttled = 0
        self._last_adjustment = now
```

## Request Queueing Strategies

### Priority queue for AI-forespørsler

```python
import asyncio
import heapq
from enum import IntEnum
from dataclasses import dataclass, field
from typing import Any

class Priority(IntEnum):
    URGENT = 1
    HIGH = 2
    NORMAL = 3
    LOW = 4
    BACKGROUND = 5

@dataclass(order=True)
class PrioritizedRequest:
    priority: int
    timestamp: float = field(compare=True)
    request: Any = field(compare=False)
    future: asyncio.Future = field(compare=False, repr=False)

class PriorityRequestQueue:
    """Priority queue for AI requests with concurrency control."""

    def __init__(self, max_concurrent: int = 20):
        self._queue: list[PrioritizedRequest] = []
        self._semaphore = asyncio.Semaphore(max_concurrent)
        self._processing = True

    async def submit(
        self,
        request: dict,
        priority: Priority = Priority.NORMAL
    ) -> asyncio.Future:
        """Submit request with priority. Returns future."""
        future = asyncio.get_event_loop().create_future()
        item = PrioritizedRequest(
            priority=priority.value,
            timestamp=time.time(),
            request=request,
            future=future
        )
        heapq.heappush(self._queue, item)
        return future

    async def process_loop(self, process_fn):
        """Continuously process queued requests."""
        while self._processing:
            if not self._queue:
                await asyncio.sleep(0.01)
                continue

            await self._semaphore.acquire()
            item = heapq.heappop(self._queue)

            asyncio.create_task(
                self._process_item(item, process_fn))

    async def _process_item(self, item, process_fn):
        try:
            result = await process_fn(item.request)
            if not item.future.done():
                item.future.set_result(result)
        except Exception as e:
            if not item.future.done():
                item.future.set_exception(e)
        finally:
            self._semaphore.release()
```

## Deadlock Prevention

### Unngå resource starvation

```python
class DeadlockPreventionWrapper:
    """Prevent deadlocks in concurrent AI request processing."""

    def __init__(
        self,
        client: AsyncAzureOpenAI,
        max_concurrent: int = 20,
        request_timeout: float = 120.0,
        starvation_timeout: float = 300.0
    ):
        self.client = client
        self.semaphore = asyncio.Semaphore(max_concurrent)
        self.request_timeout = request_timeout
        self.starvation_timeout = starvation_timeout
        self._active_requests: dict[str, float] = {}

    async def execute(self, request_id: str, **kwargs):
        """Execute with timeout and starvation protection."""
        # Timeout på semaphore acquire — forhindrer deadlock
        try:
            await asyncio.wait_for(
                self.semaphore.acquire(),
                timeout=self.starvation_timeout
            )
        except asyncio.TimeoutError:
            raise TimeoutError(
                f"Request {request_id} starved waiting for "
                f"concurrency slot for {self.starvation_timeout}s. "
                f"Consider increasing max_concurrent or reducing "
                f"request volume.")

        self._active_requests[request_id] = time.time()

        try:
            # Timeout på selve forespørselen
            result = await asyncio.wait_for(
                self.client.chat.completions.create(**kwargs),
                timeout=self.request_timeout
            )
            return result

        except asyncio.TimeoutError:
            raise TimeoutError(
                f"Request {request_id} timed out after "
                f"{self.request_timeout}s")

        finally:
            self._active_requests.pop(request_id, None)
            self.semaphore.release()

    @property
    def active_count(self) -> int:
        return len(self._active_requests)

    def get_stuck_requests(self, threshold_seconds: float = 60) -> list:
        """Identify requests that may be stuck."""
        now = time.time()
        return [
            {"id": rid, "age_seconds": round(now - start, 1)}
            for rid, start in self._active_requests.items()
            if now - start > threshold_seconds
        ]
```

## Resource Contention Resolution

### Token bucket for fair scheduling

```python
import time

class TokenBucket:
    """Token bucket rate limiter for fair resource sharing."""

    def __init__(
        self,
        tokens_per_second: float,
        max_burst: int = 10
    ):
        self.rate = tokens_per_second
        self.max_burst = max_burst
        self._tokens = max_burst
        self._last_refill = time.time()
        self._lock = asyncio.Lock()

    async def acquire(self, tokens: int = 1) -> float:
        """Acquire tokens, waiting if necessary. Returns wait time."""
        async with self._lock:
            self._refill()

            if self._tokens >= tokens:
                self._tokens -= tokens
                return 0

            # Beregn ventetid
            deficit = tokens - self._tokens
            wait_time = deficit / self.rate
            await asyncio.sleep(wait_time)
            self._refill()
            self._tokens -= tokens
            return wait_time

    def _refill(self):
        now = time.time()
        elapsed = now - self._last_refill
        self._tokens = min(
            self.max_burst,
            self._tokens + elapsed * self.rate
        )
        self._last_refill = now


class FairScheduler:
    """Fair scheduling across multiple tenants/users."""

    def __init__(self, total_rps: float, num_tenants: int):
        self.total_rps = total_rps
        per_tenant_rps = total_rps / num_tenants
        self.buckets: dict[str, TokenBucket] = {}
        self._default_rps = per_tenant_rps

    def get_bucket(self, tenant_id: str) -> TokenBucket:
        if tenant_id not in self.buckets:
            self.buckets[tenant_id] = TokenBucket(
                tokens_per_second=self._default_rps,
                max_burst=int(self._default_rps * 2)
            )
        return self.buckets[tenant_id]
```

## Norsk offentlig sektor

- **Fair use**: I multi-tenant løsninger der flere enheter deler samme Azure OpenAI-deployment, bruk per-tenant rate limiting for å sikre rettferdig fordeling.
- **Brukeropplevelse**: Sett starvation timeout til maks ventetid brukere aksepterer (typisk 30-60 sekunder). Returner informativ feilmelding ved timeout.
- **Overvåking**: Logg concurrent request-nivå, kø-dybde og starvation-hendelser i Application Insights for kapasitetsplanlegging.
- **Skalering**: Planlegg for peak-perioder (morgen 08-10, etter lunsj 12-13) med høyere concurrent limits eller ekstra kvote.

## Beslutningsrammeverk

| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Ukjent workload | Start med 10 concurrent, juster | Unngå initial throttling |
| Forutsigbar, jevn trafikk | Statisk semaphore på optimal nivå | Enklest å implementere |
| Variable peaks | Adaptive concurrency controller | Automatisk tilpasning |
| Multi-tenant | Priority queue + per-tenant bucket | Fair resource sharing |
| Kritisk latens | Lav concurrency + PTU | Forutsigbar responstid |

## Referanser

- [Manage Azure OpenAI quota](https://learn.microsoft.com/azure/foundry-classic/openai/how-to/quota) — RPM/TPM grenser
- [Performance and latency](https://learn.microsoft.com/azure/foundry/openai/how-to/latency) — Concurrent requests og throughput
- [Provisioned throughput](https://learn.microsoft.com/azure/foundry/openai/how-to/provisioned-get-started) — PTU utilization

## For Cosmo

- **Bruk denne referansen** når kunden opplever timeout, starvation eller ujevn ytelse i AI-applikasjoner med mange samtidige brukere.
- Start konservativt (10-20 concurrent) og øk gradvis mens du monitorerer throttle rate — aldri gå rett til 100 concurrent.
- Adaptive concurrency control er anbefalt for produksjon — statiske verdier fungerer dårlig når trafikkmønstre endres.
- Prioritetskøer er viktige for multi-tenant: sørg for at kritiske oppgaver (saksbehandler-beslutninger) ikke blokkeres av bakgrunnsjobber.
- Deadlock prevention med timeouts er obligatorisk — uten det kan en hengende forespørsel blokkere alle slots permanent.