ms-ai-architect/skills/ms-ai-security/references/performance-scalability/throughput-optimization-strategies.md

# Throughput Optimization Strategies

**Last updated:** 2026-06-24
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference
**Source:** https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput-billing

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Parallel Request Execution](#parallel-request-execution)
- [Request Buffering Strategies](#request-buffering-strategies)
- [Queue Depth Tuning](#queue-depth-tuning)
- [System Bottleneck Identification](#system-bottleneck-identification)
- [Implementeringsmønstre](#implementeringsmønstre)
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
- [Beslutningsrammeverk](#beslutningsrammeverk)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

Throughput-optimalisering for Azure OpenAI og Azure AI Services handler om å maksimere antall fullførte forespørsler per sekund innenfor de tildelte kvotene. Azure OpenAI måler throughput i tokens per minutt (TPM) og forespørsler per minutt (RPM), og den reelle throughputen avhenger av en kompleks kombinasjon av input-størrelse, output-størrelse, modelltype og samtidige forespørsler.

For Standard deployments bestemmer den tildelte kvoten (TPM) en øvre grense for gjennomstrømming, men faktisk throughput kan være lavere på grunn av per-forespørsel latens. For Provisioned Throughput Units (PTU) er kapasiteten dedikert, og throughputen avhenger av workload shape — forholdet mellom input- og output-tokens. Microsofts offisielle benchmarking-verktøy (azure-openai-benchmark) er anbefalt for å måle reell throughput for spesifikke workloads.

I norsk offentlig sektor, der AI-løsninger ofte betjener tusenvis av saksbehandlere eller borgere samtidig, er throughput-optimalisering direkte knyttet til brukeropplevelse og kostnadseffektivitet. En 2x forbedring i throughput kan bety halverte Azure-kostnader for samme arbeidsmengde.

## Kjernekomponenter

| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| Token quota (TPM/RPM) | Rate limiting for Standard deployments | Azure OpenAI Quota |
| Provisioned Throughput Units | Dedikert kapasitet med garantert throughput | Azure OpenAI PTU |
| Batch API | 50% rabatt for asynkrone batch-jobber | Azure OpenAI Global Batch |
| Azure Load Testing | Lasttesting og throughput-måling | Azure Load Testing |
| Azure Monitor | Throughput-metrikker og overvåking | Azure Monitor |
| azure-openai-benchmark | Offisielt benchmarking-verktøy | GitHub CLI tool |

## Parallel Request Execution

### Asynkron parallellisering i Python

```python
import asyncio
import time
from openai import AsyncAzureOpenAI
from dataclasses import dataclass

@dataclass
class ThroughputResult:
    total_requests: int
    successful: int
    failed: int
    total_tokens: int
    duration_seconds: float
    requests_per_second: float
    tokens_per_second: float

async def parallel_completions(
    client: AsyncAzureOpenAI,
    messages_batch: list[list[dict]],
    model: str = "gpt-4o",
    max_concurrent: int = 20,
    max_tokens: int = 500
) -> ThroughputResult:
    """Execute chat completions in parallel with controlled concurrency."""
    semaphore = asyncio.Semaphore(max_concurrent)
    results = {"success": 0, "failed": 0, "tokens": 0}

    async def process_one(messages: list[dict]):
        async with semaphore:
            try:
                response = await client.chat.completions.create(
                    model=model,
                    messages=messages,
                    max_tokens=max_tokens
                )
                results["success"] += 1
                results["tokens"] += response.usage.total_tokens
            except Exception as e:
                results["failed"] += 1
                if hasattr(e, 'status_code') and e.status_code == 429:
                    # Retry-After: vent og prøv igjen
                    retry_after = getattr(e, 'retry_after', 5)
                    await asyncio.sleep(retry_after)
                    await process_one(messages)  # Retry

    start = time.time()
    await asyncio.gather(*[process_one(m) for m in messages_batch])
    duration = time.time() - start

    return ThroughputResult(
        total_requests=len(messages_batch),
        successful=results["success"],
        failed=results["failed"],
        total_tokens=results["tokens"],
        duration_seconds=round(duration, 2),
        requests_per_second=round(results["success"] / duration, 2),
        tokens_per_second=round(results["tokens"] / duration, 2)
    )

# Eksempel: Prosesser 1000 forespørsler med 20 samtidige
async def main():
    client = AsyncAzureOpenAI(
        azure_endpoint="https://my-aoai.openai.azure.com",
        api_key="...",
        api_version="2024-10-21"
    )

    batch = [
        [{"role": "user", "content": f"Oppsummer dokument {i}"}]
        for i in range(1000)
    ]

    result = await parallel_completions(client, batch, max_concurrent=20)
    print(f"Throughput: {result.requests_per_second} RPS, "
          f"{result.tokens_per_second} tokens/s")
```

### .NET Parallel Processing med SemaphoreSlim

```csharp
using Azure.AI.OpenAI;
using System.Collections.Concurrent;

public class ThroughputOptimizer
{
    private readonly AzureOpenAIClient _client;
    private readonly SemaphoreSlim _semaphore;
    private readonly ConcurrentBag<RequestMetric> _metrics = new();

    public ThroughputOptimizer(AzureOpenAIClient client, int maxConcurrency = 20)
    {
        _client = client;
        _semaphore = new SemaphoreSlim(maxConcurrency, maxConcurrency);
    }

    public async Task<ThroughputReport> ProcessBatchAsync(
        IReadOnlyList<ChatMessage[]> requests,
        string deploymentName,
        CancellationToken cancellationToken = default)
    {
        var sw = System.Diagnostics.Stopwatch.StartNew();

        var tasks = requests.Select(messages =>
            ProcessSingleAsync(messages, deploymentName, cancellationToken));

        await Task.WhenAll(tasks);
        sw.Stop();

        var successful = _metrics.Where(m => m.Success).ToList();
        return new ThroughputReport
        {
            TotalRequests = requests.Count,
            Successful = successful.Count,
            Failed = _metrics.Count - successful.Count,
            TotalTokens = successful.Sum(m => m.TotalTokens),
            DurationMs = sw.ElapsedMilliseconds,
            RequestsPerSecond = Math.Round(
                successful.Count / (sw.ElapsedMilliseconds / 1000.0), 2),
            TokensPerSecond = Math.Round(
                successful.Sum(m => m.TotalTokens) /
                (sw.ElapsedMilliseconds / 1000.0), 2)
        };
    }

    private async Task ProcessSingleAsync(
        ChatMessage[] messages,
        string deploymentName,
        CancellationToken ct)
    {
        await _semaphore.WaitAsync(ct);
        try
        {
            var chatClient = _client.GetChatClient(deploymentName);
            var response = await chatClient.CompleteChatAsync(messages);

            _metrics.Add(new RequestMetric
            {
                Success = true,
                TotalTokens = response.Value.Usage.TotalTokenCount
            });
        }
        catch (Exception)
        {
            _metrics.Add(new RequestMetric { Success = false });
        }
        finally
        {
            _semaphore.Release();
        }
    }
}
```

## Request Buffering Strategies

### Mikro-batching for høy throughput

```python
import asyncio
from collections import deque
from typing import Callable, Any

class RequestBuffer:
    """Buffer requests and flush in micro-batches for throughput optimization."""

    def __init__(
        self,
        process_fn: Callable,
        max_batch_size: int = 10,
        flush_interval_ms: int = 100,
        max_queue_size: int = 1000
    ):
        self.process_fn = process_fn
        self.max_batch_size = max_batch_size
        self.flush_interval = flush_interval_ms / 1000
        self.queue: deque = deque(maxlen=max_queue_size)
        self._running = False

    async def enqueue(self, request: dict) -> asyncio.Future:
        """Add request to buffer, returns future with result."""
        future = asyncio.get_event_loop().create_future()
        self.queue.append({"request": request, "future": future})

        if len(self.queue) >= self.max_batch_size:
            await self._flush()

        return await future

    async def _flush(self):
        """Process all buffered requests."""
        batch = []
        futures = []

        while self.queue and len(batch) < self.max_batch_size:
            item = self.queue.popleft()
            batch.append(item["request"])
            futures.append(item["future"])

        if batch:
            try:
                results = await self.process_fn(batch)
                for future, result in zip(futures, results):
                    future.set_result(result)
            except Exception as e:
                for future in futures:
                    if not future.done():
                        future.set_exception(e)

    async def run(self):
        """Run flush loop."""
        self._running = True
        while self._running:
            if self.queue:
                await self._flush()
            await asyncio.sleep(self.flush_interval)
```

## Queue Depth Tuning

### Optimal kø-dybde for Azure OpenAI

```python
import math

def calculate_optimal_queue_depth(
    tpm_quota: int,
    avg_input_tokens: int,
    avg_output_tokens: int,
    avg_latency_ms: float,
    target_utilization: float = 0.85
) -> dict:
    """Calculate optimal queue depth based on quota and latency."""

    # Beregn maks concurrent requests basert på quota
    total_tokens_per_request = avg_input_tokens + avg_output_tokens
    max_rpm = tpm_quota / total_tokens_per_request

    # Maks concurrent basert på latens
    requests_per_second = max_rpm / 60
    avg_latency_s = avg_latency_ms / 1000

    # Little's Law: L = λ * W
    # L = concurrent requests, λ = arrival rate, W = service time
    optimal_concurrent = requests_per_second * avg_latency_s

    # Queue depth = concurrent * buffer factor
    queue_depth = math.ceil(optimal_concurrent * (1 / target_utilization))

    return {
        "max_rpm": round(max_rpm),
        "max_rps": round(requests_per_second, 2),
        "optimal_concurrent": math.ceil(optimal_concurrent),
        "recommended_queue_depth": queue_depth,
        "theoretical_max_tps": round(
            tpm_quota / 60 / total_tokens_per_request *
            total_tokens_per_request, 0
        )
    }

# Eksempel: 240K TPM quota, typisk RAG-workload
result = calculate_optimal_queue_depth(
    tpm_quota=240_000,
    avg_input_tokens=2000,
    avg_output_tokens=500,
    avg_latency_ms=1200
)
print(result)
# {'max_rpm': 96, 'max_rps': 1.6, 'optimal_concurrent': 2,
#  'recommended_queue_depth': 3, ...}
```

## System Bottleneck Identification

### Identifisering av flaskehalser med Azure Monitor

```python
# KQL-spørringer for throughput-analyse

# 1. Token throughput per minutt
PROCESSED_TOKENS_QUERY = """
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where Category == "RequestResponse"
| extend promptTokens = toint(properties_s.promptTokens)
| extend completionTokens = toint(properties_s.completionTokens)
| summarize
    TotalPromptTPM = sum(promptTokens),
    TotalCompletionTPM = sum(completionTokens),
    TotalTPM = sum(promptTokens) + sum(completionTokens),
    RequestCount = count()
    by bin(TimeGenerated, 1m), deploymentName_s
| order by TimeGenerated desc
"""

# 2. Identifiser throttling-mønstre
THROTTLING_ANALYSIS = """
AzureMetrics
| where MetricName == "AzureOpenAIRequests"
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
| summarize
    Total = count(),
    Throttled = countif(StatusCode == "429"),
    ServerErrors = countif(StatusCode startswith "5"),
    ThrottleRate = round(
        countif(StatusCode == "429") * 100.0 / count(), 2)
    by bin(TimeGenerated, 5m)
| where ThrottleRate > 0
| order by TimeGenerated desc
"""

# 3. Latens-distribusjon for å finne bottlenecks
LATENCY_PERCENTILES = """
AzureDiagnostics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| extend DurationMs = todouble(DurationMs)
| summarize
    P50 = percentile(DurationMs, 50),
    P90 = percentile(DurationMs, 90),
    P95 = percentile(DurationMs, 95),
    P99 = percentile(DurationMs, 99),
    Avg = avg(DurationMs)
    by bin(TimeGenerated, 5m), deploymentName_s
| order by TimeGenerated desc
"""
```

### Bottleneck Decision Tree

```
Lav throughput?
├── Høy throttle rate (>5% 429s)?
│   ├── Ja → Øk TPM-kvote eller legg til regioner
│   └── Nei → Sjekk latens
├── Høy latens (P95 > 5s)?
│   ├── Input tokens > 4K? → Reduser prompt-størrelse
│   ├── Output tokens > 2K? → Reduser max_tokens
│   └── Lav token count? → Sjekk nettverkslatens
├── Lav concurrent requests?
│   ├── Klient-side bottleneck → Øk parallellisering
│   └── Connection pool for liten → Øk pool size
└── Utilization < 50%?
    └── Under-provisjonert? → Sjekk quota allocation
```

## Implementeringsmønstre

### Batch API for ikke-tidskritisk prosessering

```python
from openai import AzureOpenAI
import json

def create_batch_file(requests: list[dict], filename: str = "batch.jsonl"):
    """Create JSONL file for Azure OpenAI Batch API."""
    with open(filename, "w") as f:
        for i, req in enumerate(requests):
            batch_request = {
                "custom_id": f"request-{i}",
                "method": "POST",
                "url": "/chat/completions",
                "body": {
                    "model": "gpt-4o",  # Must match deployment name
                    "messages": req["messages"],
                    "max_tokens": req.get("max_tokens", 1000)
                }
            }
            f.write(json.dumps(batch_request) + "\n")

def submit_batch(client: AzureOpenAI, filename: str):
    """Submit batch job — 50% cost reduction, 24hr turnaround."""
    # Upload file
    batch_file = client.files.create(
        file=open(filename, "rb"),
        purpose="batch"
    )

    # Create batch job
    batch_job = client.batches.create(
        input_file_id=batch_file.id,
        endpoint="/chat/completions",
        completion_window="24h"
    )
    return batch_job
```

## Norsk offentlig sektor

- **Kostnadseffektivitet**: Bruk Batch API for alle ikke-sanntids workloads (dokumentanalyse, klassifisering, oppsummering) for å oppnå 50% kostnadsreduksjon. Dette er spesielt relevant for store etater med høyt dokumentvolum.
- **Kapasitetsplanlegging**: Start med å estimere TPM-behov basert på forventet brukermønster (antall saksbehandlere * forespørsler per time * tokens per forespørsel). Bestill PTU for forutsigbare workloads.
- **SLA-krav**: Provisioned throughput gir forutsigbar ytelse med latens-SLA (99% > N tokens/sekund per PTU). Standard deployments har ingen latens-SLA.
- **Data residency**: Global Batch behandler data i Azure OpenAI-lokasjoner globalt — bruk Data Zone Batch for å holde data innenfor EU/EØS.

## Beslutningsrammeverk

| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Sanntids chat (<2s respons) | Standard/PTU + streaming | Lavest brukervendt latens |
| Dokumentprosessering (1000+ docs) | Batch API | 50% kostnadsreduksjon, 24h turnaround |
| Forutsigbar høy trafikk | Provisioned Throughput (PTU) | Garantert kapasitet og latens |
| Variable workloads | Standard + auto-scale quota | Betal per bruk, fleksibel skalering |
| Multi-model pipeline | Parallell execution + queue | Maksimer samlet throughput |

## Referanser

- [Performance and latency](https://learn.microsoft.com/azure/foundry/openai/how-to/latency) — Azure OpenAI latency og throughput
- [Azure OpenAI Batch API](https://learn.microsoft.com/azure/foundry/openai/how-to/batch) — Batch processing guide
- [Provisioned throughput onboarding](https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput-billing) — PTU sizing og kostnader
- [Azure OpenAI Benchmark Tool](https://github.com/Azure/azure-openai-benchmark) — Offisielt benchmarking-verktøy

## For Cosmo

- **Bruk denne referansen** når kunden trenger å maksimere throughput for AI-workloads, eller når de opplever at de ikke utnytter sin tildelte kvote effektivt.
- Batch API gir 50% kostnadsreduksjon og bør anbefales for alle ikke-sanntids workloads — mange kunder er ikke klar over denne muligheten.
- Bruk Little's Law (L = lambda * W) for å beregne optimal concurrent requests: quota bestemmer lambda, latens bestemmer W.
- Alltid benchmark med reelle workloads — den offisielle azure-openai-benchmark-verktøyet gir pålitelige tall for PTU-sizing.
- For norsk offentlig sektor: anbefal Data Zone deployments for Batch API for å holde data innenfor EU/EØS.