# Streaming Support in APIM for AI Responses

**Last updated:** 2026-06-24
**Status:** GA
**Category:** API Management & AI Gateway
**Type:** reference
**Source:** https://learn.microsoft.com/azure/api-management/genai-gateway-capabilities

---

## Innhold

- [Introduksjon](#introduksjon)
- [SSE Forwarding](#sse-forwarding)
- [Buffering Policies](#buffering-policies)
- [Chunked Responses](#chunked-responses)
- [Timeout Management for Streams](#timeout-management-for-streams)
- [Client Compatibility](#client-compatibility)
- [Logging av Streaming-requests](#logging-av-streaming-requests)
- [Token-telling for Streaming](#token-telling-for-streaming)
- [Komplett Streaming-policy](#komplett-streaming-policy)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

Streaming av AI-responser er en nøkkelfunksjon for å levere god brukeropplevelse i chat-applikasjoner. Azure OpenAI støtter Server-Sent Events (SSE) for å streame chat completions token-for-token til klienten, noe som gir umiddelbar feedback i stedet for å vente på en komplett respons. Når Azure API Management (APIM) sitter mellom klient og Azure OpenAI, krever denne streaming-arkitekturen spesifikk konfigurasjon for å fungere korrekt.

For norsk offentlig sektor som bygger AI-chatboter og assistenter er streaming kritisk for brukeropplevelsen. Uten streaming kan brukere vente 10-30 sekunder på svar fra store modeller som GPT-4o — med streaming begynner svar å vises innen 1-2 sekunder. Denne referansen dekker alle aspekter ved konfigurering av APIM for streaming av AI-responser, inkludert SSE forwarding, buffering-policyer, timeout-håndtering og klientkompatibilitet.

APIM støtter SSE gjennom klassiske og v2-tiers (ikke Consumption-tier). Korrekt konfigurasjon krever at flere aspekter justeres: response buffering må deaktiveres, timeouts må økes, og logging-konfigurasjonen må tilpasses for å unngå at streaming-responser bufres opp.

---

## SSE Forwarding

### Slik Fungerer SSE med Azure OpenAI

Når `"stream": true` settes i chat completion-forespørselen, returnerer Azure OpenAI en strøm av Server-Sent Events:

```
HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Connection: keep-alive

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hei"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" på"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" deg"},"index":0}]}

data: [DONE]
```

### APIM som SSE Proxy

APIM fungerer som en transparent proxy for SSE-trafikk mellom klient og Azure OpenAI:

```
Klient → APIM Gateway → Azure OpenAI
         (SSE proxy)    (SSE source)

1. Klient sender POST med "stream": true
2. APIM forwarder til Azure OpenAI
3. Azure OpenAI begynner å streame SSE-data
4. APIM relayer hvert SSE-event umiddelbart til klient
5. Azure OpenAI sender "data: [DONE]"
6. Forbindelsen lukkes
```

### Krav for SSE Forwarding

| Krav | Innstilling | Merknader |
|------|------------|-----------|
| APIM Tier | Classic eller v2 | Consumption-tier støttes IKKE |
| Response buffering | Deaktivert | `buffer-response="false"` |
| Keepalive | Aktivert | Unngå 4 min idle timeout |
| Response body logging | Deaktivert | Unngår buffering |
| Caching | Deaktivert | For SSE-endepunkter |

---

## Buffering Policies

### Deaktivere Response Buffering

Den viktigste konfigurasjonen for streaming er å deaktivere response buffering i `forward-request`:

```xml
<policies>
    <inbound>
        <base />
        <!-- Standard APIM inbound-policyer (autentisering, rate limiting, etc.) -->
        <set-backend-service backend-id="aoai-backend" />
    </inbound>
    <backend>
        <!-- KRITISK: buffer-response="false" for streaming -->
        <forward-request timeout="120"
                         fail-on-error-status-code="true"
                         buffer-response="false" />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>
```

### Policyer som MÅ Unngås med Streaming

Følgende policyer buffrer responsen og er IKKE kompatible med SSE:

| Policy | Problem | Alternativ |
|--------|---------|-----------|
| `validate-content` | Buffrer full respons for validering | Valider kun inbound request |
| `xml-to-json` / `json-to-xml` | Trenger full respons for konvertering | Ikke aktuelt for SSE |
| `xslt-transform` | Buffrer for transformasjon | Ikke aktuelt for SSE |
| `cache-store` | Lagrer full respons | Bruk `llm-semantic-cache-store` |
| `log-to-eventhub` (med body) | Buffrer respons for logging | Logg kun headers |

### Betinget Buffering

Aktiver buffering kun for ikke-streaming requests:

```xml
<backend>
    <choose>
        <!-- Sjekk om request er streaming -->
        <when condition="@{
            var body = context.Request.Body.As<JObject>(preserveContent: true);
            return body != null && body["stream"]?.Value<bool>() == true;
        }">
            <!-- Streaming: IKKE buffer -->
            <forward-request timeout="240"
                             fail-on-error-status-code="true"
                             buffer-response="false" />
        </when>
        <otherwise>
            <!-- Ikke-streaming: buffer er OK -->
            <forward-request timeout="120"
                             fail-on-error-status-code="true"
                             buffer-response="true" />
        </otherwise>
    </choose>
</backend>
```

---

## Chunked Responses

### Transfer-Encoding: chunked

SSE-responses fra Azure OpenAI bruker chunked transfer encoding. APIM håndterer dette automatisk når `buffer-response="false"`:

```
HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive
```

### Response Headers for Korrekt Streaming

Backend-tjenesten (Azure OpenAI) sender disse headerne:

| Header | Verdi | Formål |
|--------|-------|--------|
| `Content-Type` | `text/event-stream` | Signaliserer SSE til klient |
| `Transfer-Encoding` | `chunked` | Tillater streaming uten Content-Length |
| `Connection` | `keep-alive` | Holder TCP-forbindelsen åpen |
| `Cache-Control` | `no-cache` | Forhindrer mellomlagring |

### APIM Policy for Response Headers

Sørg for at APIM ikke overstyrer kritiske streaming-headers:

```xml
<outbound>
    <base />
    <!-- Sørg for at streaming-headers videresendes korrekt -->
    <choose>
        <when condition="@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("text/event-stream"))">
            <!-- Ikke legg til Cache-Control som kan interferere -->
            <set-header name="X-Stream-Response" exists-action="override">
                <value>true</value>
            </set-header>
        </when>
    </choose>
</outbound>
```

---

## Timeout Management for Streams

### Idle Connection Timeout

Azure Load Balancer (som brukes i APIM-infrastrukturen) har en standard idle timeout på 4 minutter. For streaming-scenarier der det kan gå tid mellom tokens:

```
Strategi 1: Backend keepalive
  → Azure OpenAI sender SSE-events fortløpende
  → Normalt ikke et problem med aktiv streaming

Strategi 2: Klient keepalive
  → Klient sender "ping" minst hvert 4. minutt
  → Aktuelt for langvarige idle-forbindelser

Strategi 3: Økt timeout via policy
  → forward-request timeout="240"
  → Dekker de fleste scenarier
```

### Timeout-verdier for Streaming

| Parameter | Standard | Anbefalt for streaming | Merknader |
|-----------|---------|----------------------|-----------|
| `forward-request timeout` | 300 sek | 120-240 sek | Avhenger av maks respons-lengde |
| Azure LB idle timeout | 240 sek | Ikke konfigurerbar i APIM | Bruk keepalive |
| DNS TTL | Varierer | N/A | Påvirker failover |

### Timeout Policy for Streaming Endpoints

```xml
<backend>
    <forward-request timeout="240"
                     fail-on-error-status-code="true"
                     buffer-response="false" />
</backend>

<on-error>
    <base />
    <choose>
        <when condition="@(context.LastError.Source == "forward-request" &&
                          context.LastError.Reason == "Timeout")">
            <return-response>
                <set-status code="504" reason="Gateway Timeout" />
                <set-body>{
    "error": {
        "code": "StreamingTimeout",
        "message": "The AI model did not complete its response within the timeout period."
    }
}</set-body>
            </return-response>
        </when>
    </choose>
</on-error>
```

---

## Client Compatibility

### JavaScript/TypeScript EventSource

```typescript
// Standard EventSource for SSE
const response = await fetch('/api/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${token}`,
    'Ocp-Apim-Subscription-Key': subscriptionKey
  },
  body: JSON.stringify({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Hei, Cosmo!' }],
    stream: true
  })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  const lines = chunk.split('\n').filter(line => line.startsWith('data: '));

  for (const line of lines) {
    const data = line.slice(6); // Fjern "data: " prefiks
    if (data === '[DONE]') break;

    const parsed = JSON.parse(data);
    const content = parsed.choices[0]?.delta?.content;
    if (content) {
      process.stdout.write(content);
    }
  }
}
```

### Python med httpx

```python
import httpx
import json

async def stream_completion(prompt: str):
    async with httpx.AsyncClient() as client:
        async with client.stream(
            "POST",
            f"{APIM_ENDPOINT}/openai/deployments/gpt-4o/chat/completions",
            params={"api-version": "2024-10-21"},
            headers={
                "Content-Type": "application/json",
                "Ocp-Apim-Subscription-Key": SUBSCRIPTION_KEY,
            },
            json={
                "messages": [{"role": "user", "content": prompt}],
                "stream": True
            },
            timeout=120.0
        ) as response:
            async for line in response.aiter_lines():
                if line.startswith("data: "):
                    data = line[6:]
                    if data == "[DONE]":
                        break
                    chunk = json.loads(data)
                    content = chunk["choices"][0]["delta"].get("content", "")
                    print(content, end="", flush=True)
```

### C# med Azure.AI.OpenAI

```csharp
var client = new AzureOpenAIClient(
    new Uri(apimEndpoint),
    new AzureKeyCredential(subscriptionKey));

var chatClient = client.GetChatClient("gpt-4o");

// Streaming via APIM
await foreach (var update in chatClient.CompleteChatStreamingAsync(
    new ChatMessage[] { new UserChatMessage("Hei, Cosmo!") }))
{
    foreach (var part in update.ContentUpdate)
    {
        Console.Write(part.Text);
    }
}
```

### Klientkrav for APIM-proxy

| Krav | Beskrivelse |
|------|-------------|
| Subscription key | `Ocp-Apim-Subscription-Key` header eller query parameter |
| Timeout | Minst 120 sekunder for streaming |
| Chunked decoding | Håndtere `Transfer-Encoding: chunked` |
| SSE parsing | Parse `data: ` prefiks og `[DONE]` sentinel |
| Connection handling | Håndtere mid-stream connection drops gracefully |

---

## Logging av Streaming-requests

### Utfordringer med Streaming-logging

Når response body logges, bufres hele responsen — noe som bryter streaming. Korrekt logging for SSE-endepunkter:

```xml
<inbound>
    <base />
    <!-- Logg inbound request (prompt) — dette er OK -->
    <log-to-eventhub logger-id="ai-eventhub-logger">
    @{
        var body = context.Request.Body.As<string>(preserveContent: true);
        return new JObject(
            new JProperty("timestamp", DateTime.UtcNow),
            new JProperty("method", context.Request.Method),
            new JProperty("url", context.Request.Url.ToString()),
            new JProperty("prompt", body)
        ).ToString();
    }
    </log-to-eventhub>
</inbound>

<outbound>
    <base />
    <!-- IKKE logg response body for streaming — det bufrer responsen -->
    <!-- Logg kun metadata -->
    <choose>
        <when condition="@(!context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("text/event-stream"))">
            <!-- Kun for ikke-streaming responses -->
            <log-to-eventhub logger-id="ai-eventhub-logger">
            @{
                return new JObject(
                    new JProperty("statusCode", context.Response.StatusCode),
                    new JProperty("responseBody", context.Response.Body.As<string>(preserveContent: true))
                ).ToString();
            }
            </log-to-eventhub>
        </when>
    </choose>
</outbound>
```

### APIM Diagnostic Settings for Streaming

Deaktiver response body logging for APIs som bruker streaming:

```
1. Naviger til API → Settings → Diagnostic Logs
2. Azure Monitor-fanen:
   - Frontend Response: Body bytes = 0
   - Backend Response: Body bytes = 0
3. Application Insights-fanen:
   - Body bytes to log: 0 (for streaming APIs)
```

### LLM API Logging (Azure Monitor)

For APIM sin innebygde LLM-logging:

```
1. APIM → Monitoring → Diagnostic settings
2. Velg "Logs related to generative AI gateway"
3. Send to Log Analytics workspace
4. NB: Log LLM messages fungerer kun for IKKE-streaming requests
```

---

## Token-telling for Streaming

### Utfordring

Ved streaming returnerer Azure OpenAI token-bruk i siste chunk (`usage` feltet). APIM sin `llm-emit-token-metric` policy krever tilgang til dette:

```json
// Siste chunk i streaming-respons
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk",
  "choices":[{"delta":{},"index":0,"finish_reason":"stop"}],
  "usage":{"prompt_tokens":15,"completion_tokens":42,"total_tokens":57}}
```

### Policy for Token-metriker (Ikke-streaming)

For ikke-streaming requests, bruk standard `llm-emit-token-metric` i outbound:

```xml
<outbound>
    <base />
    <llm-emit-token-metric namespace="ai-metrics">
        <dimension name="API" value="@(context.Api.Name)" />
        <dimension name="User" value="@(context.Subscription.Name)" />
    </llm-emit-token-metric>
</outbound>
```

**Merk:** `llm-emit-token-metric` fungerer for både streaming og ikke-streaming requests. APIM håndterer parsing av streaming-chunks for å ekstrahere token-bruk automatisk.

---

## Komplett Streaming-policy

### Full Policy for Streaming AI Gateway

```xml
<policies>
    <inbound>
        <base />

        <!-- Autentisering -->
        <validate-azure-ad-token tenant-id="{{TENANT_ID}}"
                                 header-name="Authorization" />

        <!-- Token rate limiting -->
        <llm-token-limit counter-key="@(context.Subscription.Id)"
                         tokens-per-minute="10000"
                         estimate-prompt-tokens="true" />

        <!-- Backend med managed identity -->
        <set-backend-service backend-id="aoai-pool" />
        <authentication-managed-identity
            resource="https://cognitiveservices.azure.com"
            output-token-variable-name="mi-token" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (string)context.Variables["mi-token"])</value>
        </set-header>
    </inbound>

    <backend>
        <!-- Streaming-kompatibel forwarding -->
        <forward-request timeout="240"
                         fail-on-error-status-code="true"
                         buffer-response="false" />
    </backend>

    <outbound>
        <base />

        <!-- Token-metriker (fungerer for streaming og ikke-streaming) -->
        <llm-emit-token-metric namespace="ai-metrics">
            <dimension name="Subscription" value="@(context.Subscription.Name)" />
            <dimension name="API" value="@(context.Api.Name)" />
            <dimension name="Region" value="@(context.Deployment.Region)" />
        </llm-emit-token-metric>
    </outbound>

    <on-error>
        <base />
    </on-error>
</policies>
```

---

## Referanser

- [Configure API for server-sent events](https://learn.microsoft.com/en-us/azure/api-management/how-to-server-sent-events) — Offisiell SSE-guide for APIM
- [AI gateway in Azure API Management](https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities) — AI gateway oversikt
- [Azure OpenAI REST API reference - Chat Completions](https://learn.microsoft.com/en-us/azure/foundry/openai/reference#chat-completions) — Stream-parameter dokumentasjon
- [forward-request policy](https://learn.microsoft.com/en-us/azure/api-management/forward-request-policy) — Policy-referanse for forwarding
- [Log token usage, prompts, and completions](https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-llm-logs) — LLM-logging i APIM

---

## For Cosmo

- **Bruk denne referansen** når kunder implementerer chat-applikasjoner eller AI-assistenter gjennom APIM og trenger streaming-støtte.
- Den absolutt viktigste innstillingen er `buffer-response="false"` i `forward-request`. Uten dette bufres hele SSE-responsen og leveres som én stor blob — som dreper brukeropplevelsen.
- Advar om at Consumption-tier IKKE støtter langvarige HTTP-forbindelser som SSE krever. Anbefal v2 eller Premium tier for streaming-scenarier.
- For logging av streaming-requests: Bruk `llm-emit-token-metric` for token-metriker (fungerer med streaming). Unngå response body logging som bryter streaming.
- Kombiner streaming med retry-policy forsiktig — retry fungerer kun for initial connection failure, ikke for mid-stream feil.