ms-ai-architect/skills/ms-ai-engineering/references/api-management/streaming-support-apim.md

Streaming Support in APIM for AI Responses

Last updated: 2026-06-24 Status: GA Category: API Management & AI Gateway Type: reference Source: https://learn.microsoft.com/azure/api-management/genai-gateway-capabilities

---

Innhold

• Introduksjon
• SSE Forwarding
• Buffering Policies
• Chunked Responses
• Timeout Management for Streams
• Client Compatibility
• Logging av Streaming-requests
• Token-telling for Streaming
• Komplett Streaming-policy
• Referanser
• For Cosmo

Introduksjon

Streaming av AI-responser er en nøkkelfunksjon for å levere god brukeropplevelse i chat-applikasjoner. Azure OpenAI støtter Server-Sent Events (SSE) for å streame chat completions token-for-token til klienten, noe som gir umiddelbar feedback i stedet for å vente på en komplett respons. Når Azure API Management (APIM) sitter mellom klient og Azure OpenAI, krever denne streaming-arkitekturen spesifikk konfigurasjon for å fungere korrekt.

For norsk offentlig sektor som bygger AI-chatboter og assistenter er streaming kritisk for brukeropplevelsen. Uten streaming kan brukere vente 10-30 sekunder på svar fra store modeller som GPT-4o — med streaming begynner svar å vises innen 1-2 sekunder. Denne referansen dekker alle aspekter ved konfigurering av APIM for streaming av AI-responser, inkludert SSE forwarding, buffering-policyer, timeout-håndtering og klientkompatibilitet.

APIM støtter SSE gjennom klassiske og v2-tiers (ikke Consumption-tier). Korrekt konfigurasjon krever at flere aspekter justeres: response buffering må deaktiveres, timeouts må økes, og logging-konfigurasjonen må tilpasses for å unngå at streaming-responser bufres opp.

---

SSE Forwarding

Slik Fungerer SSE med Azure OpenAI

Når "stream": true settes i chat completion-forespørselen, returnerer Azure OpenAI en strøm av Server-Sent Events:

HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Connection: keep-alive

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"role":"assistant"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":"Hei"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" på"},"index":0}]}

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","choices":[{"delta":{"content":" deg"},"index":0}]}

data: [DONE]

APIM som SSE Proxy

APIM fungerer som en transparent proxy for SSE-trafikk mellom klient og Azure OpenAI:

Klient → APIM Gateway → Azure OpenAI
         (SSE proxy)    (SSE source)

1. Klient sender POST med "stream": true
2. APIM forwarder til Azure OpenAI
3. Azure OpenAI begynner å streame SSE-data
4. APIM relayer hvert SSE-event umiddelbart til klient
5. Azure OpenAI sender "data: [DONE]"
6. Forbindelsen lukkes

Krav for SSE Forwarding

Krav	Innstilling	Merknader
APIM Tier	Classic eller v2	Consumption-tier støttes IKKE
Response buffering	Deaktivert	buffer-response="false"
Keepalive	Aktivert	Unngå 4 min idle timeout
Response body logging	Deaktivert	Unngår buffering
Caching	Deaktivert	For SSE-endepunkter

---

Buffering Policies

Deaktivere Response Buffering

Den viktigste konfigurasjonen for streaming er å deaktivere response buffering i forward-request:

<policies>
    <inbound>
        <base />
        <!-- Standard APIM inbound-policyer (autentisering, rate limiting, etc.) -->
        <set-backend-service backend-id="aoai-backend" />
    </inbound>
    <backend>
        <!-- KRITISK: buffer-response="false" for streaming -->
        <forward-request timeout="120"
                         fail-on-error-status-code="true"
                         buffer-response="false" />
    </backend>
    <outbound>
        <base />
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Policyer som MÅ Unngås med Streaming

Følgende policyer buffrer responsen og er IKKE kompatible med SSE:

Policy	Problem	Alternativ
validate-content	Buffrer full respons for validering	Valider kun inbound request
xml-to-json / json-to-xml	Trenger full respons for konvertering	Ikke aktuelt for SSE
xslt-transform	Buffrer for transformasjon	Ikke aktuelt for SSE
cache-store	Lagrer full respons	Bruk llm-semantic-cache-store
log-to-eventhub (med body)	Buffrer respons for logging	Logg kun headers

Betinget Buffering

Aktiver buffering kun for ikke-streaming requests:

<backend>
    <choose>
        <!-- Sjekk om request er streaming -->
        <when condition="@{
            var body = context.Request.Body.As<JObject>(preserveContent: true);
            return body != null && body["stream"]?.Value<bool>() == true;
        }">
            <!-- Streaming: IKKE buffer -->
            <forward-request timeout="240"
                             fail-on-error-status-code="true"
                             buffer-response="false" />
        </when>
        <otherwise>
            <!-- Ikke-streaming: buffer er OK -->
            <forward-request timeout="120"
                             fail-on-error-status-code="true"
                             buffer-response="true" />
        </otherwise>
    </choose>
</backend>

---

Chunked Responses

Transfer-Encoding: chunked

SSE-responses fra Azure OpenAI bruker chunked transfer encoding. APIM håndterer dette automatisk når buffer-response="false":

HTTP/1.1 200 OK
Content-Type: text/event-stream
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive

Response Headers for Korrekt Streaming

Backend-tjenesten (Azure OpenAI) sender disse headerne:

Header	Verdi	Formål
Content-Type	text/event-stream	Signaliserer SSE til klient
Transfer-Encoding	chunked	Tillater streaming uten Content-Length
Connection	keep-alive	Holder TCP-forbindelsen åpen
Cache-Control	no-cache	Forhindrer mellomlagring

APIM Policy for Response Headers

Sørg for at APIM ikke overstyrer kritiske streaming-headers:

<outbound>
    <base />
    <!-- Sørg for at streaming-headers videresendes korrekt -->
    <choose>
        <when condition="@(context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("text/event-stream"))">
            <!-- Ikke legg til Cache-Control som kan interferere -->
            <set-header name="X-Stream-Response" exists-action="override">
                <value>true</value>
            </set-header>
        </when>
    </choose>
</outbound>

---

Timeout Management for Streams

Idle Connection Timeout

Azure Load Balancer (som brukes i APIM-infrastrukturen) har en standard idle timeout på 4 minutter. For streaming-scenarier der det kan gå tid mellom tokens:

Strategi 1: Backend keepalive
  → Azure OpenAI sender SSE-events fortløpende
  → Normalt ikke et problem med aktiv streaming

Strategi 2: Klient keepalive
  → Klient sender "ping" minst hvert 4. minutt
  → Aktuelt for langvarige idle-forbindelser

Strategi 3: Økt timeout via policy
  → forward-request timeout="240"
  → Dekker de fleste scenarier

Timeout-verdier for Streaming

Parameter	Standard	Anbefalt for streaming	Merknader
forward-request timeout	300 sek	120-240 sek	Avhenger av maks respons-lengde
Azure LB idle timeout	240 sek	Ikke konfigurerbar i APIM	Bruk keepalive
DNS TTL	Varierer	N/A	Påvirker failover

Timeout Policy for Streaming Endpoints

<backend>
    <forward-request timeout="240"
                     fail-on-error-status-code="true"
                     buffer-response="false" />
</backend>

<on-error>
    <base />
    <choose>
        <when condition="@(context.LastError.Source == "forward-request" &&
                          context.LastError.Reason == "Timeout")">
            <return-response>
                <set-status code="504" reason="Gateway Timeout" />
                <set-body>{
    "error": {
        "code": "StreamingTimeout",
        "message": "The AI model did not complete its response within the timeout period."
    }
}</set-body>
            </return-response>
        </when>
    </choose>
</on-error>

---

Client Compatibility

JavaScript/TypeScript EventSource

// Standard EventSource for SSE
const response = await fetch('/api/chat/completions', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'Authorization': `Bearer ${token}`,
    'Ocp-Apim-Subscription-Key': subscriptionKey
  },
  body: JSON.stringify({
    model: 'gpt-4o',
    messages: [{ role: 'user', content: 'Hei, Cosmo!' }],
    stream: true
  })
});

const reader = response.body.getReader();
const decoder = new TextDecoder();

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  const chunk = decoder.decode(value);
  const lines = chunk.split('\n').filter(line => line.startsWith('data: '));

  for (const line of lines) {
    const data = line.slice(6); // Fjern "data: " prefiks
    if (data === '[DONE]') break;

    const parsed = JSON.parse(data);
    const content = parsed.choices[0]?.delta?.content;
    if (content) {
      process.stdout.write(content);
    }
  }
}

Python med httpx

import httpx
import json

async def stream_completion(prompt: str):
    async with httpx.AsyncClient() as client:
        async with client.stream(
            "POST",
            f"{APIM_ENDPOINT}/openai/deployments/gpt-4o/chat/completions",
            params={"api-version": "2024-10-21"},
            headers={
                "Content-Type": "application/json",
                "Ocp-Apim-Subscription-Key": SUBSCRIPTION_KEY,
            },
            json={
                "messages": [{"role": "user", "content": prompt}],
                "stream": True
            },
            timeout=120.0
        ) as response:
            async for line in response.aiter_lines():
                if line.startswith("data: "):
                    data = line[6:]
                    if data == "[DONE]":
                        break
                    chunk = json.loads(data)
                    content = chunk["choices"][0]["delta"].get("content", "")
                    print(content, end="", flush=True)

C# med Azure.AI.OpenAI

var client = new AzureOpenAIClient(
    new Uri(apimEndpoint),
    new AzureKeyCredential(subscriptionKey));

var chatClient = client.GetChatClient("gpt-4o");

// Streaming via APIM
await foreach (var update in chatClient.CompleteChatStreamingAsync(
    new ChatMessage[] { new UserChatMessage("Hei, Cosmo!") }))
{
    foreach (var part in update.ContentUpdate)
    {
        Console.Write(part.Text);
    }
}

Klientkrav for APIM-proxy

Krav	Beskrivelse
Subscription key	Ocp-Apim-Subscription-Key header eller query parameter
Timeout	Minst 120 sekunder for streaming
Chunked decoding	Håndtere Transfer-Encoding: chunked
SSE parsing	Parse data: prefiks og [DONE] sentinel
Connection handling	Håndtere mid-stream connection drops gracefully

---

Logging av Streaming-requests

Utfordringer med Streaming-logging

Når response body logges, bufres hele responsen — noe som bryter streaming. Korrekt logging for SSE-endepunkter:

<inbound>
    <base />
    <!-- Logg inbound request (prompt) — dette er OK -->
    <log-to-eventhub logger-id="ai-eventhub-logger">
    @{
        var body = context.Request.Body.As<string>(preserveContent: true);
        return new JObject(
            new JProperty("timestamp", DateTime.UtcNow),
            new JProperty("method", context.Request.Method),
            new JProperty("url", context.Request.Url.ToString()),
            new JProperty("prompt", body)
        ).ToString();
    }
    </log-to-eventhub>
</inbound>

<outbound>
    <base />
    <!-- IKKE logg response body for streaming — det bufrer responsen -->
    <!-- Logg kun metadata -->
    <choose>
        <when condition="@(!context.Response.Headers.GetValueOrDefault("Content-Type","").Contains("text/event-stream"))">
            <!-- Kun for ikke-streaming responses -->
            <log-to-eventhub logger-id="ai-eventhub-logger">
            @{
                return new JObject(
                    new JProperty("statusCode", context.Response.StatusCode),
                    new JProperty("responseBody", context.Response.Body.As<string>(preserveContent: true))
                ).ToString();
            }
            </log-to-eventhub>
        </when>
    </choose>
</outbound>

APIM Diagnostic Settings for Streaming

Deaktiver response body logging for APIs som bruker streaming:

1. Naviger til API → Settings → Diagnostic Logs
2. Azure Monitor-fanen:
   - Frontend Response: Body bytes = 0
   - Backend Response: Body bytes = 0
3. Application Insights-fanen:
   - Body bytes to log: 0 (for streaming APIs)

LLM API Logging (Azure Monitor)

For APIM sin innebygde LLM-logging:

1. APIM → Monitoring → Diagnostic settings
2. Velg "Logs related to generative AI gateway"
3. Send to Log Analytics workspace
4. NB: Log LLM messages fungerer kun for IKKE-streaming requests

---

Token-telling for Streaming

Utfordring

Ved streaming returnerer Azure OpenAI token-bruk i siste chunk (usage feltet). APIM sin llm-emit-token-metric policy krever tilgang til dette:

// Siste chunk i streaming-respons
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk",
  "choices":[{"delta":{},"index":0,"finish_reason":"stop"}],
  "usage":{"prompt_tokens":15,"completion_tokens":42,"total_tokens":57}}

Policy for Token-metriker (Ikke-streaming)

For ikke-streaming requests, bruk standard llm-emit-token-metric i outbound:

<outbound>
    <base />
    <llm-emit-token-metric namespace="ai-metrics">
        <dimension name="API" value="@(context.Api.Name)" />
        <dimension name="User" value="@(context.Subscription.Name)" />
    </llm-emit-token-metric>
</outbound>

Merk: llm-emit-token-metric fungerer for både streaming og ikke-streaming requests. APIM håndterer parsing av streaming-chunks for å ekstrahere token-bruk automatisk.

---

Komplett Streaming-policy

Full Policy for Streaming AI Gateway

<policies>
    <inbound>
        <base />

        <!-- Autentisering -->
        <validate-azure-ad-token tenant-id="{{TENANT_ID}}"
                                 header-name="Authorization" />

        <!-- Token rate limiting -->
        <llm-token-limit counter-key="@(context.Subscription.Id)"
                         tokens-per-minute="10000"
                         estimate-prompt-tokens="true" />

        <!-- Backend med managed identity -->
        <set-backend-service backend-id="aoai-pool" />
        <authentication-managed-identity
            resource="https://cognitiveservices.azure.com"
            output-token-variable-name="mi-token" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (string)context.Variables["mi-token"])</value>
        </set-header>
    </inbound>

    <backend>
        <!-- Streaming-kompatibel forwarding -->
        <forward-request timeout="240"
                         fail-on-error-status-code="true"
                         buffer-response="false" />
    </backend>

    <outbound>
        <base />

        <!-- Token-metriker (fungerer for streaming og ikke-streaming) -->
        <llm-emit-token-metric namespace="ai-metrics">
            <dimension name="Subscription" value="@(context.Subscription.Name)" />
            <dimension name="API" value="@(context.Api.Name)" />
            <dimension name="Region" value="@(context.Deployment.Region)" />
        </llm-emit-token-metric>
    </outbound>

    <on-error>
        <base />
    </on-error>
</policies>

---

Referanser

• Configure API for server-sent events — Offisiell SSE-guide for APIM
• AI gateway in Azure API Management — AI gateway oversikt
• Azure OpenAI REST API reference - Chat Completions — Stream-parameter dokumentasjon
• forward-request policy — Policy-referanse for forwarding
• Log token usage, prompts, and completions — LLM-logging i APIM

---

For Cosmo

• Bruk denne referansen når kunder implementerer chat-applikasjoner eller AI-assistenter gjennom APIM og trenger streaming-støtte.
• Den absolutt viktigste innstillingen er buffer-response="false" i forward-request. Uten dette bufres hele SSE-responsen og leveres som én stor blob — som dreper brukeropplevelsen.
• Advar om at Consumption-tier IKKE støtter langvarige HTTP-forbindelser som SSE krever. Anbefal v2 eller Premium tier for streaming-scenarier.
• For logging av streaming-requests: Bruk llm-emit-token-metric for token-metriker (fungerer med streaming). Unngå response body logging som bryter streaming.
• Kombiner streaming med retry-policy forsiktig — retry fungerer kun for initial connection failure, ikke for mid-stream feil.