ms-ai-architect/skills/ms-ai-security/references/performance-scalability/streaming-response-patterns.md

Streaming Response Patterns

Last updated: 2026-02 Status: GA Category: Performance & Scalability

---

Introduksjon

Streaming av AI-responser er en kritisk teknikk for a forbedre brukeropplevelsen i interaktive AI-applikasjoner. Istedenfor a vente pa at hele responsen genereres for den vises, lar streaming brukeren se svaret bygges opp token for token. For norsk offentlig sektor, der innbyggerportaler og saksbehandlingssystemer i okende grad integrerer AI, er streaming avgjorende for akseptabel responstid.

Azure OpenAI stotter streaming gjennom Server-Sent Events (SSE)-protokollen, som er en enkel, unidireksjonell strommingsmekanisme over HTTP. Denne tilnaermingen er spesielt verdifull for chat-grensesnitt, dokumentgenerering og andre bruksomrader der brukeren forventer umiddelbar tilbakemelding.

Denne referansen dekker arkitekturmonstre for streaming i Azure OpenAI-baserte applikasjoner, fra grunnleggende SSE-implementasjon til avansert feilhandtering og mellomlag-konfigurasjon.

Server-Sent Events (SSE) Grunnleggende

Hva er SSE?

Server-Sent Events er en W3C-standard for enveis stromming fra server til klient over HTTP:

Egenskap	SSE	WebSocket	Long Polling
Retning	Server -> Klient	Bidireksjonell	Klient -> Server -> Klient
Protokoll	HTTP/1.1+	WebSocket (ws://)	HTTP
Automatisk reconnect	Ja (innebygd)	Nei (manuell)	Nei
Kompleksitet	Lav	Hoy	Middels
Azure OpenAI-stotte	Ja	Ja (Realtime API)	Nei

SSE-format

Azure OpenAI returnerer data i SSE-format:

HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hei"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Viktige SSE-regler:

• Hver hendelse er prefixet med data:
• Hendelser separeres med to linjeskift (\n\n)
• Siste hendelse er alltid data: [DONE]
• delta-feltet inneholder inkrementelt innhold (ikke kumulativt)
• finish_reason er null til generering er ferdig

Grunnleggende Streaming-implementasjon

Python med Azure OpenAI SDK

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

def stream_chat_response(user_message: str) -> str:
    """Stream en chat completion og bygg opp komplett respons."""
    full_response = ""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Du er en hjelpesom assistent."},
            {"role": "user", "content": user_message}
        ],
        stream=True,
        max_tokens=500
    )

    for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)  # Vis inkrementelt

    print()  # Ny linje etter ferdig streaming
    return full_response

Async Python Streaming

from openai import AsyncAzureOpenAI
import asyncio

async_client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

async def stream_async(user_message: str):
    """Asynkron streaming for hoy-throughput applikasjoner."""
    response = await async_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_message}],
        stream=True,
        max_tokens=500
    )

    collected_content = []
    async for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            collected_content.append(content)
            yield content  # Yield for videre prosessering

    return "".join(collected_content)

TypeScript/JavaScript Streaming

import { AzureOpenAI } from "openai";

const client = new AzureOpenAI({
  endpoint: "https://your-resource.openai.azure.com/",
  apiKey: "your-api-key",
  apiVersion: "2025-03-01-preview",
});

async function* streamChatResponse(
  userMessage: string
): AsyncGenerator<string> {
  const stream = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: userMessage }],
    stream: true,
    max_tokens: 500,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      yield content;
    }
  }
}

// Bruk i en web-handler
async function handleStreamRequest(req: Request): Promise<Response> {
  const encoder = new TextEncoder();
  const readableStream = new ReadableStream({
    async start(controller) {
      for await (const token of streamChatResponse("Hva er GDPR?")) {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify({ content: token })}\n\n`));
      }
      controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      controller.close();
    },
  });

  return new Response(readableStream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      "Connection": "keep-alive",
    },
  });
}

Chunked Transfer Encoding

HTTP-konfigurasjon for streaming

For at streaming skal fungere gjennom hele infrastrukturen, ma alle mellomlag konfigureres korrekt:

Komponent	Nodvendig konfigurasjon
Azure OpenAI	stream: true i request
API Management	buffer-response="false" i forward-request
Application Gateway	Deaktiver response buffering
Azure Front Door	Route-spesifikk konfigurasjon
Klient (browser)	Accept: text/event-stream header

API Management for SSE

<!-- APIM policy for SSE pass-through -->
<policies>
    <inbound>
        <base />
    </inbound>
    <backend>
        <!-- KRITISK: buffer-response="false" for streaming -->
        <forward-request timeout="120"
                        fail-on-error-status-code="true"
                        buffer-response="false" />
    </backend>
    <outbound>
        <base />
        <!-- VIKTIG: Deaktiver body-logging for SSE-APIer -->
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>

Viktige APIM-hensyn for SSE:

1. Deaktiver response buffering (buffer-response="false")
2. Deaktiver validate-content-policy (buffrer respons)
3. Deaktiver request/response body-logging for Azure Monitor og Application Insights
4. Deaktiver response caching for streaming-endepunkter
5. Okt timeout (minimum 120 sekunder)
6. Hold forbindelser i live med TCP keepalive

Application Gateway for SSE

{
  "properties": {
    "responseBufferPolicy": {
      "responseSendTimeoutInSeconds": 120,
      "bufferResponseBody": false
    },
    "backendHttpSettings": {
      "requestTimeout": 120,
      "connectionDraining": {
        "enabled": true,
        "drainTimeoutInSec": 30
      }
    }
  }
}

Azure Front Door Route Policy

For SSE gjennom Azure Front Door:

{
  "routePolicy": {
    "routeTimeout": "0s"
  }
}

Merk: Idle timeout for Application Gateway for Containers er 5 minutter. Send keepalive-meldinger for a forhindre at forbindelsen lukkes:

: keep-alive\n\n

Client-Side Stream Handling

React/Next.js Frontend

// React hook for SSE streaming fra Azure OpenAI
import { useState, useCallback } from "react";

interface StreamState {
  content: string;
  isStreaming: boolean;
  error: string | null;
}

function useAIStream() {
  const [state, setState] = useState<StreamState>({
    content: "",
    isStreaming: false,
    error: null,
  });

  const startStream = useCallback(async (prompt: string) => {
    setState({ content: "", isStreaming: true, error: null });

    try {
      const response = await fetch("/api/chat", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Accept: "text/event-stream",
        },
        body: JSON.stringify({ message: prompt }),
      });

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      const reader = response.body?.getReader();
      const decoder = new TextDecoder();

      if (!reader) throw new Error("No reader available");

      let accumulated = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const chunk = decoder.decode(value, { stream: true });
        const lines = chunk.split("\n");

        for (const line of lines) {
          if (line.startsWith("data: ")) {
            const data = line.slice(6);
            if (data === "[DONE]") continue;

            try {
              const parsed = JSON.parse(data);
              if (parsed.content) {
                accumulated += parsed.content;
                setState((prev) => ({
                  ...prev,
                  content: accumulated,
                }));
              }
            } catch {
              // Ignorer parsing-feil for ufullstendige chunks
            }
          }
        }
      }

      setState((prev) => ({ ...prev, isStreaming: false }));
    } catch (error) {
      setState((prev) => ({
        ...prev,
        isStreaming: false,
        error: error instanceof Error ? error.message : "Ukjent feil",
      }));
    }
  }, []);

  return { ...state, startStream };
}

Python SSE Client

import httpx
import json
from typing import AsyncGenerator

async def consume_sse_stream(
    url: str,
    payload: dict,
    api_key: str
) -> AsyncGenerator[str, None]:
    """Konsumer SSE-strom fra Azure OpenAI via HTTP."""
    headers = {
        "Content-Type": "application/json",
        "api-key": api_key,
        "Accept": "text/event-stream"
    }

    async with httpx.AsyncClient(timeout=120.0) as client:
        async with client.stream("POST", url, json=payload, headers=headers) as response:
            response.raise_for_status()

            buffer = ""
            async for chunk in response.aiter_text():
                buffer += chunk
                while "\n\n" in buffer:
                    event, buffer = buffer.split("\n\n", 1)
                    for line in event.split("\n"):
                        if line.startswith("data: "):
                            data = line[6:]
                            if data == "[DONE]":
                                return
                            try:
                                parsed = json.loads(data)
                                content = parsed["choices"][0]["delta"].get("content", "")
                                if content:
                                    yield content
                            except (json.JSONDecodeError, KeyError, IndexError):
                                continue

Error Recovery in Streams

Haandtering av avbrutte strommer

Streaming-forbindelser kan avbrytes av flere arsaker:

Feiltype	Arsak	Handtering
Nettverksavbrudd	Ustabil forbindelse	Reconnect med checkpoint
Timeout	Idle > 4 min (Azure LB)	Keepalive-meldinger
429 Rate Limit	Kapasitetsgrense	Retry med backoff
500 Server Error	Midlertidig serverfeil	Retry etter pause
Content Filter	Innhold blokkert	Vis melding til bruker

Robust Streaming med Retry

import asyncio
import time
from openai import AsyncAzureOpenAI, APIStatusError, APIConnectionError

async_client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

async def resilient_stream(
    messages: list,
    max_retries: int = 3,
    model: str = "gpt-4o"
) -> AsyncGenerator[str, None]:
    """Streaming med automatisk retry og feilhandtering."""
    collected_tokens = []
    attempt = 0

    while attempt < max_retries:
        try:
            response = await async_client.chat.completions.create(
                model=model,
                messages=messages,
                stream=True,
                max_tokens=1000
            )

            async for chunk in response:
                if chunk.choices and chunk.choices[0].delta.content:
                    token = chunk.choices[0].delta.content
                    collected_tokens.append(token)
                    yield token

                # Sjekk finish_reason
                if chunk.choices and chunk.choices[0].finish_reason:
                    reason = chunk.choices[0].finish_reason
                    if reason == "content_filter":
                        yield "\n[Innhold filtrert av sikkerhetsfilter]"
                    return  # Ferdig

            return  # Stromming fullfort

        except APIStatusError as e:
            attempt += 1
            if e.status_code == 429:
                retry_after = int(e.response.headers.get("retry-after", "5"))
                await asyncio.sleep(retry_after)
            elif e.status_code >= 500:
                await asyncio.sleep(2 ** attempt)  # Eksponentiell backoff
            else:
                raise  # Ikke-gjenforsokbar feil

        except APIConnectionError:
            attempt += 1
            await asyncio.sleep(2 ** attempt)

    raise Exception(f"Streaming feilet etter {max_retries} forsok")

Streaming med Partial Response Recovery

async def stream_with_checkpoint(
    messages: list,
    on_token: callable,
    on_complete: callable,
    on_error: callable
):
    """Stream med checkpoint for delvis gjenoppretting."""
    partial_response = []
    last_chunk_time = time.time()

    try:
        response = await async_client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            stream=True,
            max_tokens=1000
        )

        async for chunk in response:
            current_time = time.time()

            # Detekter unormalt lang pause mellom chunks
            if current_time - last_chunk_time > 30:
                # Mulig hengende forbindelse
                break

            last_chunk_time = current_time

            if chunk.choices and chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                partial_response.append(token)
                await on_token(token)

            if chunk.choices and chunk.choices[0].finish_reason == "stop":
                await on_complete("".join(partial_response))
                return

        # Hvis vi nar hit uten "stop", har streamingen avbrultt
        if partial_response:
            await on_complete(
                "".join(partial_response) +
                "\n\n[Merk: Respons kan vaere ufullstendig]"
            )

    except Exception as e:
        if partial_response:
            await on_error(e, "".join(partial_response))
        else:
            await on_error(e, None)

Nar bruke streaming vs. non-streaming

Scenario	Anbefaling	Begrunnelse
Chat-grensesnitt	Streaming	Bedre opplevd responstid
Innbyggerportal	Streaming	Visuell tilbakemelding under generering
Batch-klassifisering	Non-streaming	Kun sluttresultat er relevant
Dokumentanalyse	Non-streaming	Strukturert output, ingen inkrementell visning
Saksbehandlingsforslag	Streaming	Lang generering, bruker venter
API-integrasjon (maskin-til-maskin)	Non-streaming	Enklere feilhandtering
Sanntidsoversetning	Streaming	Lavest opplevd latens

Avanserte monstre

Server-side Streaming Proxy med FastAPI

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncAzureOpenAI

app = FastAPI()
client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

@app.post("/api/chat/stream")
async def chat_stream(request: ChatRequest):
    """Server-side proxy for Azure OpenAI streaming."""

    async def generate():
        try:
            response = await client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": request.message}],
                stream=True,
                max_tokens=1000
            )
            async for chunk in response:
                if chunk.choices and chunk.choices[0].delta.content:
                    data = {"content": chunk.choices[0].delta.content}
                    yield f"data: {json.dumps(data)}\n\n"

            yield "data: [DONE]\n\n"

        except Exception as e:
            error_data = {"error": str(e)}
            yield f"data: {json.dumps(error_data)}\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no"  # Deaktiver nginx buffering
        }
    )

Token-telling under streaming

import tiktoken

async def stream_with_token_counting(messages: list, model: str = "gpt-4o"):
    """Stream med sanntids token-telling for kostnadsovervaking."""
    encoding = tiktoken.encoding_for_model(model)
    output_tokens = 0

    response = await async_client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        stream_options={"include_usage": True}  # Inkluder bruksdata
    )

    async for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            output_tokens += len(encoding.encode(content))
            yield content

        # Sjekk usage i siste chunk
        if chunk.usage:
            actual_tokens = chunk.usage.completion_tokens
            cached_tokens = getattr(
                chunk.usage.prompt_tokens_details, 'cached_tokens', 0
            )
            print(f"Faktisk token-bruk: {actual_tokens}")
            print(f"Cache-treff: {cached_tokens}")

Ytelsesmal for streaming

Metrikk	Mal (P95)	Kritisk terskel
Time to First Token	< 500 ms	> 2000 ms
Inter-token latens	< 50 ms	> 200 ms
Reconnect-tid	< 2 s	> 10 s
Stream completion rate	> 99%	< 95%

For Cosmo

• Streaming er obligatorisk for alle brukerrettede AI-grensesnitt. Forskjellen i opplevd latens er dramatisk: 200 ms TTFT vs. 3-5 sekunders ventetid for komplett respons.
• Infrastruktur-konfigurasjon er kritisk: Hele kjeden (APIM, App Gateway, Front Door) ma ha response buffering deaktivert. En enkelt feilkonfigurert komponent blokkerer all streaming.
• Feilhandtering i strommer krever eget design: Implementer alltid reconnect-logikk, partial response recovery, og eksponentiell backoff for 429/5xx-feil.
• Content filtering pavirker streaming: finish_reason: content_filter kan oppsta midt i en strom. Klient-koden ma handtere dette gracefully med en brukermelding.
• Token-telling under streaming: Bruk stream_options: {"include_usage": true} for a fa eksakt token-bruk i siste chunk, viktig for kostnadsovervaking.