ms-ai-architect/skills/ms-ai-security/references/performance-scalability/streaming-response-patterns.md

# Streaming Response Patterns

**Last updated:** 2026-02
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference

---

## Innhold

- [Introduksjon](#introduksjon)
- [Server-Sent Events (SSE) Grunnleggende](#server-sent-events-sse-grunnleggende)
- [Grunnleggende Streaming-implementasjon](#grunnleggende-streaming-implementasjon)
- [Chunked Transfer Encoding](#chunked-transfer-encoding)
- [Client-Side Stream Handling](#client-side-stream-handling)
- [Error Recovery in Streams](#error-recovery-in-streams)
- [Nar bruke streaming vs. non-streaming](#nar-bruke-streaming-vs-non-streaming)
- [Avanserte monstre](#avanserte-monstre)
- [Ytelsesmal for streaming](#ytelsesmal-for-streaming)
- [For Cosmo](#for-cosmo)

## Introduksjon

Streaming av AI-responser er en kritisk teknikk for a forbedre brukeropplevelsen i interaktive AI-applikasjoner. Istedenfor a vente pa at hele responsen genereres for den vises, lar streaming brukeren se svaret bygges opp token for token. For norsk offentlig sektor, der innbyggerportaler og saksbehandlingssystemer i okende grad integrerer AI, er streaming avgjorende for akseptabel responstid.

Azure OpenAI stotter streaming gjennom Server-Sent Events (SSE)-protokollen, som er en enkel, unidireksjonell strommingsmekanisme over HTTP. Denne tilnaermingen er spesielt verdifull for chat-grensesnitt, dokumentgenerering og andre bruksomrader der brukeren forventer umiddelbar tilbakemelding.

Denne referansen dekker arkitekturmonstre for streaming i Azure OpenAI-baserte applikasjoner, fra grunnleggende SSE-implementasjon til avansert feilhandtering og mellomlag-konfigurasjon.

## Server-Sent Events (SSE) Grunnleggende

### Hva er SSE?

Server-Sent Events er en W3C-standard for enveis stromming fra server til klient over HTTP:

| Egenskap | SSE | WebSocket | Long Polling |
|----------|-----|-----------|--------------|
| Retning | Server -> Klient | Bidireksjonell | Klient -> Server -> Klient |
| Protokoll | HTTP/1.1+ | WebSocket (ws://) | HTTP |
| Automatisk reconnect | Ja (innebygd) | Nei (manuell) | Nei |
| Kompleksitet | Lav | Hoy | Middels |
| Azure OpenAI-stotte | Ja | Ja (Realtime API) | Nei |

### SSE-format

Azure OpenAI returnerer data i SSE-format:

```
HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hei"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]
```

**Viktige SSE-regler:**
- Hver hendelse er prefixet med `data: `
- Hendelser separeres med to linjeskift (`\n\n`)
- Siste hendelse er alltid `data: [DONE]`
- `delta`-feltet inneholder inkrementelt innhold (ikke kumulativt)
- `finish_reason` er `null` til generering er ferdig

## Grunnleggende Streaming-implementasjon

### Python med Azure OpenAI SDK

```python
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

def stream_chat_response(user_message: str) -> str:
    """Stream en chat completion og bygg opp komplett respons."""
    full_response = ""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": "Du er en hjelpesom assistent."},
            {"role": "user", "content": user_message}
        ],
        stream=True,
        max_tokens=500
    )

    for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            full_response += content
            print(content, end="", flush=True)  # Vis inkrementelt

    print()  # Ny linje etter ferdig streaming
    return full_response
```

### Async Python Streaming

```python
from openai import AsyncAzureOpenAI
import asyncio

async_client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

async def stream_async(user_message: str):
    """Asynkron streaming for hoy-throughput applikasjoner."""
    response = await async_client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": user_message}],
        stream=True,
        max_tokens=500
    )

    collected_content = []
    async for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            collected_content.append(content)
            yield content  # Yield for videre prosessering

    return "".join(collected_content)
```

### TypeScript/JavaScript Streaming

```typescript
import { AzureOpenAI } from "openai";

const client = new AzureOpenAI({
  endpoint: "https://your-resource.openai.azure.com/",
  apiKey: "your-api-key",
  apiVersion: "2025-03-01-preview",
});

async function* streamChatResponse(
  userMessage: string
): AsyncGenerator<string> {
  const stream = await client.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: userMessage }],
    stream: true,
    max_tokens: 500,
  });

  for await (const chunk of stream) {
    const content = chunk.choices[0]?.delta?.content;
    if (content) {
      yield content;
    }
  }
}

// Bruk i en web-handler
async function handleStreamRequest(req: Request): Promise<Response> {
  const encoder = new TextEncoder();
  const readableStream = new ReadableStream({
    async start(controller) {
      for await (const token of streamChatResponse("Hva er GDPR?")) {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify({ content: token })}\n\n`));
      }
      controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      controller.close();
    },
  });

  return new Response(readableStream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      "Connection": "keep-alive",
    },
  });
}
```

## Chunked Transfer Encoding

### HTTP-konfigurasjon for streaming

For at streaming skal fungere gjennom hele infrastrukturen, ma alle mellomlag konfigureres korrekt:

| Komponent | Nodvendig konfigurasjon |
|-----------|------------------------|
| Azure OpenAI | `stream: true` i request |
| API Management | `buffer-response="false"` i forward-request |
| Application Gateway | Deaktiver response buffering |
| Azure Front Door | Route-spesifikk konfigurasjon |
| Klient (browser) | `Accept: text/event-stream` header |

### API Management for SSE

```xml
<!-- APIM policy for SSE pass-through -->
<policies>
    <inbound>
        <base />
    </inbound>
    <backend>
        <!-- KRITISK: buffer-response="false" for streaming -->
        <forward-request timeout="120"
                        fail-on-error-status-code="true"
                        buffer-response="false" />
    </backend>
    <outbound>
        <base />
        <!-- VIKTIG: Deaktiver body-logging for SSE-APIer -->
    </outbound>
    <on-error>
        <base />
    </on-error>
</policies>
```

**Viktige APIM-hensyn for SSE:**
1. Deaktiver response buffering (`buffer-response="false"`)
2. Deaktiver `validate-content`-policy (buffrer respons)
3. Deaktiver request/response body-logging for Azure Monitor og Application Insights
4. Deaktiver response caching for streaming-endepunkter
5. Okt timeout (minimum 120 sekunder)
6. Hold forbindelser i live med TCP keepalive

### Application Gateway for SSE

```json
{
  "properties": {
    "responseBufferPolicy": {
      "responseSendTimeoutInSeconds": 120,
      "bufferResponseBody": false
    },
    "backendHttpSettings": {
      "requestTimeout": 120,
      "connectionDraining": {
        "enabled": true,
        "drainTimeoutInSec": 30
      }
    }
  }
}
```

### Azure Front Door Route Policy

For SSE gjennom Azure Front Door:

```json
{
  "routePolicy": {
    "routeTimeout": "0s"
  }
}
```

**Merk:** Idle timeout for Application Gateway for Containers er 5 minutter. Send keepalive-meldinger for a forhindre at forbindelsen lukkes:

```
: keep-alive\n\n
```

## Client-Side Stream Handling

### React/Next.js Frontend

```typescript
// React hook for SSE streaming fra Azure OpenAI
import { useState, useCallback } from "react";

interface StreamState {
  content: string;
  isStreaming: boolean;
  error: string | null;
}

function useAIStream() {
  const [state, setState] = useState<StreamState>({
    content: "",
    isStreaming: false,
    error: null,
  });

  const startStream = useCallback(async (prompt: string) => {
    setState({ content: "", isStreaming: true, error: null });

    try {
      const response = await fetch("/api/chat", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Accept: "text/event-stream",
        },
        body: JSON.stringify({ message: prompt }),
      });

      if (!response.ok) {
        throw new Error(`HTTP ${response.status}: ${response.statusText}`);
      }

      const reader = response.body?.getReader();
      const decoder = new TextDecoder();

      if (!reader) throw new Error("No reader available");

      let accumulated = "";

      while (true) {
        const { done, value } = await reader.read();
        if (done) break;

        const chunk = decoder.decode(value, { stream: true });
        const lines = chunk.split("\n");

        for (const line of lines) {
          if (line.startsWith("data: ")) {
            const data = line.slice(6);
            if (data === "[DONE]") continue;

            try {
              const parsed = JSON.parse(data);
              if (parsed.content) {
                accumulated += parsed.content;
                setState((prev) => ({
                  ...prev,
                  content: accumulated,
                }));
              }
            } catch {
              // Ignorer parsing-feil for ufullstendige chunks
            }
          }
        }
      }

      setState((prev) => ({ ...prev, isStreaming: false }));
    } catch (error) {
      setState((prev) => ({
        ...prev,
        isStreaming: false,
        error: error instanceof Error ? error.message : "Ukjent feil",
      }));
    }
  }, []);

  return { ...state, startStream };
}
```

### Python SSE Client

```python
import httpx
import json
from typing import AsyncGenerator

async def consume_sse_stream(
    url: str,
    payload: dict,
    api_key: str
) -> AsyncGenerator[str, None]:
    """Konsumer SSE-strom fra Azure OpenAI via HTTP."""
    headers = {
        "Content-Type": "application/json",
        "api-key": api_key,
        "Accept": "text/event-stream"
    }

    async with httpx.AsyncClient(timeout=120.0) as client:
        async with client.stream("POST", url, json=payload, headers=headers) as response:
            response.raise_for_status()

            buffer = ""
            async for chunk in response.aiter_text():
                buffer += chunk
                while "\n\n" in buffer:
                    event, buffer = buffer.split("\n\n", 1)
                    for line in event.split("\n"):
                        if line.startswith("data: "):
                            data = line[6:]
                            if data == "[DONE]":
                                return
                            try:
                                parsed = json.loads(data)
                                content = parsed["choices"][0]["delta"].get("content", "")
                                if content:
                                    yield content
                            except (json.JSONDecodeError, KeyError, IndexError):
                                continue
```

## Error Recovery in Streams

### Haandtering av avbrutte strommer

Streaming-forbindelser kan avbrytes av flere arsaker:

| Feiltype | Arsak | Handtering |
|----------|-------|------------|
| Nettverksavbrudd | Ustabil forbindelse | Reconnect med checkpoint |
| Timeout | Idle > 4 min (Azure LB) | Keepalive-meldinger |
| 429 Rate Limit | Kapasitetsgrense | Retry med backoff |
| 500 Server Error | Midlertidig serverfeil | Retry etter pause |
| Content Filter | Innhold blokkert | Vis melding til bruker |

### Robust Streaming med Retry

```python
import asyncio
import time
from openai import AsyncAzureOpenAI, APIStatusError, APIConnectionError

async_client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

async def resilient_stream(
    messages: list,
    max_retries: int = 3,
    model: str = "gpt-4o"
) -> AsyncGenerator[str, None]:
    """Streaming med automatisk retry og feilhandtering."""
    collected_tokens = []
    attempt = 0

    while attempt < max_retries:
        try:
            response = await async_client.chat.completions.create(
                model=model,
                messages=messages,
                stream=True,
                max_tokens=1000
            )

            async for chunk in response:
                if chunk.choices and chunk.choices[0].delta.content:
                    token = chunk.choices[0].delta.content
                    collected_tokens.append(token)
                    yield token

                # Sjekk finish_reason
                if chunk.choices and chunk.choices[0].finish_reason:
                    reason = chunk.choices[0].finish_reason
                    if reason == "content_filter":
                        yield "\n[Innhold filtrert av sikkerhetsfilter]"
                    return  # Ferdig

            return  # Stromming fullfort

        except APIStatusError as e:
            attempt += 1
            if e.status_code == 429:
                retry_after = int(e.response.headers.get("retry-after", "5"))
                await asyncio.sleep(retry_after)
            elif e.status_code >= 500:
                await asyncio.sleep(2 ** attempt)  # Eksponentiell backoff
            else:
                raise  # Ikke-gjenforsokbar feil

        except APIConnectionError:
            attempt += 1
            await asyncio.sleep(2 ** attempt)

    raise Exception(f"Streaming feilet etter {max_retries} forsok")
```

### Streaming med Partial Response Recovery

```python
async def stream_with_checkpoint(
    messages: list,
    on_token: callable,
    on_complete: callable,
    on_error: callable
):
    """Stream med checkpoint for delvis gjenoppretting."""
    partial_response = []
    last_chunk_time = time.time()

    try:
        response = await async_client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            stream=True,
            max_tokens=1000
        )

        async for chunk in response:
            current_time = time.time()

            # Detekter unormalt lang pause mellom chunks
            if current_time - last_chunk_time > 30:
                # Mulig hengende forbindelse
                break

            last_chunk_time = current_time

            if chunk.choices and chunk.choices[0].delta.content:
                token = chunk.choices[0].delta.content
                partial_response.append(token)
                await on_token(token)

            if chunk.choices and chunk.choices[0].finish_reason == "stop":
                await on_complete("".join(partial_response))
                return

        # Hvis vi nar hit uten "stop", har streamingen avbrultt
        if partial_response:
            await on_complete(
                "".join(partial_response) +
                "\n\n[Merk: Respons kan vaere ufullstendig]"
            )

    except Exception as e:
        if partial_response:
            await on_error(e, "".join(partial_response))
        else:
            await on_error(e, None)
```

## Nar bruke streaming vs. non-streaming

| Scenario | Anbefaling | Begrunnelse |
|----------|-----------|-------------|
| Chat-grensesnitt | Streaming | Bedre opplevd responstid |
| Innbyggerportal | Streaming | Visuell tilbakemelding under generering |
| Batch-klassifisering | Non-streaming | Kun sluttresultat er relevant |
| Dokumentanalyse | Non-streaming | Strukturert output, ingen inkrementell visning |
| Saksbehandlingsforslag | Streaming | Lang generering, bruker venter |
| API-integrasjon (maskin-til-maskin) | Non-streaming | Enklere feilhandtering |
| Sanntidsoversetning | Streaming | Lavest opplevd latens |

## Avanserte monstre

### Server-side Streaming Proxy med FastAPI

```python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncAzureOpenAI

app = FastAPI()
client = AsyncAzureOpenAI(
    azure_endpoint="https://your-resource.openai.azure.com/",
    api_key="your-api-key",
    api_version="2025-03-01-preview"
)

@app.post("/api/chat/stream")
async def chat_stream(request: ChatRequest):
    """Server-side proxy for Azure OpenAI streaming."""

    async def generate():
        try:
            response = await client.chat.completions.create(
                model="gpt-4o",
                messages=[{"role": "user", "content": request.message}],
                stream=True,
                max_tokens=1000
            )
            async for chunk in response:
                if chunk.choices and chunk.choices[0].delta.content:
                    data = {"content": chunk.choices[0].delta.content}
                    yield f"data: {json.dumps(data)}\n\n"

            yield "data: [DONE]\n\n"

        except Exception as e:
            error_data = {"error": str(e)}
            yield f"data: {json.dumps(error_data)}\n\n"

    return StreamingResponse(
        generate(),
        media_type="text/event-stream",
        headers={
            "Cache-Control": "no-cache",
            "Connection": "keep-alive",
            "X-Accel-Buffering": "no"  # Deaktiver nginx buffering
        }
    )
```

### Token-telling under streaming

```python
import tiktoken

async def stream_with_token_counting(messages: list, model: str = "gpt-4o"):
    """Stream med sanntids token-telling for kostnadsovervaking."""
    encoding = tiktoken.encoding_for_model(model)
    output_tokens = 0

    response = await async_client.chat.completions.create(
        model=model,
        messages=messages,
        stream=True,
        stream_options={"include_usage": True}  # Inkluder bruksdata
    )

    async for chunk in response:
        if chunk.choices and chunk.choices[0].delta.content:
            content = chunk.choices[0].delta.content
            output_tokens += len(encoding.encode(content))
            yield content

        # Sjekk usage i siste chunk
        if chunk.usage:
            actual_tokens = chunk.usage.completion_tokens
            cached_tokens = getattr(
                chunk.usage.prompt_tokens_details, 'cached_tokens', 0
            )
            print(f"Faktisk token-bruk: {actual_tokens}")
            print(f"Cache-treff: {cached_tokens}")
```

## Ytelsesmal for streaming

| Metrikk | Mal (P95) | Kritisk terskel |
|---------|-----------|-----------------|
| Time to First Token | < 500 ms | > 2000 ms |
| Inter-token latens | < 50 ms | > 200 ms |
| Reconnect-tid | < 2 s | > 10 s |
| Stream completion rate | > 99% | < 95% |

## For Cosmo

- **Streaming er obligatorisk** for alle brukerrettede AI-grensesnitt. Forskjellen i opplevd latens er dramatisk: 200 ms TTFT vs. 3-5 sekunders ventetid for komplett respons.
- **Infrastruktur-konfigurasjon er kritisk:** Hele kjeden (APIM, App Gateway, Front Door) ma ha response buffering deaktivert. En enkelt feilkonfigurert komponent blokkerer all streaming.
- **Feilhandtering i strommer krever eget design:** Implementer alltid reconnect-logikk, partial response recovery, og eksponentiell backoff for 429/5xx-feil.
- **Content filtering pavirker streaming:** `finish_reason: content_filter` kan oppsta midt i en strom. Klient-koden ma handtere dette gracefully med en brukermelding.
- **Token-telling under streaming:** Bruk `stream_options: {"include_usage": true}` for a fa eksakt token-bruk i siste chunk, viktig for kostnadsovervaking.