open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Browse source 
Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

634
skills/ms-ai-security/references/performance-scalability/streaming-response-patterns.md Normal file
View file

@ -0,0 +1,634 @@
# Streaming Response Patterns
**Last updated:** 2026-02
**Status:** GA
**Category:** Performance & Scalability
---
## Introduksjon
Streaming av AI-responser er en kritisk teknikk for a forbedre brukeropplevelsen i interaktive AI-applikasjoner. Istedenfor a vente pa at hele responsen genereres for den vises, lar streaming brukeren se svaret bygges opp token for token. For norsk offentlig sektor, der innbyggerportaler og saksbehandlingssystemer i okende grad integrerer AI, er streaming avgjorende for akseptabel responstid.
Azure OpenAI stotter streaming gjennom Server-Sent Events (SSE)-protokollen, som er en enkel, unidireksjonell strommingsmekanisme over HTTP. Denne tilnaermingen er spesielt verdifull for chat-grensesnitt, dokumentgenerering og andre bruksomrader der brukeren forventer umiddelbar tilbakemelding.
Denne referansen dekker arkitekturmonstre for streaming i Azure OpenAI-baserte applikasjoner, fra grunnleggende SSE-implementasjon til avansert feilhandtering og mellomlag-konfigurasjon.
## Server-Sent Events (SSE) Grunnleggende
### Hva er SSE?
Server-Sent Events er en W3C-standard for enveis stromming fra server til klient over HTTP:
| Egenskap | SSE | WebSocket | Long Polling |
|----------|-----|-----------|--------------|
| Retning | Server -> Klient | Bidireksjonell | Klient -> Server -> Klient |
| Protokoll | HTTP/1.1+ | WebSocket (ws://) | HTTP |
| Automatisk reconnect | Ja (innebygd) | Nei (manuell) | Nei |
| Kompleksitet | Lav | Hoy | Middels |
| Azure OpenAI-stotte | Ja | Ja (Realtime API) | Nei |
### SSE-format
Azure OpenAI returnerer data i SSE-format:
```
HTTP/1.1 200 OK
Content-Type: text/event-stream; charset=utf-8
Transfer-Encoding: chunked
Cache-Control: no-cache
Connection: keep-alive
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"Hei"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}
data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1694268190,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]
```
**Viktige SSE-regler:**
- Hver hendelse er prefixet med `data: `
- Hendelser separeres med to linjeskift (`\n\n`)
- Siste hendelse er alltid `data: [DONE]`
- `delta`-feltet inneholder inkrementelt innhold (ikke kumulativt)
- `finish_reason` er `null` til generering er ferdig
## Grunnleggende Streaming-implementasjon
### Python med Azure OpenAI SDK
```python
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint="https://your-resource.openai.azure.com/",
api_key="your-api-key",
api_version="2025-03-01-preview"
)
def stream_chat_response(user_message: str) -> str:
"""Stream en chat completion og bygg opp komplett respons."""
full_response = ""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": "Du er en hjelpesom assistent."},
{"role": "user", "content": user_message}
],
stream=True,
max_tokens=500
)
for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
full_response += content
print(content, end="", flush=True) # Vis inkrementelt
print() # Ny linje etter ferdig streaming
return full_response
```
### Async Python Streaming
```python
from openai import AsyncAzureOpenAI
import asyncio
async_client = AsyncAzureOpenAI(
azure_endpoint="https://your-resource.openai.azure.com/",
api_key="your-api-key",
api_version="2025-03-01-preview"
)
async def stream_async(user_message: str):
"""Asynkron streaming for hoy-throughput applikasjoner."""
response = await async_client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": user_message}],
stream=True,
max_tokens=500
)
collected_content = []
async for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
collected_content.append(content)
yield content # Yield for videre prosessering
return "".join(collected_content)
```
### TypeScript/JavaScript Streaming
```typescript
import { AzureOpenAI } from "openai";
const client = new AzureOpenAI({
endpoint: "https://your-resource.openai.azure.com/",
apiKey: "your-api-key",
apiVersion: "2025-03-01-preview",
});
async function* streamChatResponse(
userMessage: string
): AsyncGenerator<string> {
const stream = await client.chat.completions.create({
model: "gpt-4o",
messages: [{ role: "user", content: userMessage }],
stream: true,
max_tokens: 500,
});
for await (const chunk of stream) {
const content = chunk.choices[0]?.delta?.content;
if (content) {
yield content;
}
}
}
// Bruk i en web-handler
async function handleStreamRequest(req: Request): Promise<Response> {
const encoder = new TextEncoder();
const readableStream = new ReadableStream({
async start(controller) {
for await (const token of streamChatResponse("Hva er GDPR?")) {
controller.enqueue(encoder.encode(`data: ${JSON.stringify({ content: token })}\n\n`));
}
controller.enqueue(encoder.encode("data: [DONE]\n\n"));
controller.close();
},
});
return new Response(readableStream, {
headers: {
"Content-Type": "text/event-stream",
"Cache-Control": "no-cache",
"Connection": "keep-alive",
},
});
}
```
## Chunked Transfer Encoding
### HTTP-konfigurasjon for streaming
For at streaming skal fungere gjennom hele infrastrukturen, ma alle mellomlag konfigureres korrekt:
| Komponent | Nodvendig konfigurasjon |
|-----------|------------------------|
| Azure OpenAI | `stream: true` i request |
| API Management | `buffer-response="false"` i forward-request |
| Application Gateway | Deaktiver response buffering |
| Azure Front Door | Route-spesifikk konfigurasjon |
| Klient (browser) | `Accept: text/event-stream` header |
### API Management for SSE
```xml
<!-- APIM policy for SSE pass-through -->
<policies>
<inbound>
<base />
</inbound>
<backend>
<!-- KRITISK: buffer-response="false" for streaming -->
<forward-request timeout="120"
fail-on-error-status-code="true"
buffer-response="false" />
</backend>
<outbound>
<base />
<!-- VIKTIG: Deaktiver body-logging for SSE-APIer -->
</outbound>
<on-error>
<base />
</on-error>
</policies>
```
**Viktige APIM-hensyn for SSE:**
1. Deaktiver response buffering (`buffer-response="false"`)
2. Deaktiver `validate-content`-policy (buffrer respons)
3. Deaktiver request/response body-logging for Azure Monitor og Application Insights
4. Deaktiver response caching for streaming-endepunkter
5. Okt timeout (minimum 120 sekunder)
6. Hold forbindelser i live med TCP keepalive
### Application Gateway for SSE
```json
{
"properties": {
"responseBufferPolicy": {
"responseSendTimeoutInSeconds": 120,
"bufferResponseBody": false
},
"backendHttpSettings": {
"requestTimeout": 120,
"connectionDraining": {
"enabled": true,
"drainTimeoutInSec": 30
}
}
}
}
```
### Azure Front Door Route Policy
For SSE gjennom Azure Front Door:
```json
{
"routePolicy": {
"routeTimeout": "0s"
}
}
```
**Merk:** Idle timeout for Application Gateway for Containers er 5 minutter. Send keepalive-meldinger for a forhindre at forbindelsen lukkes:
```
: keep-alive\n\n
```
## Client-Side Stream Handling
### React/Next.js Frontend
```typescript
// React hook for SSE streaming fra Azure OpenAI
import { useState, useCallback } from "react";
interface StreamState {
content: string;
isStreaming: boolean;
error: string | null;
}
function useAIStream() {
const [state, setState] = useState<StreamState>({
content: "",
isStreaming: false,
error: null,
});
const startStream = useCallback(async (prompt: string) => {
setState({ content: "", isStreaming: true, error: null });
try {
const response = await fetch("/api/chat", {
method: "POST",
headers: {
"Content-Type": "application/json",
Accept: "text/event-stream",
},
body: JSON.stringify({ message: prompt }),
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`);
}
const reader = response.body?.getReader();
const decoder = new TextDecoder();
if (!reader) throw new Error("No reader available");
let accumulated = "";
while (true) {
const { done, value } = await reader.read();
if (done) break;
const chunk = decoder.decode(value, { stream: true });
const lines = chunk.split("\n");
for (const line of lines) {
if (line.startsWith("data: ")) {
const data = line.slice(6);
if (data === "[DONE]") continue;
try {
const parsed = JSON.parse(data);
if (parsed.content) {
accumulated += parsed.content;
setState((prev) => ({
...prev,
content: accumulated,
}));
}
} catch {
// Ignorer parsing-feil for ufullstendige chunks
}
}
}
}
setState((prev) => ({ ...prev, isStreaming: false }));
} catch (error) {
setState((prev) => ({
...prev,
isStreaming: false,
error: error instanceof Error ? error.message : "Ukjent feil",
}));
}
}, []);
return { ...state, startStream };
}
```
### Python SSE Client
```python
import httpx
import json
from typing import AsyncGenerator
async def consume_sse_stream(
url: str,
payload: dict,
api_key: str
) -> AsyncGenerator[str, None]:
"""Konsumer SSE-strom fra Azure OpenAI via HTTP."""
headers = {
"Content-Type": "application/json",
"api-key": api_key,
"Accept": "text/event-stream"
}
async with httpx.AsyncClient(timeout=120.0) as client:
async with client.stream("POST", url, json=payload, headers=headers) as response:
response.raise_for_status()
buffer = ""
async for chunk in response.aiter_text():
buffer += chunk
while "\n\n" in buffer:
event, buffer = buffer.split("\n\n", 1)
for line in event.split("\n"):
if line.startswith("data: "):
data = line[6:]
if data == "[DONE]":
return
try:
parsed = json.loads(data)
content = parsed["choices"][0]["delta"].get("content", "")
if content:
yield content
except (json.JSONDecodeError, KeyError, IndexError):
continue
```
## Error Recovery in Streams
### Haandtering av avbrutte strommer
Streaming-forbindelser kan avbrytes av flere arsaker:
| Feiltype | Arsak | Handtering |
|----------|-------|------------|
| Nettverksavbrudd | Ustabil forbindelse | Reconnect med checkpoint |
| Timeout | Idle > 4 min (Azure LB) | Keepalive-meldinger |
| 429 Rate Limit | Kapasitetsgrense | Retry med backoff |
| 500 Server Error | Midlertidig serverfeil | Retry etter pause |
| Content Filter | Innhold blokkert | Vis melding til bruker |
### Robust Streaming med Retry
```python
import asyncio
import time
from openai import AsyncAzureOpenAI, APIStatusError, APIConnectionError
async_client = AsyncAzureOpenAI(
azure_endpoint="https://your-resource.openai.azure.com/",
api_key="your-api-key",
api_version="2025-03-01-preview"
)
async def resilient_stream(
messages: list,
max_retries: int = 3,
model: str = "gpt-4o"
) -> AsyncGenerator[str, None]:
"""Streaming med automatisk retry og feilhandtering."""
collected_tokens = []
attempt = 0
while attempt < max_retries:
try:
response = await async_client.chat.completions.create(
model=model,
messages=messages,
stream=True,
max_tokens=1000
)
async for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
token = chunk.choices[0].delta.content
collected_tokens.append(token)
yield token
# Sjekk finish_reason
if chunk.choices and chunk.choices[0].finish_reason:
reason = chunk.choices[0].finish_reason
if reason == "content_filter":
yield "\n[Innhold filtrert av sikkerhetsfilter]"
return # Ferdig
return # Stromming fullfort
except APIStatusError as e:
attempt += 1
if e.status_code == 429:
retry_after = int(e.response.headers.get("retry-after", "5"))
await asyncio.sleep(retry_after)
elif e.status_code >= 500:
await asyncio.sleep(2 ** attempt) # Eksponentiell backoff
else:
raise # Ikke-gjenforsokbar feil
except APIConnectionError:
attempt += 1
await asyncio.sleep(2 ** attempt)
raise Exception(f"Streaming feilet etter {max_retries} forsok")
```
### Streaming med Partial Response Recovery
```python
async def stream_with_checkpoint(
messages: list,
on_token: callable,
on_complete: callable,
on_error: callable
):
"""Stream med checkpoint for delvis gjenoppretting."""
partial_response = []
last_chunk_time = time.time()
try:
response = await async_client.chat.completions.create(
model="gpt-4o",
messages=messages,
stream=True,
max_tokens=1000
)
async for chunk in response:
current_time = time.time()
# Detekter unormalt lang pause mellom chunks
if current_time - last_chunk_time > 30:
# Mulig hengende forbindelse
break
last_chunk_time = current_time
if chunk.choices and chunk.choices[0].delta.content:
token = chunk.choices[0].delta.content
partial_response.append(token)
await on_token(token)
if chunk.choices and chunk.choices[0].finish_reason == "stop":
await on_complete("".join(partial_response))
return
# Hvis vi nar hit uten "stop", har streamingen avbrultt
if partial_response:
await on_complete(
"".join(partial_response) +
"\n\n[Merk: Respons kan vaere ufullstendig]"
)
except Exception as e:
if partial_response:
await on_error(e, "".join(partial_response))
else:
await on_error(e, None)
```
## Nar bruke streaming vs. non-streaming
| Scenario | Anbefaling | Begrunnelse |
|----------|-----------|-------------|
| Chat-grensesnitt | Streaming | Bedre opplevd responstid |
| Innbyggerportal | Streaming | Visuell tilbakemelding under generering |
| Batch-klassifisering | Non-streaming | Kun sluttresultat er relevant |
| Dokumentanalyse | Non-streaming | Strukturert output, ingen inkrementell visning |
| Saksbehandlingsforslag | Streaming | Lang generering, bruker venter |
| API-integrasjon (maskin-til-maskin) | Non-streaming | Enklere feilhandtering |
| Sanntidsoversetning | Streaming | Lavest opplevd latens |
## Avanserte monstre
### Server-side Streaming Proxy med FastAPI
```python
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncAzureOpenAI
app = FastAPI()
client = AsyncAzureOpenAI(
azure_endpoint="https://your-resource.openai.azure.com/",
api_key="your-api-key",
api_version="2025-03-01-preview"
)
@app.post("/api/chat/stream")
async def chat_stream(request: ChatRequest):
"""Server-side proxy for Azure OpenAI streaming."""
async def generate():
try:
response = await client.chat.completions.create(
model="gpt-4o",
messages=[{"role": "user", "content": request.message}],
stream=True,
max_tokens=1000
)
async for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
data = {"content": chunk.choices[0].delta.content}
yield f"data: {json.dumps(data)}\n\n"
yield "data: [DONE]\n\n"
except Exception as e:
error_data = {"error": str(e)}
yield f"data: {json.dumps(error_data)}\n\n"
return StreamingResponse(
generate(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no" # Deaktiver nginx buffering
}
)
```
### Token-telling under streaming
```python
import tiktoken
async def stream_with_token_counting(messages: list, model: str = "gpt-4o"):
"""Stream med sanntids token-telling for kostnadsovervaking."""
encoding = tiktoken.encoding_for_model(model)
output_tokens = 0
response = await async_client.chat.completions.create(
model=model,
messages=messages,
stream=True,
stream_options={"include_usage": True} # Inkluder bruksdata
)
async for chunk in response:
if chunk.choices and chunk.choices[0].delta.content:
content = chunk.choices[0].delta.content
output_tokens += len(encoding.encode(content))
yield content
# Sjekk usage i siste chunk
if chunk.usage:
actual_tokens = chunk.usage.completion_tokens
cached_tokens = getattr(
chunk.usage.prompt_tokens_details, 'cached_tokens', 0
)
print(f"Faktisk token-bruk: {actual_tokens}")
print(f"Cache-treff: {cached_tokens}")
```
## Ytelsesmal for streaming
| Metrikk | Mal (P95) | Kritisk terskel |
|---------|-----------|-----------------|
| Time to First Token | < 500 ms | > 2000 ms |
| Inter-token latens | < 50 ms | > 200 ms |
| Reconnect-tid | < 2 s | > 10 s |
| Stream completion rate | > 99% | < 95% |
## For Cosmo
- **Streaming er obligatorisk** for alle brukerrettede AI-grensesnitt. Forskjellen i opplevd latens er dramatisk: 200 ms TTFT vs. 3-5 sekunders ventetid for komplett respons.
- **Infrastruktur-konfigurasjon er kritisk:** Hele kjeden (APIM, App Gateway, Front Door) ma ha response buffering deaktivert. En enkelt feilkonfigurert komponent blokkerer all streaming.
- **Feilhandtering i strommer krever eget design:** Implementer alltid reconnect-logikk, partial response recovery, og eksponentiell backoff for 429/5xx-feil.
- **Content filtering pavirker streaming:** `finish_reason: content_filter` kan oppsta midt i en strom. Klient-koden ma handtere dette gracefully med en brukermelding.
- **Token-telling under streaming:** Bruk `stream_options: {"include_usage": true}` for a fa eksakt token-bruk i siste chunk, viktig for kostnadsovervaking.