feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,478 @@
|
|||
# Response Chunking Strategies
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Status:** GA
|
||||
**Category:** Performance & Scalability
|
||||
|
||||
---
|
||||
|
||||
## Introduksjon
|
||||
|
||||
Response chunking handler om hvordan store AI-modellresponser fra Azure OpenAI brytes opp og leveres til klienter. Det finnes to hovedtilnærminger: streaming via Server-Sent Events (SSE) der modellens output leveres token-for-token i sanntid, og chunking av store responser der output deles opp i semantisk meningsfulle blokker for videre prosessering.
|
||||
|
||||
Streaming er den mest brukte chunking-strategien for Azure OpenAI. Når `stream: true` settes i API-kallet, returnerer tjenesten delta-oppdateringer som Server-Sent Events ettersom tokens genereres. Dette gir brukeren umiddelbar feedback (time-to-first-token typisk 200-500ms) i stedet for å vente på hele responsen (som kan ta 5-30 sekunder for lange output). For programmatisk prosessering der hele responsen trengs, er chunking av det endelige resultatet i semantisk koherente blokker viktig for downstream-systemer.
|
||||
|
||||
For norsk offentlig sektor der AI brukes til å generere lange dokumenter (saksframlegg, utredninger, rapporter), er response chunking avgjørende for å levere god brukeropplevelse og for å kunne prosessere store responser effektivt i saksbehandlingssystemer.
|
||||
|
||||
## Kjernekomponenter
|
||||
|
||||
| Komponent | Formål | Teknologi |
|
||||
|-----------|--------|-----------|
|
||||
| Server-Sent Events (SSE) | Real-time streaming av tokens | HTTP SSE |
|
||||
| stream_options | Konfigurer streaming-oppførsel | Azure OpenAI API |
|
||||
| Application Gateway | SSE proxy og load balancing | Azure App Gateway |
|
||||
| API Management | SSE-støtte med policy-basert routing | Azure APIM |
|
||||
| SignalR | Real-time push til web-klienter | Azure SignalR |
|
||||
|
||||
## Streaming med Server-Sent Events
|
||||
|
||||
### Python streaming-implementasjon
|
||||
|
||||
```python
|
||||
from openai import AzureOpenAI
|
||||
import sys
|
||||
|
||||
client = AzureOpenAI(
|
||||
azure_endpoint="https://my-aoai.openai.azure.com",
|
||||
api_key="...",
|
||||
api_version="2024-10-21"
|
||||
)
|
||||
|
||||
def stream_chat_completion(messages: list[dict], model: str = "gpt-4o"):
|
||||
"""Stream response with real-time token delivery."""
|
||||
collected_content = []
|
||||
|
||||
stream = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
stream=True,
|
||||
stream_options={"include_usage": True}, # Få token-bruk til slutt
|
||||
max_tokens=2000
|
||||
)
|
||||
|
||||
for chunk in stream:
|
||||
if chunk.choices and chunk.choices[0].delta.content:
|
||||
token = chunk.choices[0].delta.content
|
||||
collected_content.append(token)
|
||||
sys.stdout.write(token)
|
||||
sys.stdout.flush()
|
||||
|
||||
# Siste chunk inneholder usage
|
||||
if hasattr(chunk, 'usage') and chunk.usage:
|
||||
return {
|
||||
"content": "".join(collected_content),
|
||||
"prompt_tokens": chunk.usage.prompt_tokens,
|
||||
"completion_tokens": chunk.usage.completion_tokens,
|
||||
"total_tokens": chunk.usage.total_tokens
|
||||
}
|
||||
|
||||
return {"content": "".join(collected_content)}
|
||||
|
||||
|
||||
# Asynkron streaming
|
||||
async def async_stream_completion(
|
||||
client: AsyncAzureOpenAI,
|
||||
messages: list[dict],
|
||||
model: str = "gpt-4o",
|
||||
on_token: callable = None
|
||||
):
|
||||
"""Async stream with callback per token."""
|
||||
chunks = []
|
||||
|
||||
async with client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
stream=True,
|
||||
stream_options={"include_usage": True}
|
||||
) as stream:
|
||||
async for chunk in stream:
|
||||
if chunk.choices and chunk.choices[0].delta.content:
|
||||
token = chunk.choices[0].delta.content
|
||||
chunks.append(token)
|
||||
if on_token:
|
||||
await on_token(token)
|
||||
|
||||
return "".join(chunks)
|
||||
```
|
||||
|
||||
### .NET streaming med IAsyncEnumerable
|
||||
|
||||
```csharp
|
||||
using Azure.AI.OpenAI;
|
||||
using OpenAI.Chat;
|
||||
|
||||
public class StreamingService
|
||||
{
|
||||
private readonly AzureOpenAIClient _client;
|
||||
|
||||
public async IAsyncEnumerable<string> StreamCompletionAsync(
|
||||
string deploymentName,
|
||||
IList<ChatMessage> messages,
|
||||
int maxTokens = 2000)
|
||||
{
|
||||
var chatClient = _client.GetChatClient(deploymentName);
|
||||
|
||||
var options = new ChatCompletionOptions
|
||||
{
|
||||
MaxOutputTokenCount = maxTokens
|
||||
};
|
||||
|
||||
// Stream deltas
|
||||
await foreach (var update in
|
||||
chatClient.CompleteChatStreamingAsync(messages, options))
|
||||
{
|
||||
foreach (var part in update.ContentUpdate)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(part.Text))
|
||||
{
|
||||
yield return part.Text;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Bruk i ASP.NET controller
|
||||
public async Task StreamToClient(
|
||||
HttpContext context,
|
||||
string deploymentName,
|
||||
IList<ChatMessage> messages)
|
||||
{
|
||||
context.Response.ContentType = "text/event-stream";
|
||||
context.Response.Headers.Append("Cache-Control", "no-cache");
|
||||
context.Response.Headers.Append("Connection", "keep-alive");
|
||||
|
||||
var writer = new StreamWriter(context.Response.Body);
|
||||
|
||||
await foreach (var token in StreamCompletionAsync(
|
||||
deploymentName, messages))
|
||||
{
|
||||
await writer.WriteAsync($"data: {token}\n\n");
|
||||
await writer.FlushAsync();
|
||||
}
|
||||
|
||||
await writer.WriteAsync("data: [DONE]\n\n");
|
||||
await writer.FlushAsync();
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Semantic Chunking Approaches
|
||||
|
||||
### Chunk store responser i meningsfulle blokker
|
||||
|
||||
```python
|
||||
import re
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class SemanticChunk:
|
||||
index: int
|
||||
content: str
|
||||
chunk_type: str # "heading", "paragraph", "code", "list", "table"
|
||||
token_count: int
|
||||
|
||||
def semantic_chunk_response(
|
||||
response_text: str,
|
||||
max_chunk_tokens: int = 500,
|
||||
model: str = "gpt-4o"
|
||||
) -> list[SemanticChunk]:
|
||||
"""Split AI response into semantically coherent chunks."""
|
||||
import tiktoken
|
||||
enc = tiktoken.encoding_for_model(model)
|
||||
|
||||
chunks = []
|
||||
current_chunk = []
|
||||
current_tokens = 0
|
||||
chunk_type = "paragraph"
|
||||
|
||||
# Del på naturlige grenser
|
||||
lines = response_text.split('\n')
|
||||
|
||||
for line in lines:
|
||||
line_tokens = len(enc.encode(line))
|
||||
|
||||
# Identifiser chunk-type
|
||||
if line.startswith('#'):
|
||||
chunk_type = "heading"
|
||||
elif line.startswith('```'):
|
||||
chunk_type = "code"
|
||||
elif line.startswith('- ') or line.startswith('* '):
|
||||
chunk_type = "list"
|
||||
elif line.startswith('|'):
|
||||
chunk_type = "table"
|
||||
else:
|
||||
chunk_type = "paragraph"
|
||||
|
||||
# Ny chunk ved heading eller ved token-grense
|
||||
if (line.startswith('#') and current_chunk) or \
|
||||
(current_tokens + line_tokens > max_chunk_tokens and current_chunk):
|
||||
chunks.append(SemanticChunk(
|
||||
index=len(chunks),
|
||||
content='\n'.join(current_chunk),
|
||||
chunk_type=chunk_type,
|
||||
token_count=current_tokens
|
||||
))
|
||||
current_chunk = []
|
||||
current_tokens = 0
|
||||
|
||||
current_chunk.append(line)
|
||||
current_tokens += line_tokens
|
||||
|
||||
# Siste chunk
|
||||
if current_chunk:
|
||||
chunks.append(SemanticChunk(
|
||||
index=len(chunks),
|
||||
content='\n'.join(current_chunk),
|
||||
chunk_type=chunk_type,
|
||||
token_count=current_tokens
|
||||
))
|
||||
|
||||
return chunks
|
||||
```
|
||||
|
||||
### Streaming accumulator med chunk-deteksjon
|
||||
|
||||
```python
|
||||
class StreamingChunkAccumulator:
|
||||
"""Accumulate streaming tokens into semantic chunks."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
on_chunk_complete: callable = None,
|
||||
chunk_boundary_pattern: str = r'\n#{1,3}\s'
|
||||
):
|
||||
self.buffer = []
|
||||
self.chunks = []
|
||||
self.on_chunk_complete = on_chunk_complete
|
||||
self.boundary_pattern = re.compile(chunk_boundary_pattern)
|
||||
|
||||
async def feed_token(self, token: str):
|
||||
"""Feed a streaming token to the accumulator."""
|
||||
self.buffer.append(token)
|
||||
|
||||
# Sjekk om vi har nådd en chunk-grense
|
||||
current_text = ''.join(self.buffer)
|
||||
if self.boundary_pattern.search(current_text):
|
||||
# Del på grensen
|
||||
parts = self.boundary_pattern.split(current_text, maxsplit=1)
|
||||
if len(parts) > 1:
|
||||
completed = parts[0]
|
||||
remaining = current_text[len(completed):]
|
||||
|
||||
if completed.strip():
|
||||
chunk = SemanticChunk(
|
||||
index=len(self.chunks),
|
||||
content=completed.strip(),
|
||||
chunk_type=self._detect_type(completed),
|
||||
token_count=len(completed.split()) # Estimat
|
||||
)
|
||||
self.chunks.append(chunk)
|
||||
|
||||
if self.on_chunk_complete:
|
||||
await self.on_chunk_complete(chunk)
|
||||
|
||||
self.buffer = [remaining]
|
||||
|
||||
def finalize(self) -> list[SemanticChunk]:
|
||||
"""Finalize and return all chunks."""
|
||||
remaining = ''.join(self.buffer).strip()
|
||||
if remaining:
|
||||
self.chunks.append(SemanticChunk(
|
||||
index=len(self.chunks),
|
||||
content=remaining,
|
||||
chunk_type=self._detect_type(remaining),
|
||||
token_count=len(remaining.split())
|
||||
))
|
||||
return self.chunks
|
||||
|
||||
def _detect_type(self, text: str) -> str:
|
||||
if text.startswith('```'):
|
||||
return "code"
|
||||
if text.startswith('#'):
|
||||
return "heading"
|
||||
if text.startswith('- ') or text.startswith('* '):
|
||||
return "list"
|
||||
return "paragraph"
|
||||
```
|
||||
|
||||
## Client-Side Reassembly
|
||||
|
||||
### Web-klient med progressiv rendering
|
||||
|
||||
```typescript
|
||||
// TypeScript: Client-side SSE consumption with chunk assembly
|
||||
interface StreamChunk {
|
||||
content: string;
|
||||
isComplete: boolean;
|
||||
tokenCount: number;
|
||||
}
|
||||
|
||||
class AIResponseAssembler {
|
||||
private chunks: string[] = [];
|
||||
private onUpdate: (text: string) => void;
|
||||
private onComplete: (text: string, stats: object) => void;
|
||||
|
||||
constructor(
|
||||
onUpdate: (text: string) => void,
|
||||
onComplete: (text: string, stats: object) => void
|
||||
) {
|
||||
this.onUpdate = onUpdate;
|
||||
this.onComplete = onComplete;
|
||||
}
|
||||
|
||||
async streamFromEndpoint(url: string, body: object): Promise<void> {
|
||||
const response = await fetch(url, {
|
||||
method: 'POST',
|
||||
headers: { 'Content-Type': 'application/json' },
|
||||
body: JSON.stringify({ ...body, stream: true }),
|
||||
});
|
||||
|
||||
if (!response.body) throw new Error('No response body');
|
||||
|
||||
const reader = response.body
|
||||
.pipeThrough(new TextDecoderStream())
|
||||
.getReader();
|
||||
|
||||
let fullText = '';
|
||||
let buffer = '';
|
||||
|
||||
while (true) {
|
||||
const { done, value } = await reader.read();
|
||||
if (done) break;
|
||||
|
||||
buffer += value;
|
||||
const lines = buffer.split('\n');
|
||||
buffer = lines.pop() || '';
|
||||
|
||||
for (const line of lines) {
|
||||
if (line.startsWith('data: ')) {
|
||||
const data = line.slice(6);
|
||||
if (data === '[DONE]') {
|
||||
this.onComplete(fullText, {
|
||||
totalChunks: this.chunks.length,
|
||||
totalLength: fullText.length
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const parsed = JSON.parse(data);
|
||||
const token = parsed.choices?.[0]?.delta?.content || '';
|
||||
if (token) {
|
||||
fullText += token;
|
||||
this.chunks.push(token);
|
||||
this.onUpdate(fullText);
|
||||
}
|
||||
} catch { /* skip malformed */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling in Chunks
|
||||
|
||||
### Robust feilhåndtering for streaming
|
||||
|
||||
```python
|
||||
class ResilientStreamProcessor:
|
||||
"""Handle errors during streaming response."""
|
||||
|
||||
def __init__(self, client: AsyncAzureOpenAI, max_retries: int = 3):
|
||||
self.client = client
|
||||
self.max_retries = max_retries
|
||||
|
||||
async def stream_with_recovery(
|
||||
self,
|
||||
messages: list[dict],
|
||||
model: str = "gpt-4o",
|
||||
max_tokens: int = 2000
|
||||
) -> dict:
|
||||
"""Stream with automatic recovery on failure."""
|
||||
accumulated = []
|
||||
total_tokens_generated = 0
|
||||
|
||||
for attempt in range(self.max_retries):
|
||||
try:
|
||||
stream = await self.client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
stream=True,
|
||||
max_tokens=max_tokens - total_tokens_generated
|
||||
)
|
||||
|
||||
async for chunk in stream:
|
||||
if chunk.choices and chunk.choices[0].delta.content:
|
||||
token = chunk.choices[0].delta.content
|
||||
accumulated.append(token)
|
||||
total_tokens_generated += 1
|
||||
|
||||
# Sjekk for finish_reason
|
||||
if chunk.choices and chunk.choices[0].finish_reason:
|
||||
return {
|
||||
"content": "".join(accumulated),
|
||||
"finish_reason": chunk.choices[0].finish_reason,
|
||||
"attempts": attempt + 1,
|
||||
"recovered": attempt > 0
|
||||
}
|
||||
|
||||
# Stream fullført uten finish_reason
|
||||
return {
|
||||
"content": "".join(accumulated),
|
||||
"finish_reason": "stop",
|
||||
"attempts": attempt + 1,
|
||||
"recovered": attempt > 0
|
||||
}
|
||||
|
||||
except Exception as e:
|
||||
if attempt < self.max_retries - 1:
|
||||
# Fortsett fra der vi stoppet
|
||||
partial = "".join(accumulated)
|
||||
if partial:
|
||||
# Legg til partial output som assistant-melding
|
||||
messages = messages + [
|
||||
{"role": "assistant", "content": partial},
|
||||
{"role": "user", "content": "Fortsett fra der du stoppet."}
|
||||
]
|
||||
await asyncio.sleep(2 ** attempt)
|
||||
else:
|
||||
return {
|
||||
"content": "".join(accumulated),
|
||||
"finish_reason": "error",
|
||||
"error": str(e),
|
||||
"attempts": attempt + 1
|
||||
}
|
||||
```
|
||||
|
||||
## Norsk offentlig sektor
|
||||
|
||||
- **Universell utforming**: Streaming gir bedre brukeropplevelse for skjermlesere og sakte nettverk — bruker ser innhold progressivt i stedet for å vente.
|
||||
- **Saksbehandlingssystemer**: Chunk store AI-responser i semantiske blokker (overskrifter, avsnitt, tabeller) for enkel integrasjon i saksbehandlingsdokumenter.
|
||||
- **Logging og audit**: Ved streaming, logg den komplette responsen etter fullføring for arkiverings- og revisjonskrav.
|
||||
- **Application Gateway**: Konfigurer response buffer disabled for SSE-støtte gjennom Azure Application Gateway eller API Management.
|
||||
|
||||
## Beslutningsrammeverk
|
||||
|
||||
| Scenario | Anbefaling | Begrunnelse |
|
||||
|----------|------------|-------------|
|
||||
| Interaktiv chat UI | SSE streaming | Umiddelbar bruker-feedback |
|
||||
| Batch dokumentprosessering | Ikke-streaming + semantic chunking | Enklere feilhåndtering |
|
||||
| API-til-API integrasjon | Ikke-streaming | Enklere å parse komplett respons |
|
||||
| Lang respons (>2000 tokens) | Streaming + chunk accumulator | Reduser opplevd ventetid |
|
||||
| Kritisk pålitelighet | Streaming med recovery | Gjenoppta ved feil |
|
||||
|
||||
## Referanser
|
||||
|
||||
- [Azure OpenAI streaming](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/responses) — Streaming API
|
||||
- [Server-Sent Events with Application Gateway](https://learn.microsoft.com/azure/application-gateway/use-server-sent-events) — SSE proxy
|
||||
- [API Management SSE configuration](https://learn.microsoft.com/azure/api-management/how-to-server-sent-events) — APIM SSE
|
||||
- [Server-Sent Events with App Gateway for Containers](https://learn.microsoft.com/azure/application-gateway/for-containers/server-sent-events) — Container SSE
|
||||
|
||||
## For Cosmo
|
||||
|
||||
- **Bruk denne referansen** når kunden implementerer streaming i AI-applikasjoner, trenger å chunke store responser, eller har feilhåndteringsproblemer med SSE.
|
||||
- Streaming er alltid anbefalt for brukervendte applikasjoner — time-to-first-token reduseres fra sekunder til millisekunder.
|
||||
- Konfigurer `stream_options: { include_usage: true }` for å få token-bruk i siste chunk — uten dette mangler kostnadssporing.
|
||||
- Ved bruk av Application Gateway eller API Management som proxy: deaktiver response buffering for SSE-kompatibilitet.
|
||||
- Implementer alltid recovery-logikk for streaming — nettverksavbrudd er uunngåelig i produksjon, og delvis generert output bør gjenbrukes.
|
||||
Loading…
Add table
Add a link
Reference in a new issue