feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,447 @@
|
|||
# Throughput Optimization Strategies
|
||||
|
||||
**Last updated:** 2026-02
|
||||
**Status:** GA
|
||||
**Category:** Performance & Scalability
|
||||
|
||||
---
|
||||
|
||||
## Introduksjon
|
||||
|
||||
Throughput-optimalisering for Azure OpenAI og Azure AI Services handler om å maksimere antall fullførte forespørsler per sekund innenfor de tildelte kvotene. Azure OpenAI måler throughput i tokens per minutt (TPM) og forespørsler per minutt (RPM), og den reelle throughputen avhenger av en kompleks kombinasjon av input-størrelse, output-størrelse, modelltype og samtidige forespørsler.
|
||||
|
||||
For Standard deployments bestemmer den tildelte kvoten (TPM) en øvre grense for gjennomstrømming, men faktisk throughput kan være lavere på grunn av per-forespørsel latens. For Provisioned Throughput Units (PTU) er kapasiteten dedikert, og throughputen avhenger av workload shape — forholdet mellom input- og output-tokens. Microsofts offisielle benchmarking-verktøy (azure-openai-benchmark) er anbefalt for å måle reell throughput for spesifikke workloads.
|
||||
|
||||
I norsk offentlig sektor, der AI-løsninger ofte betjener tusenvis av saksbehandlere eller borgere samtidig, er throughput-optimalisering direkte knyttet til brukeropplevelse og kostnadseffektivitet. En 2x forbedring i throughput kan bety halverte Azure-kostnader for samme arbeidsmengde.
|
||||
|
||||
## Kjernekomponenter
|
||||
|
||||
| Komponent | Formål | Teknologi |
|
||||
|-----------|--------|-----------|
|
||||
| Token quota (TPM/RPM) | Rate limiting for Standard deployments | Azure OpenAI Quota |
|
||||
| Provisioned Throughput Units | Dedikert kapasitet med garantert throughput | Azure OpenAI PTU |
|
||||
| Batch API | 50% rabatt for asynkrone batch-jobber | Azure OpenAI Global Batch |
|
||||
| Azure Load Testing | Lasttesting og throughput-måling | Azure Load Testing |
|
||||
| Azure Monitor | Throughput-metrikker og overvåking | Azure Monitor |
|
||||
| azure-openai-benchmark | Offisielt benchmarking-verktøy | GitHub CLI tool |
|
||||
|
||||
## Parallel Request Execution
|
||||
|
||||
### Asynkron parallellisering i Python
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import time
|
||||
from openai import AsyncAzureOpenAI
|
||||
from dataclasses import dataclass
|
||||
|
||||
@dataclass
|
||||
class ThroughputResult:
|
||||
total_requests: int
|
||||
successful: int
|
||||
failed: int
|
||||
total_tokens: int
|
||||
duration_seconds: float
|
||||
requests_per_second: float
|
||||
tokens_per_second: float
|
||||
|
||||
async def parallel_completions(
|
||||
client: AsyncAzureOpenAI,
|
||||
messages_batch: list[list[dict]],
|
||||
model: str = "gpt-4o",
|
||||
max_concurrent: int = 20,
|
||||
max_tokens: int = 500
|
||||
) -> ThroughputResult:
|
||||
"""Execute chat completions in parallel with controlled concurrency."""
|
||||
semaphore = asyncio.Semaphore(max_concurrent)
|
||||
results = {"success": 0, "failed": 0, "tokens": 0}
|
||||
|
||||
async def process_one(messages: list[dict]):
|
||||
async with semaphore:
|
||||
try:
|
||||
response = await client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
max_tokens=max_tokens
|
||||
)
|
||||
results["success"] += 1
|
||||
results["tokens"] += response.usage.total_tokens
|
||||
except Exception as e:
|
||||
results["failed"] += 1
|
||||
if hasattr(e, 'status_code') and e.status_code == 429:
|
||||
# Retry-After: vent og prøv igjen
|
||||
retry_after = getattr(e, 'retry_after', 5)
|
||||
await asyncio.sleep(retry_after)
|
||||
await process_one(messages) # Retry
|
||||
|
||||
start = time.time()
|
||||
await asyncio.gather(*[process_one(m) for m in messages_batch])
|
||||
duration = time.time() - start
|
||||
|
||||
return ThroughputResult(
|
||||
total_requests=len(messages_batch),
|
||||
successful=results["success"],
|
||||
failed=results["failed"],
|
||||
total_tokens=results["tokens"],
|
||||
duration_seconds=round(duration, 2),
|
||||
requests_per_second=round(results["success"] / duration, 2),
|
||||
tokens_per_second=round(results["tokens"] / duration, 2)
|
||||
)
|
||||
|
||||
# Eksempel: Prosesser 1000 forespørsler med 20 samtidige
|
||||
async def main():
|
||||
client = AsyncAzureOpenAI(
|
||||
azure_endpoint="https://my-aoai.openai.azure.com",
|
||||
api_key="...",
|
||||
api_version="2024-10-21"
|
||||
)
|
||||
|
||||
batch = [
|
||||
[{"role": "user", "content": f"Oppsummer dokument {i}"}]
|
||||
for i in range(1000)
|
||||
]
|
||||
|
||||
result = await parallel_completions(client, batch, max_concurrent=20)
|
||||
print(f"Throughput: {result.requests_per_second} RPS, "
|
||||
f"{result.tokens_per_second} tokens/s")
|
||||
```
|
||||
|
||||
### .NET Parallel Processing med SemaphoreSlim
|
||||
|
||||
```csharp
|
||||
using Azure.AI.OpenAI;
|
||||
using System.Collections.Concurrent;
|
||||
|
||||
public class ThroughputOptimizer
|
||||
{
|
||||
private readonly AzureOpenAIClient _client;
|
||||
private readonly SemaphoreSlim _semaphore;
|
||||
private readonly ConcurrentBag<RequestMetric> _metrics = new();
|
||||
|
||||
public ThroughputOptimizer(AzureOpenAIClient client, int maxConcurrency = 20)
|
||||
{
|
||||
_client = client;
|
||||
_semaphore = new SemaphoreSlim(maxConcurrency, maxConcurrency);
|
||||
}
|
||||
|
||||
public async Task<ThroughputReport> ProcessBatchAsync(
|
||||
IReadOnlyList<ChatMessage[]> requests,
|
||||
string deploymentName,
|
||||
CancellationToken cancellationToken = default)
|
||||
{
|
||||
var sw = System.Diagnostics.Stopwatch.StartNew();
|
||||
|
||||
var tasks = requests.Select(messages =>
|
||||
ProcessSingleAsync(messages, deploymentName, cancellationToken));
|
||||
|
||||
await Task.WhenAll(tasks);
|
||||
sw.Stop();
|
||||
|
||||
var successful = _metrics.Where(m => m.Success).ToList();
|
||||
return new ThroughputReport
|
||||
{
|
||||
TotalRequests = requests.Count,
|
||||
Successful = successful.Count,
|
||||
Failed = _metrics.Count - successful.Count,
|
||||
TotalTokens = successful.Sum(m => m.TotalTokens),
|
||||
DurationMs = sw.ElapsedMilliseconds,
|
||||
RequestsPerSecond = Math.Round(
|
||||
successful.Count / (sw.ElapsedMilliseconds / 1000.0), 2),
|
||||
TokensPerSecond = Math.Round(
|
||||
successful.Sum(m => m.TotalTokens) /
|
||||
(sw.ElapsedMilliseconds / 1000.0), 2)
|
||||
};
|
||||
}
|
||||
|
||||
private async Task ProcessSingleAsync(
|
||||
ChatMessage[] messages,
|
||||
string deploymentName,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await _semaphore.WaitAsync(ct);
|
||||
try
|
||||
{
|
||||
var chatClient = _client.GetChatClient(deploymentName);
|
||||
var response = await chatClient.CompleteChatAsync(messages);
|
||||
|
||||
_metrics.Add(new RequestMetric
|
||||
{
|
||||
Success = true,
|
||||
TotalTokens = response.Value.Usage.TotalTokenCount
|
||||
});
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
_metrics.Add(new RequestMetric { Success = false });
|
||||
}
|
||||
finally
|
||||
{
|
||||
_semaphore.Release();
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Request Buffering Strategies
|
||||
|
||||
### Mikro-batching for høy throughput
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from collections import deque
|
||||
from typing import Callable, Any
|
||||
|
||||
class RequestBuffer:
|
||||
"""Buffer requests and flush in micro-batches for throughput optimization."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
process_fn: Callable,
|
||||
max_batch_size: int = 10,
|
||||
flush_interval_ms: int = 100,
|
||||
max_queue_size: int = 1000
|
||||
):
|
||||
self.process_fn = process_fn
|
||||
self.max_batch_size = max_batch_size
|
||||
self.flush_interval = flush_interval_ms / 1000
|
||||
self.queue: deque = deque(maxlen=max_queue_size)
|
||||
self._running = False
|
||||
|
||||
async def enqueue(self, request: dict) -> asyncio.Future:
|
||||
"""Add request to buffer, returns future with result."""
|
||||
future = asyncio.get_event_loop().create_future()
|
||||
self.queue.append({"request": request, "future": future})
|
||||
|
||||
if len(self.queue) >= self.max_batch_size:
|
||||
await self._flush()
|
||||
|
||||
return await future
|
||||
|
||||
async def _flush(self):
|
||||
"""Process all buffered requests."""
|
||||
batch = []
|
||||
futures = []
|
||||
|
||||
while self.queue and len(batch) < self.max_batch_size:
|
||||
item = self.queue.popleft()
|
||||
batch.append(item["request"])
|
||||
futures.append(item["future"])
|
||||
|
||||
if batch:
|
||||
try:
|
||||
results = await self.process_fn(batch)
|
||||
for future, result in zip(futures, results):
|
||||
future.set_result(result)
|
||||
except Exception as e:
|
||||
for future in futures:
|
||||
if not future.done():
|
||||
future.set_exception(e)
|
||||
|
||||
async def run(self):
|
||||
"""Run flush loop."""
|
||||
self._running = True
|
||||
while self._running:
|
||||
if self.queue:
|
||||
await self._flush()
|
||||
await asyncio.sleep(self.flush_interval)
|
||||
```
|
||||
|
||||
## Queue Depth Tuning
|
||||
|
||||
### Optimal kø-dybde for Azure OpenAI
|
||||
|
||||
```python
|
||||
import math
|
||||
|
||||
def calculate_optimal_queue_depth(
|
||||
tpm_quota: int,
|
||||
avg_input_tokens: int,
|
||||
avg_output_tokens: int,
|
||||
avg_latency_ms: float,
|
||||
target_utilization: float = 0.85
|
||||
) -> dict:
|
||||
"""Calculate optimal queue depth based on quota and latency."""
|
||||
|
||||
# Beregn maks concurrent requests basert på quota
|
||||
total_tokens_per_request = avg_input_tokens + avg_output_tokens
|
||||
max_rpm = tpm_quota / total_tokens_per_request
|
||||
|
||||
# Maks concurrent basert på latens
|
||||
requests_per_second = max_rpm / 60
|
||||
avg_latency_s = avg_latency_ms / 1000
|
||||
|
||||
# Little's Law: L = λ * W
|
||||
# L = concurrent requests, λ = arrival rate, W = service time
|
||||
optimal_concurrent = requests_per_second * avg_latency_s
|
||||
|
||||
# Queue depth = concurrent * buffer factor
|
||||
queue_depth = math.ceil(optimal_concurrent * (1 / target_utilization))
|
||||
|
||||
return {
|
||||
"max_rpm": round(max_rpm),
|
||||
"max_rps": round(requests_per_second, 2),
|
||||
"optimal_concurrent": math.ceil(optimal_concurrent),
|
||||
"recommended_queue_depth": queue_depth,
|
||||
"theoretical_max_tps": round(
|
||||
tpm_quota / 60 / total_tokens_per_request *
|
||||
total_tokens_per_request, 0
|
||||
)
|
||||
}
|
||||
|
||||
# Eksempel: 240K TPM quota, typisk RAG-workload
|
||||
result = calculate_optimal_queue_depth(
|
||||
tpm_quota=240_000,
|
||||
avg_input_tokens=2000,
|
||||
avg_output_tokens=500,
|
||||
avg_latency_ms=1200
|
||||
)
|
||||
print(result)
|
||||
# {'max_rpm': 96, 'max_rps': 1.6, 'optimal_concurrent': 2,
|
||||
# 'recommended_queue_depth': 3, ...}
|
||||
```
|
||||
|
||||
## System Bottleneck Identification
|
||||
|
||||
### Identifisering av flaskehalser med Azure Monitor
|
||||
|
||||
```python
|
||||
# KQL-spørringer for throughput-analyse
|
||||
|
||||
# 1. Token throughput per minutt
|
||||
PROCESSED_TOKENS_QUERY = """
|
||||
AzureDiagnostics
|
||||
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
|
||||
| where Category == "RequestResponse"
|
||||
| extend promptTokens = toint(properties_s.promptTokens)
|
||||
| extend completionTokens = toint(properties_s.completionTokens)
|
||||
| summarize
|
||||
TotalPromptTPM = sum(promptTokens),
|
||||
TotalCompletionTPM = sum(completionTokens),
|
||||
TotalTPM = sum(promptTokens) + sum(completionTokens),
|
||||
RequestCount = count()
|
||||
by bin(TimeGenerated, 1m), deploymentName_s
|
||||
| order by TimeGenerated desc
|
||||
"""
|
||||
|
||||
# 2. Identifiser throttling-mønstre
|
||||
THROTTLING_ANALYSIS = """
|
||||
AzureMetrics
|
||||
| where MetricName == "AzureOpenAIRequests"
|
||||
| extend StatusCode = tostring(split(DimensionValue, ",")[0])
|
||||
| summarize
|
||||
Total = count(),
|
||||
Throttled = countif(StatusCode == "429"),
|
||||
ServerErrors = countif(StatusCode startswith "5"),
|
||||
ThrottleRate = round(
|
||||
countif(StatusCode == "429") * 100.0 / count(), 2)
|
||||
by bin(TimeGenerated, 5m)
|
||||
| where ThrottleRate > 0
|
||||
| order by TimeGenerated desc
|
||||
"""
|
||||
|
||||
# 3. Latens-distribusjon for å finne bottlenecks
|
||||
LATENCY_PERCENTILES = """
|
||||
AzureDiagnostics
|
||||
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
|
||||
| extend DurationMs = todouble(DurationMs)
|
||||
| summarize
|
||||
P50 = percentile(DurationMs, 50),
|
||||
P90 = percentile(DurationMs, 90),
|
||||
P95 = percentile(DurationMs, 95),
|
||||
P99 = percentile(DurationMs, 99),
|
||||
Avg = avg(DurationMs)
|
||||
by bin(TimeGenerated, 5m), deploymentName_s
|
||||
| order by TimeGenerated desc
|
||||
"""
|
||||
```
|
||||
|
||||
### Bottleneck Decision Tree
|
||||
|
||||
```
|
||||
Lav throughput?
|
||||
├── Høy throttle rate (>5% 429s)?
|
||||
│ ├── Ja → Øk TPM-kvote eller legg til regioner
|
||||
│ └── Nei → Sjekk latens
|
||||
├── Høy latens (P95 > 5s)?
|
||||
│ ├── Input tokens > 4K? → Reduser prompt-størrelse
|
||||
│ ├── Output tokens > 2K? → Reduser max_tokens
|
||||
│ └── Lav token count? → Sjekk nettverkslatens
|
||||
├── Lav concurrent requests?
|
||||
│ ├── Klient-side bottleneck → Øk parallellisering
|
||||
│ └── Connection pool for liten → Øk pool size
|
||||
└── Utilization < 50%?
|
||||
└── Under-provisjonert? → Sjekk quota allocation
|
||||
```
|
||||
|
||||
## Implementeringsmønstre
|
||||
|
||||
### Batch API for ikke-tidskritisk prosessering
|
||||
|
||||
```python
|
||||
from openai import AzureOpenAI
|
||||
import json
|
||||
|
||||
def create_batch_file(requests: list[dict], filename: str = "batch.jsonl"):
|
||||
"""Create JSONL file for Azure OpenAI Batch API."""
|
||||
with open(filename, "w") as f:
|
||||
for i, req in enumerate(requests):
|
||||
batch_request = {
|
||||
"custom_id": f"request-{i}",
|
||||
"method": "POST",
|
||||
"url": "/chat/completions",
|
||||
"body": {
|
||||
"model": "gpt-4o", # Must match deployment name
|
||||
"messages": req["messages"],
|
||||
"max_tokens": req.get("max_tokens", 1000)
|
||||
}
|
||||
}
|
||||
f.write(json.dumps(batch_request) + "\n")
|
||||
|
||||
def submit_batch(client: AzureOpenAI, filename: str):
|
||||
"""Submit batch job — 50% cost reduction, 24hr turnaround."""
|
||||
# Upload file
|
||||
batch_file = client.files.create(
|
||||
file=open(filename, "rb"),
|
||||
purpose="batch"
|
||||
)
|
||||
|
||||
# Create batch job
|
||||
batch_job = client.batches.create(
|
||||
input_file_id=batch_file.id,
|
||||
endpoint="/chat/completions",
|
||||
completion_window="24h"
|
||||
)
|
||||
return batch_job
|
||||
```
|
||||
|
||||
## Norsk offentlig sektor
|
||||
|
||||
- **Kostnadseffektivitet**: Bruk Batch API for alle ikke-sanntids workloads (dokumentanalyse, klassifisering, oppsummering) for å oppnå 50% kostnadsreduksjon. Dette er spesielt relevant for store etater med høyt dokumentvolum.
|
||||
- **Kapasitetsplanlegging**: Start med å estimere TPM-behov basert på forventet brukermønster (antall saksbehandlere * forespørsler per time * tokens per forespørsel). Bestill PTU for forutsigbare workloads.
|
||||
- **SLA-krav**: Provisioned throughput gir forutsigbar ytelse med latens-SLA (99% > N tokens/sekund per PTU). Standard deployments har ingen latens-SLA.
|
||||
- **Data residency**: Global Batch behandler data i Azure OpenAI-lokasjoner globalt — bruk Data Zone Batch for å holde data innenfor EU/EØS.
|
||||
|
||||
## Beslutningsrammeverk
|
||||
|
||||
| Scenario | Anbefaling | Begrunnelse |
|
||||
|----------|------------|-------------|
|
||||
| Sanntids chat (<2s respons) | Standard/PTU + streaming | Lavest brukervendt latens |
|
||||
| Dokumentprosessering (1000+ docs) | Batch API | 50% kostnadsreduksjon, 24h turnaround |
|
||||
| Forutsigbar høy trafikk | Provisioned Throughput (PTU) | Garantert kapasitet og latens |
|
||||
| Variable workloads | Standard + auto-scale quota | Betal per bruk, fleksibel skalering |
|
||||
| Multi-model pipeline | Parallell execution + queue | Maksimer samlet throughput |
|
||||
|
||||
## Referanser
|
||||
|
||||
- [Performance and latency](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/latency) — Azure OpenAI latency og throughput
|
||||
- [Azure OpenAI Batch API](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/batch) — Batch processing guide
|
||||
- [Provisioned throughput onboarding](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding) — PTU sizing og kostnader
|
||||
- [Azure OpenAI Benchmark Tool](https://github.com/Azure/azure-openai-benchmark) — Offisielt benchmarking-verktøy
|
||||
|
||||
## For Cosmo
|
||||
|
||||
- **Bruk denne referansen** når kunden trenger å maksimere throughput for AI-workloads, eller når de opplever at de ikke utnytter sin tildelte kvote effektivt.
|
||||
- Batch API gir 50% kostnadsreduksjon og bør anbefales for alle ikke-sanntids workloads — mange kunder er ikke klar over denne muligheten.
|
||||
- Bruk Little's Law (L = lambda * W) for å beregne optimal concurrent requests: quota bestemmer lambda, latens bestemmer W.
|
||||
- Alltid benchmark med reelle workloads — den offisielle azure-openai-benchmark-verktøyet gir pålitelige tall for PTU-sizing.
|
||||
- For norsk offentlig sektor: anbefal Data Zone deployments for Batch API for å holde data innenfor EU/EØS.
|
||||
Loading…
Add table
Add a link
Reference in a new issue