open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command

Browse source 
Add /ultraresearch-local for structured research combining local codebase
analysis with external knowledge via parallel agent swarms. Produces research
briefs with triangulation, confidence ratings, and source quality assessment.

New command: /ultraresearch-local with modes --quick, --local, --external, --fg.
New agents: research-orchestrator (opus), docs-researcher, community-researcher,
security-researcher, contrarian-researcher, gemini-bridge (all sonnet).
New template: research-brief-template.md.

Integration: --research flag in /ultraplan-local accepts pre-built research
briefs (up to 3), enriches the interview and exploration phases. Planning
orchestrator cross-references brief findings during synthesis.

Design principle: Context Engineering — right information to right agent at
right time. Research briefs are structured artifacts in the pipeline:
ultraresearch → brief → ultraplan --research → plan → ultraexecute.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-04-08 08:58:35 +02:00
commit baa2d0220b
488 changed files with 213221 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

343
skills/ms-ai-security/references/performance-scalability/token-per-second-optimization.md Normal file
View file

@ -0,0 +1,343 @@
# Token-Per-Second Optimization
**Last updated:** 2026-02
**Status:** GA
**Category:** Performance & Scalability
---
## Introduksjon
Token-per-second (TPS) er en kritisk ytelsesmetrikk for Azure OpenAI-deployments som måler hvor raskt modellen genererer output-tokens. Denne metrikken påvirker direkte brukeropplevelsen ved streaming og den totale gjennomstrømmingen for batch-workloads. Azure OpenAI tilbyr latens-mål per PTU som varierer fra 25 TPS (o1) til 100 TPS (gpt-4.1-nano), og optimalisering av TPS er nøkkelen til å utnytte tildelt kapasitet effektivt.
TPS avhenger av flere faktorer: modellstørrelse, prompt-lengde (antall input-tokens), requested output length (max_tokens), samtidige forespørsler, og om prompt caching er aktiv. For Provisioned Throughput (PTU) deployments er TPS direkte koblet til utilization — når utilization nærmer seg 100%, begynner nye forespørsler å få 429-feil. For Standard deployments er TPS begrenset av den tildelte TPM-kvoten.
For norsk offentlig sektor, der AI-assistenter brukes av saksbehandlere i sanntid, er TPS-optimalisering direkte knyttet til arbeidseffektivitet. Forskjellen mellom 25 TPS og 80 TPS betyr at en 400-tokens respons leveres på 16 sekunder vs. 5 sekunder.
## Kjernekomponenter
| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| Provisioned Throughput (PTU) | Dedikert kapasitet med TPS-garantier | Azure OpenAI PTU |
| Prompt Caching | Reduser input-prosessering for bedre TPS | Azure OpenAI Caching |
| Predicted Outputs | Spekulative output for raskere generering | Azure OpenAI Preview |
| Azure Monitor | TPS- og utilization-metrikker | Azure Monitor |
| Capacity Calculator | PTU-estimering basert på workload | Azure AI Foundry |
## Batch Sizing Impact
### Hvordan batch-størrelse påvirker TPS
```python
# Demonstrer forholdet mellom concurrent requests og throughput
import asyncio
import time
from openai import AsyncAzureOpenAI
async def measure_tps_at_concurrency(
client: AsyncAzureOpenAI,
model: str,
concurrency: int,
num_requests: int = 50,
max_tokens: int = 200
) -> dict:
"""Measure tokens per second at different concurrency levels."""
semaphore = asyncio.Semaphore(concurrency)
total_output_tokens = 0
successful = 0
async def single_request():
nonlocal total_output_tokens, successful
async with semaphore:
try:
response = await client.chat.completions.create(
model=model,
messages=[{"role": "user",
"content": "Skriv en kort forklaring av KI."}],
max_tokens=max_tokens
)
total_output_tokens += response.usage.completion_tokens
successful += 1
except Exception:
pass
start = time.time()
await asyncio.gather(*[single_request() for _ in range(num_requests)])
duration = time.time() - start
return {
"concurrency": concurrency,
"total_output_tokens": total_output_tokens,
"duration_seconds": round(duration, 2),
"aggregate_tps": round(total_output_tokens / duration, 1),
"per_request_tps": round(
total_output_tokens / max(successful, 1) /
(duration / max(successful, 1)), 1),
"successful": successful
}
# Kjør test ved ulike concurrency-nivåer
async def find_optimal_concurrency(client, model):
results = []
for concurrency in [1, 5, 10, 20, 50]:
result = await measure_tps_at_concurrency(
client, model, concurrency)
results.append(result)
print(f"Concurrency {concurrency}: "
f"{result['aggregate_tps']} aggregate TPS")
return results
# Typisk resultat for PTU deployment:
# Concurrency 1: ~40 TPS (per-request max)
# Concurrency 5: ~180 TPS (aggregate)
# Concurrency 10: ~320 TPS (aggregate, nær optimal)
# Concurrency 20: ~350 TPS (aggregate, diminishing returns)
# Concurrency 50: ~350 TPS (aggregate, 429s starter)
```
## Prompt Length Optimization
### Reduser input-tokens for bedre TPS
```python
def optimize_prompt_for_tps(
system_prompt: str,
user_input: str,
max_system_tokens: int = 500,
context_budget: int = 4000
) -> dict:
"""Optimize prompt length to improve TPS."""
import tiktoken
enc = tiktoken.encoding_for_model("gpt-4o")
original_system_tokens = len(enc.encode(system_prompt))
original_user_tokens = len(enc.encode(user_input))
original_total = original_system_tokens + original_user_tokens
optimizations = []
# 1. Kompakt system prompt
if original_system_tokens > max_system_tokens:
optimizations.append(
f"System prompt er {original_system_tokens} tokens "
f"(mål: {max_system_tokens}). Vurder å flytte eksempler "
f"til fine-tuning.")
# 2. Trunkér brukerinput til kontekstbudsjett
if original_user_tokens > context_budget:
optimizations.append(
f"User input er {original_user_tokens} tokens "
f"(budsjett: {context_budget}). Bruk chunking eller "
f"pre-summarization.")
# 3. Fjern redundans
# Identifiser gjentatte seksjoner i prompt
lines = system_prompt.split('\n')
unique_lines = list(dict.fromkeys(lines)) # Behold rekkefølge
if len(unique_lines) < len(lines):
optimizations.append(
f"Fjern {len(lines) - len(unique_lines)} dupliserte linjer "
f"i system prompt.")
return {
"original_tokens": original_total,
"estimated_savings": len(optimizations) * 100, # Estimat
"optimizations": optimizations,
"tps_impact": (
"Kortere prompts → raskere prefill → "
"lavere time-to-first-token")
}
# Prompt caching for gjentatte prefixer
def design_cacheable_prompt(
static_system: str,
static_examples: list[str],
dynamic_input: str
) -> list[dict]:
"""Design prompt structure optimized for prompt caching."""
# Plaserer alt statisk innhold FØRST (minimum 1024 tokens)
# Prompt caching cacher identiske prefixer
messages = [
{
"role": "system",
"content": static_system # Statisk — cacheable
}
]
# Legg til eksempler som del av cacheable prefix
for example in static_examples:
messages.extend([
{"role": "user", "content": example["input"]},
{"role": "assistant", "content": example["output"]}
])
# Dynamisk input SIST
messages.append({
"role": "user",
"content": dynamic_input # Varierer per forespørsel
})
return messages
```
## GPU Utilization og throughput-monitorering
### Azure Monitor-metrikker for TPS
```python
# KQL: Beregn faktisk TPS fra Azure Monitor
TPS_CALCULATION = """
AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "GeneratedTokens"
| summarize
TotalTokens = sum(Total),
AvgTokensPerMinute = avg(Total),
MaxTokensPerMinute = max(Total)
by bin(TimeGenerated, 1m), deploymentName = tostring(
split(DimensionValue, ",")[0])
| extend TokensPerSecond = TotalTokens / 60.0
| project TimeGenerated, deploymentName,
TokensPerSecond = round(TokensPerSecond, 1),
AvgTPM = round(AvgTokensPerMinute, 0)
| order by TimeGenerated desc
"""
# Utilization vs TPS-korrelasjon
UTILIZATION_VS_TPS = """
AzureMetrics
| where MetricName in ("ProvisionedManagedUtilizationV2", "GeneratedTokens")
| summarize
Utilization = avgif(Average,
MetricName == "ProvisionedManagedUtilizationV2"),
TPS = sumif(Total,
MetricName == "GeneratedTokens") / 60.0
by bin(TimeGenerated, 5m)
| where Utilization > 0
| project TimeGenerated,
Utilization = round(Utilization, 1),
TPS = round(TPS, 1)
| order by TimeGenerated desc
"""
```
### TPS-overvåking dashboard
```python
# Alert rule: Varsle når TPS faller under terskel
ALERT_CONFIG = {
"name": "Low TPS Alert",
"description": "Tokens per second under forventet nivå",
"condition": {
"query": """
AzureMetrics
| where MetricName == "GeneratedTokens"
| summarize TPS = sum(Total) / 300.0
by bin(TimeGenerated, 5m)
| where TPS < 20
""",
"threshold": 0,
"frequency_minutes": 5,
"window_minutes": 15
},
"severity": 2, # Warning
"action_group": "ai-ops-team"
}
```
## Throughput per PTU per modell
### Offisielle TPS-mål fra Microsoft
| Modell | Input TPM/PTU | Latens-mål (TPS) | Min PTU (Global) | Min PTU (Regional) |
|--------|--------------|-------------------|-------------------|---------------------|
| gpt-5.2 | 3,400 | 99% > 50 TPS | 15 | 50 |
| gpt-5.1 | 4,750 | 99% > 50 TPS | 15 | 50 |
| gpt-5 | 4,750 | 99% > 50 TPS | 15 | 50 |
| gpt-5-mini | 23,750 | 99% > 80 TPS | 15 | 25 |
| gpt-4.1 | 3,000 | 99% > 80 TPS | 15 | 50 |
| gpt-4.1-mini | 14,900 | 99% > 90 TPS | 15 | 25 |
| gpt-4.1-nano | 59,400 | 99% > 100 TPS | 15 | 25 |
| o3 | 3,000 | 99% > 80 TPS | 15 | 50 |
| o4-mini | 5,400 | 99% > 90 TPS | 15 | 25 |
| gpt-4o | 2,500 | 99% > 25 TPS | 15 | 50 |
| gpt-4o-mini | 37,000 | 99% > 33 TPS | 15 | 25 |
### Predicted Outputs for TPS-forbedring
```python
# Predicted outputs kan øke TPS for forutsigbare oppgaver
# Eksempel: Kode-refactoring der output ligner input
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint="https://my-aoai.openai.azure.com",
api_key="...",
api_version="2024-12-01-preview"
)
# Original kode som skal refaktoreres
original_code = """
def process_data(data):
result = []
for item in data:
if item['status'] == 'active':
result.append(item['value'])
return result
"""
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "user",
"content": f"Refaktorer med list comprehension:\n{original_code}"}
],
prediction={
"type": "content",
"content": original_code # Forventet output ligner input
}
)
# Sjekk prediction effektivitet
usage = response.usage.completion_tokens_details
print(f"Accepted predictions: {usage.accepted_prediction_tokens}")
print(f"Rejected predictions: {usage.rejected_prediction_tokens}")
# Accepted tokens reduserer latens uten ekstra kostnad
```
## Norsk offentlig sektor
- **Brukeropplevelse**: For AI-assistenter brukt av saksbehandlere er TPS direkte merkbar. Streaming med > 50 TPS føles "responsivt", mens < 20 TPS føles tregt.
- **PTU-valg**: Velg modell og PTU-nivå basert på brukerforventninger — gpt-4.1-nano med 100 TPS for enkle oppgaver, gpt-4.1 med 80 TPS for komplekse analyser.
- **Kostnadsoptimalisering**: Prompt caching gir opptil 100% rabatt på cached input tokens for PTU — design prompts med lange statiske prefixer for å maksimere cache hit rate.
- **SLA-krav**: Dokumenter forventet TPS i tjenesteavtaler. PTU-mål er "99% > N TPS beregnet som p50 over 5-minutters vinduer".
## Beslutningsrammeverk
| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Sanntids chat (høy TPS viktig) | PTU med gpt-4.1-mini/nano | Høyest TPS per PTU |
| Kompleks analyse (kvalitet > TPS) | PTU med gpt-4.1 eller o3 | Akseptabel TPS med best kvalitet |
| Prompt caching mulig | Design lange statiske prefixer | Opptil 100% rabatt på cached tokens |
| Forutsigbart output | Predicted Outputs | Reduserer latens for matching tokens |
| Variabel workload | Standard deployment | Betal per token, ingen PTU-forpliktelse |
## Referanser
- [Performance and latency](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/latency) — TPS og throughput forklaring
- [Provisioned throughput onboarding](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/provisioned-throughput-onboarding) — PTU TPS-mål per modell
- [Prompt caching](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/prompt-caching) — Cache-basert TPS-forbedring
- [Predicted outputs](https://learn.microsoft.com/azure/ai-foundry/openai/how-to/predicted-outputs) — Spekulativ generering
- [Foundry PTU calculator](https://ai.azure.com/resource/calculator) — Kapasitetskalkulator
## For Cosmo
- **Bruk denne referansen** når kunden ønsker å optimalisere responstid for AI-tjenester, eller når de skal dimensjonere PTU-deployments.
- TPS-mål varierer dramatisk mellom modeller: gpt-4.1-nano gir 100 TPS vs. gpt-4o med 25 TPS — velg modell basert på oppgavens kompleksitet.
- Prompt caching er den enkleste TPS-forbedringen — sørg for at de første 1024+ tokens er identiske mellom forespørsler.
- Predicted Outputs gir latensreduksjon for oppgaver der output ligner input (kode-refactoring, oversettelse) men kan øke kostnad ved lav acceptance rate.
- Monitorer PTU utilization — når den nærmer seg 100%, øker latens drastisk og nye forespørsler throttles.