feat(ultraplan-local): v1.6.0 — /ultraresearch-local deep research command
Add /ultraresearch-local for structured research combining local codebase analysis with external knowledge via parallel agent swarms. Produces research briefs with triangulation, confidence ratings, and source quality assessment. New command: /ultraresearch-local with modes --quick, --local, --external, --fg. New agents: research-orchestrator (opus), docs-researcher, community-researcher, security-researcher, contrarian-researcher, gemini-bridge (all sonnet). New template: research-brief-template.md. Integration: --research flag in /ultraplan-local accepts pre-built research briefs (up to 3), enriches the interview and exploration phases. Planning orchestrator cross-references brief findings during synthesis. Design principle: Context Engineering — right information to right agent at right time. Research briefs are structured artifacts in the pipeline: ultraresearch → brief → ultraplan --research → plan → ultraexecute. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen
commit
baa2d0220b
488 changed files with 213221 additions and 0 deletions
|
|
@ -0,0 +1,609 @@
|
|||
# Distributed Tracing for AI Pipelines
|
||||
|
||||
**Kategori:** Monitoring & Observability
|
||||
**Dato:** 2026-02-05
|
||||
**Status:** ✅ Komplett
|
||||
|
||||
## Innledning
|
||||
|
||||
Distributed tracing (distribuert sporing) gir end-to-end synlighet gjennom hele AI-pipelinens kjede av operasjoner — fra brukerforespørsel, via LLM-kall, tool-anrop og multi-agent-samarbeid, til ferdig respons. Dette er kritisk for å diagnostisere ytelsesflaskehalser, identifisere feiltilstander, og optimalisere komplekse agentic AI-systemer.
|
||||
|
||||
Microsoft sin tilnærming er bygget på **OpenTelemetry**-standarder og integrerer sømløst med **Azure Monitor Application Insights**, med native støtte for AI-spesifikke semantiske konvensjoner (OpenTelemetry Gen AI Semantic Conventions).
|
||||
|
||||
## Nøkkelkonsepter
|
||||
|
||||
### Traces, Spans og Correlation
|
||||
|
||||
- **Trace:** Fullstendig reise for en operasjon gjennom systemet (f.eks. én brukerforespørsel til en AI-agent)
|
||||
- **Span:** Individuell operasjon innenfor en trace (LLM-kall, tool-invokasjon, HTTP-request)
|
||||
- **Attributes:** Key-value metadata knyttet til spans (model name, token count, tool parameters)
|
||||
- **Correlation ID:** `operation_Id` og `operation_ParentId` som knytter alle spans i en trace sammen
|
||||
|
||||
### W3C Trace Context
|
||||
|
||||
Microsoft støtter W3C Trace Context-standarden for cross-service propagation:
|
||||
|
||||
- **traceparent:** Globally unique operation ID + span ID (propageres via HTTP-headers)
|
||||
- **tracestate:** System-spesifikk trace-kontekst
|
||||
- **Bakoverkompatibilitet:** Application Insights SDK støtter både W3C og legacy Request-Id-protokoller
|
||||
|
||||
## OpenTelemetry for AI Pipelines
|
||||
|
||||
### Semantic Conventions for Generative AI
|
||||
|
||||
OpenTelemetry definerer standardiserte span-navn og attributter for AI-operasjoner:
|
||||
|
||||
**Standard AI Spans:**
|
||||
- `gen_ai.model.completion` — LLM-inferens
|
||||
- `gen_ai.tool.execution` — Tool/function-kall
|
||||
- `gen_ai.agent.invoke` — Agent-invokasjon
|
||||
- `gen_ai.agent_planning` — Agent-planleggingssteg
|
||||
- `gen_ai.agent_to_agent_interaction` — Multi-agent-kommunikasjon
|
||||
|
||||
**Standard Attributter:**
|
||||
- `gen_ai.system` — AI-system (OpenAI, Azure AI, etc.)
|
||||
- `gen_ai.request.model` — Modellnavn
|
||||
- `gen_ai.usage.prompt_tokens` — Prompt-tokens
|
||||
- `gen_ai.usage.completion_tokens` — Completion-tokens
|
||||
- `gen_ai.response.finish_reason` — Årsak til ferdigstillelse
|
||||
|
||||
### Multi-Agent Observability
|
||||
|
||||
Microsoft har utviklet nye semantic conventions for multi-agent-systemer (i samarbeid med Cisco Outshift):
|
||||
|
||||
| Span Type | Formål | Eksempel |
|
||||
|-----------|--------|----------|
|
||||
| `execute_task` | Overvåker task-dekomponering og event-propagering | Bryter ned kompleks forespørsel |
|
||||
| `agent_to_agent_interaction` | Sporer kommunikasjon mellom agenter | Agent A ber Agent B om data |
|
||||
| `agent.state.management` | Kontekst- og minnehåndtering | Long-term memory-oppdatering |
|
||||
| `agent_planning` | Agentens interne planleggingssteg | Reasoning-steg før tool-valg |
|
||||
| `agent_orchestration` | Agent-til-agent-orkestrering | Main agent delegerer til sub-agents |
|
||||
|
||||
## Implementering i Microsoft-stakken
|
||||
|
||||
### 1. Azure AI Foundry + Azure Monitor
|
||||
|
||||
**Setup (Python):**
|
||||
|
||||
```python
|
||||
import os
|
||||
from azure.ai.projects import AIProjectClient
|
||||
from azure.identity import DefaultAzureCredential
|
||||
from azure.monitor.opentelemetry import configure_azure_monitor
|
||||
from opentelemetry import trace
|
||||
|
||||
# Enable content recording (valgfritt - kan inneholde sensitive data)
|
||||
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"
|
||||
|
||||
# Koble til AI Foundry-prosjekt
|
||||
project_client = AIProjectClient(
|
||||
credential=DefaultAzureCredential(),
|
||||
endpoint=os.environ["PROJECT_ENDPOINT"]
|
||||
)
|
||||
|
||||
# Hent Application Insights connection string
|
||||
connection_string = project_client.telemetry.get_application_insights_connection_string()
|
||||
|
||||
# Konfigurer Azure Monitor
|
||||
configure_azure_monitor(connection_string=connection_string)
|
||||
|
||||
# Start tracing
|
||||
tracer = trace.get_tracer(__name__)
|
||||
|
||||
with tracer.start_as_current_span("ai-agent-session"):
|
||||
agent = project_client.agents.create_agent(
|
||||
model="gpt-4o",
|
||||
name="support-agent",
|
||||
instructions="Du er en supportagent"
|
||||
)
|
||||
thread = project_client.agents.threads.create()
|
||||
message = project_client.agents.messages.create(
|
||||
thread_id=thread.id,
|
||||
role="user",
|
||||
content="Hjelp meg med å feilsøke"
|
||||
)
|
||||
run = project_client.agents.runs.create_and_process(
|
||||
thread_id=thread.id,
|
||||
agent_id=agent.id
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Azure Functions + OpenTelemetry
|
||||
|
||||
**Konfigurer host.json:**
|
||||
|
||||
```json
|
||||
{
|
||||
"version": "2.0",
|
||||
"telemetryMode": "OpenTelemetry",
|
||||
"extensions": {
|
||||
"serviceBus": {
|
||||
"maxConcurrentCalls": 10
|
||||
}
|
||||
},
|
||||
"extensionBundle": {
|
||||
"id": "Microsoft.Azure.Functions.ExtensionBundle",
|
||||
"version": "[4.*, 5.0.0)"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Python Function med tracing:**
|
||||
|
||||
```python
|
||||
import azure.functions as func
|
||||
from azure.monitor.opentelemetry import configure_azure_monitor
|
||||
import os
|
||||
|
||||
# Konfigurer Azure Monitor
|
||||
configure_azure_monitor(
|
||||
connection_string=os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"]
|
||||
)
|
||||
|
||||
app = func.FunctionApp()
|
||||
|
||||
@app.function_name("orchestrator")
|
||||
@app.route(route="orchestrator", auth_level=func.AuthLevel.ANONYMOUS)
|
||||
def orchestrator(req: func.HttpRequest) -> func.HttpResponse:
|
||||
# Automatisk tracet av Azure Functions OpenTelemetry-integrasjon
|
||||
# Alle HTTP-kall, Service Bus-meldinger, og dependencies trackes
|
||||
return func.HttpResponse("OK", status_code=200)
|
||||
```
|
||||
|
||||
### 3. LangChain/LangGraph + Azure AI Tracing
|
||||
|
||||
**Setup:**
|
||||
|
||||
```python
|
||||
from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
|
||||
from langchain_openai import AzureChatOpenAI
|
||||
import os
|
||||
|
||||
# Opprett tracer
|
||||
azure_tracer = AzureAIOpenTelemetryTracer(
|
||||
connection_string=os.environ["APPLICATION_INSIGHTS_CONNECTION_STRING"],
|
||||
enable_content_recording=True,
|
||||
name="LangChain Agent",
|
||||
id="langchain_agent_v1"
|
||||
)
|
||||
|
||||
# Konfigurer model med callbacks
|
||||
model = AzureChatOpenAI(
|
||||
azure_deployment=os.environ["AZURE_OPENAI_CHAT_DEPLOYMENT"],
|
||||
azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],
|
||||
api_version="2024-08-01-preview",
|
||||
callbacks=[azure_tracer]
|
||||
)
|
||||
|
||||
# Alle LLM-kall, tool-invokasjon, og agent-steg trackes automatisk
|
||||
```
|
||||
|
||||
### 4. Semantic Kernel
|
||||
|
||||
Semantic Kernel har innebygd OpenTelemetry-støtte:
|
||||
|
||||
**Automatisk metrics:**
|
||||
- `semantic_kernel.function.invocation.duration` (Histogram) — Funksjonsutførelsestid
|
||||
- `semantic_kernel.function.streaming.duration` (Histogram) — Streaming-utførelsestid
|
||||
- `semantic_kernel.function.invocation.token_usage.prompt` — Prompt-tokens
|
||||
- `semantic_kernel.function.invocation.token_usage.completion` — Completion-tokens
|
||||
|
||||
**Aktiviteter (Spans):**
|
||||
- Hver kernel function-execution genererer en Activity
|
||||
- Hver AI-modellkall genereres som egen Activity
|
||||
- Activity source: `"Microsoft.SemanticKernel"`
|
||||
|
||||
### 5. Custom Functions og Tools
|
||||
|
||||
**Trace egne funksjoner:**
|
||||
|
||||
```python
|
||||
from opentelemetry import trace
|
||||
|
||||
tracer = trace.get_tracer(__name__)
|
||||
|
||||
def rag_retrieval(query: str) -> list[str]:
|
||||
with tracer.start_as_current_span("rag_retrieval") as span:
|
||||
span.set_attribute("query", query)
|
||||
span.set_attribute("retrieval.database", "azure_ai_search")
|
||||
|
||||
# Utfør retrieval
|
||||
results = search_index(query)
|
||||
|
||||
span.set_attribute("retrieval.results_count", len(results))
|
||||
span.set_attribute("retrieval.latency_ms", 120)
|
||||
|
||||
return results
|
||||
|
||||
def agent_tool_call(tool_name: str, arguments: dict):
|
||||
with tracer.start_as_current_span("execute_tool") as span:
|
||||
span.set_attribute("tool.name", tool_name)
|
||||
span.set_attribute("tool.call.arguments", str(arguments))
|
||||
|
||||
result = execute_tool(tool_name, arguments)
|
||||
|
||||
span.set_attribute("tool.call.results", str(result))
|
||||
return result
|
||||
```
|
||||
|
||||
## End-to-End Trace Correlation
|
||||
|
||||
### Distribuert Tracing Across Services
|
||||
|
||||
**Scenario:** Bruker → Azure Functions → Azure OpenAI → Azure AI Search → Response
|
||||
|
||||
**Trace Flow:**
|
||||
|
||||
1. **HTTP Request** (traceparent-header propageres automatisk)
|
||||
- `operation_Id`: `abc123def456`
|
||||
- Span: `GET /api/chat`
|
||||
|
||||
2. **Azure Function Processing**
|
||||
- `operation_ParentId`: `abc123def456`
|
||||
- Span: `process_chat_request`
|
||||
|
||||
3. **Azure OpenAI API Call** (dependency tracked)
|
||||
- `operation_ParentId`: `process_chat_request`
|
||||
- Span: `gen_ai.model.completion`
|
||||
- Attributes: `model=gpt-4o`, `prompt_tokens=150`, `completion_tokens=75`
|
||||
|
||||
4. **Azure AI Search Query** (dependency tracked)
|
||||
- `operation_ParentId`: `process_chat_request`
|
||||
- Span: `azure_ai_search.query`
|
||||
- Attributes: `index=knowledge_base`, `results_count=5`
|
||||
|
||||
5. **Service Bus Message** (context propageres via message properties)
|
||||
- `operation_ParentId`: `process_chat_request`
|
||||
- Span: `servicebus.send`
|
||||
|
||||
**Resultat i Application Insights:**
|
||||
- Application Map viser alle tjenester grafisk
|
||||
- Transaction Search viser fullstendig call stack
|
||||
- End-to-End Transaction Details viser timing for hver operasjon
|
||||
|
||||
### Query Traces i Application Insights
|
||||
|
||||
**Kusto Query for å finne relatert telemetri:**
|
||||
|
||||
```kusto
|
||||
let operationId = "abc123def456";
|
||||
(requests | union dependencies | union traces | union exceptions)
|
||||
| where operation_Id == operationId
|
||||
| project timestamp, itemType, name, id, operation_ParentId, operation_Id, duration
|
||||
| order by timestamp asc
|
||||
```
|
||||
|
||||
**Analyse AI-spesifikke spans:**
|
||||
|
||||
```kusto
|
||||
dependencies
|
||||
| where type == "AI"
|
||||
| extend model = tostring(customDimensions.["gen_ai.request.model"])
|
||||
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
|
||||
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
|
||||
| summarize
|
||||
avgDuration = avg(duration),
|
||||
totalPromptTokens = sum(promptTokens),
|
||||
totalCompletionTokens = sum(completionTokens),
|
||||
requestCount = count()
|
||||
by model
|
||||
| order by avgDuration desc
|
||||
```
|
||||
|
||||
## Trace Visualization og Analysis
|
||||
|
||||
### Application Insights Features
|
||||
|
||||
**1. Application Map**
|
||||
- Visuell representasjon av tjeneste-dependencies
|
||||
- Automatisk deteksjon av performance-problemer
|
||||
- Highlighting av feiltilstander
|
||||
|
||||
**2. Transaction Search**
|
||||
- Søk etter spesifikke traces basert på:
|
||||
- Operation ID
|
||||
- Tidsvindu
|
||||
- Resultat (success/failure)
|
||||
- Duration threshold
|
||||
|
||||
**3. End-to-End Transaction Details**
|
||||
- Komplett trace timeline
|
||||
- Span-detaljer (start/end times, attributes)
|
||||
- Korrelerte logger
|
||||
- Performance metrics per span
|
||||
|
||||
**4. Performance View**
|
||||
- Gjennomsnittlig duration per operation
|
||||
- P95/P99 latency
|
||||
- Dependency latency breakdown
|
||||
|
||||
**5. Failures Blade**
|
||||
- Exception tracking korrelert med traces
|
||||
- Failure rate per endpoint
|
||||
- Root cause analysis
|
||||
|
||||
### Local Tracing (Development)
|
||||
|
||||
**Aspire Dashboard (lokal OTLP viewer):**
|
||||
|
||||
```bash
|
||||
pip install opentelemetry-exporter-otlp
|
||||
|
||||
# Start Aspire Dashboard
|
||||
docker run --rm -it -p 18888:18888 -p 4317:18889 \
|
||||
mcr.microsoft.com/dotnet/aspire-dashboard:latest
|
||||
```
|
||||
|
||||
**Console Export (debugging):**
|
||||
|
||||
```python
|
||||
from opentelemetry.sdk.trace.export import ConsoleSpanExporter, SimpleSpanProcessor
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
|
||||
span_exporter = ConsoleSpanExporter()
|
||||
tracer_provider = TracerProvider()
|
||||
tracer_provider.add_span_processor(SimpleSpanProcessor(span_exporter))
|
||||
trace.set_tracer_provider(tracer_provider)
|
||||
```
|
||||
|
||||
## Performance Bottleneck Identification
|
||||
|
||||
### Analyse Latency Distribution
|
||||
|
||||
**Identifiser trege spans:**
|
||||
|
||||
```kusto
|
||||
dependencies
|
||||
| where operation_Name == "chat_completion"
|
||||
| summarize
|
||||
p50 = percentile(duration, 50),
|
||||
p90 = percentile(duration, 90),
|
||||
p99 = percentile(duration, 99)
|
||||
by name
|
||||
| where p99 > 5000 // Over 5 sekunder
|
||||
```
|
||||
|
||||
**Finn flaskehalser i multi-step pipeline:**
|
||||
|
||||
```kusto
|
||||
let traceId = "abc123";
|
||||
dependencies
|
||||
| where operation_Id == traceId
|
||||
| project timestamp, name, duration, operation_ParentId
|
||||
| order by timestamp asc
|
||||
// Visualiser i Timeline-chart for å se hvor tid brukes
|
||||
```
|
||||
|
||||
### Token Usage Analysis
|
||||
|
||||
```kusto
|
||||
traces
|
||||
| where message contains "gen_ai.usage"
|
||||
| extend promptTokens = toint(customDimensions.["gen_ai.usage.prompt_tokens"])
|
||||
| extend completionTokens = toint(customDimensions.["gen_ai.usage.completion_tokens"])
|
||||
| summarize
|
||||
totalCost = sum((promptTokens * 0.00003) + (completionTokens * 0.00006))
|
||||
by bin(timestamp, 1h)
|
||||
| render timechart
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Consistent Span Attributes
|
||||
|
||||
Bruk standardiserte attributt-navn:
|
||||
- `gen_ai.*` for AI-spesifikke spans
|
||||
- `tool.*` for tool-invokasjon
|
||||
- `agent.*` for agent-metadata
|
||||
- Følg OpenTelemetry Semantic Conventions
|
||||
|
||||
### 2. Redact Sensitive Content
|
||||
|
||||
**Ikke log sensitive data i spans:**
|
||||
|
||||
```python
|
||||
# IKKE gjør dette:
|
||||
span.set_attribute("user.password", password)
|
||||
|
||||
# Gjør dette i stedet:
|
||||
span.set_attribute("user.id", user_id)
|
||||
span.set_attribute("request.sanitized", True)
|
||||
```
|
||||
|
||||
**Deaktiver content recording i prod:**
|
||||
|
||||
```python
|
||||
# Development
|
||||
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"
|
||||
|
||||
# Production
|
||||
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "false"
|
||||
```
|
||||
|
||||
### 3. Correlate Evaluation Runs
|
||||
|
||||
Knytt trace IDs til evaluation-runs:
|
||||
|
||||
```python
|
||||
span.set_attribute("evaluation.run_id", evaluation_run_id)
|
||||
span.set_attribute("evaluation.metrics", json.dumps(metrics))
|
||||
```
|
||||
|
||||
### 4. Service Name for Multi-App Scenarios
|
||||
|
||||
Identifiser tjenester via `OTEL_SERVICE_NAME`:
|
||||
|
||||
```bash
|
||||
export OTEL_SERVICE_NAME="support-agent-api"
|
||||
export OTEL_RESOURCE_ATTRIBUTES="service.namespace=production,service.instance.id=instance-01"
|
||||
```
|
||||
|
||||
I Application Insights mappes dette til `cloud_RoleName`:
|
||||
|
||||
```kusto
|
||||
traces
|
||||
| where cloud_RoleName == "support-agent-api"
|
||||
```
|
||||
|
||||
### 5. Sampling for High-Volume Scenarios
|
||||
|
||||
**Adaptive sampling (automatisk i Azure Monitor):**
|
||||
- Reduserer volum uten å miste viktige traces
|
||||
- Prioriterer feil og trege forespørsler
|
||||
|
||||
**Custom sampling (avansert):**
|
||||
|
||||
```python
|
||||
from opentelemetry.sdk.trace.sampling import TraceIdRatioBased
|
||||
|
||||
# Sample 10% av traces
|
||||
sampler = TraceIdRatioBased(rate=0.1)
|
||||
tracer_provider = TracerProvider(sampler=sampler)
|
||||
```
|
||||
|
||||
## Azure Functions OpenTelemetry Pattern
|
||||
|
||||
### Multi-Function Distributed Trace
|
||||
|
||||
**Function 1 (HTTP Trigger):**
|
||||
|
||||
```python
|
||||
@app.route(route="function1")
|
||||
def function1(req: func.HttpRequest) -> func.HttpResponse:
|
||||
# Caller function2 (automatic trace propagation)
|
||||
response = requests.get(f"{base_url}/api/function2")
|
||||
return func.HttpResponse(response.text)
|
||||
```
|
||||
|
||||
**Function 2 (HTTP Trigger + Service Bus Output):**
|
||||
|
||||
```python
|
||||
@app.route(route="function2")
|
||||
@app.service_bus_queue_output(
|
||||
arg_name="outputmsg",
|
||||
queue_name="processing-queue",
|
||||
connection="ServiceBusConnection"
|
||||
)
|
||||
def function2(req: func.HttpRequest, outputmsg: func.Out[str]):
|
||||
# Send message (trace context propageres automatisk)
|
||||
outputmsg.set("Process this")
|
||||
return func.HttpResponse("OK")
|
||||
```
|
||||
|
||||
**Function 3 (Service Bus Trigger):**
|
||||
|
||||
```python
|
||||
@app.service_bus_queue_trigger(
|
||||
arg_name="msg",
|
||||
queue_name="processing-queue",
|
||||
connection="ServiceBusConnection"
|
||||
)
|
||||
def function3(msg: func.ServiceBusMessage):
|
||||
# Automatisk korrelert med function1 og function2
|
||||
logging.info(f"Processing: {msg.get_body().decode()}")
|
||||
```
|
||||
|
||||
**Resultat:** En enkelt HTTP-request til function1 genererer en komplett trace som viser:
|
||||
- HTTP request → function1
|
||||
- function1 → function2 (HTTP dependency)
|
||||
- function2 → Service Bus (messaging dependency)
|
||||
- Service Bus → function3 (queue trigger)
|
||||
|
||||
## Integrasjon med AI Foundry Tracing
|
||||
|
||||
### View Traces i Foundry Portal
|
||||
|
||||
1. Naviger til **Tracing** i AI Foundry-prosjekt
|
||||
2. Filtrer traces etter:
|
||||
- Tidsvindu
|
||||
- Status (success/failed)
|
||||
- Agent/model
|
||||
3. Drill-down i individual trace for span-detaljer
|
||||
|
||||
### Thread Logs i Agents Playground
|
||||
|
||||
- **Thread details:** Fullstendig konversasjonshistorikk
|
||||
- **Run information:** Agent execution metadata
|
||||
- **Ordered run steps:** Sekvens av operasjoner
|
||||
- **Tool calls:** Input/output for hver tool-invokasjon
|
||||
- **Linked evaluations:** Automatic quality metrics (hvis aktivert)
|
||||
|
||||
## Troubleshooting Common Issues
|
||||
|
||||
### Problem: Traces not appearing in Application Insights
|
||||
|
||||
**Løsning:**
|
||||
1. Verifiser connection string:
|
||||
```python
|
||||
print(os.environ["APPLICATIONINSIGHTS_CONNECTION_STRING"])
|
||||
```
|
||||
2. Sjekk at `configure_azure_monitor()` kalles tidlig i app lifecycle
|
||||
3. Vent 2-5 minutter (ingestion lag)
|
||||
4. Sjekk sampling rate (hvis custom sampling)
|
||||
|
||||
### Problem: Missing trace context across services
|
||||
|
||||
**Løsning:**
|
||||
1. Verifiser W3C Trace Context headers propageres:
|
||||
```python
|
||||
# Inspect outgoing request headers
|
||||
print(request.headers.get("traceparent"))
|
||||
```
|
||||
2. Bruk instrumentation libraries (ikke manual HTTP calls uten context propagation)
|
||||
3. For Azure Functions: Sjekk at alle functions har `"telemetryMode": "OpenTelemetry"`
|
||||
|
||||
### Problem: High cardinality attributes causing performance issues
|
||||
|
||||
**Løsning:**
|
||||
- Unngå unique IDs som span attributes (bruk aggregated metrics i stedet)
|
||||
- Reduser sampling rate for høy-volum scenarios
|
||||
- Bruk tags/dimensions med lav cardinality
|
||||
|
||||
## For Cosmo
|
||||
|
||||
Ved arkitekturveiledning:
|
||||
|
||||
**Når bruker spør om:**
|
||||
- "Hvordan kan jeg feilsøke min AI-pipeline?"
|
||||
- "Hvordan tracke end-to-end ytelse i multi-agent-systemet?"
|
||||
- "Hvordan finne flaskehalser i RAG-pipeline?"
|
||||
- "Hvordan korrelere LLM-kall med tool-invokasjon?"
|
||||
|
||||
**Svar med:**
|
||||
1. **Beskriv trace-arkitektur:** Spans → Traces → Operation ID correlation
|
||||
2. **Anbefal OpenTelemetry + Azure Monitor:** Native støtte, AI-spesifikke semantics
|
||||
3. **Gi konkret implementering:** Vis code snippets for brukerens plattform (Foundry, Functions, LangChain, etc.)
|
||||
4. **Highlight Application Insights features:** Application Map, Transaction Search, Performance View
|
||||
5. **Sikkerhet:** Påminn om content recording (deaktiver i prod hvis sensitive data)
|
||||
6. **Query-eksempler:** Gi Kusto-queries for vanlige analyse-scenarioer
|
||||
|
||||
**Decision factors:**
|
||||
- **High-volume scenarios:** Vurder adaptive sampling
|
||||
- **Multi-region deployments:** Bruk `cloud_RoleName` og `cloud_RoleInstance` for å skille instances
|
||||
- **Compliance-krav:** Deaktiver content recording, bruk private Application Insights
|
||||
- **Local development:** Anbefal Aspire Dashboard for rask feedback
|
||||
|
||||
**Trade-offs:**
|
||||
- **Detailed tracing vs. storage cost:** Mer spans = høyere Application Insights-kostnad
|
||||
- **Content recording vs. privacy:** Recording av prompts/completions kan eksponere PII
|
||||
- **Real-time vs. historical analysis:** Live Metrics vs. Kusto queries
|
||||
|
||||
---
|
||||
|
||||
## Kilder og verifisering
|
||||
|
||||
Adapted from Microsoft Learn documentation ([CC BY 4.0](https://creativecommons.org/licenses/by/4.0/)):
|
||||
|
||||
- [Tracing in Azure AI Foundry](https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/trace-local-sdk)
|
||||
- [Azure Monitor OpenTelemetry overview](https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview)
|
||||
- [Azure Functions OpenTelemetry](https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto)
|
||||
- [Distributed tracing in Application Insights](https://learn.microsoft.com/en-us/azure/azure-monitor/app/distributed-trace-data)
|
||||
- [Semantic Kernel observability](https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/)
|
||||
|
||||
Content has been translated to Norwegian, reorganized, and augmented with implementation guidance.
|
||||
|
||||
**Relaterte referanser:**
|
||||
- `azure-monitor-foundations.md` — Application Insights-grunnlag
|
||||
- `token-tracking.md` — Token usage monitoring
|
||||
- `alerting-ai-systems.md` — Alerting på trace data
|
||||
- `app-insights-ai-integration.md` — Application Insights AI-features
|
||||
Loading…
Add table
Add a link
Reference in a new issue