818 lines
24 KiB
Markdown
818 lines
24 KiB
Markdown
# Application Insights for LLM Monitoring
|
|
|
|
**Category:** Monitoring & Observability
|
|
**Last updated:** 2026-02-05
|
|
**Status:** Komplett
|
|
**Type:** reference
|
|
|
|
## Innhold
|
|
|
|
- [Oversikt](#oversikt)
|
|
- [Hvorfor Application Insights for LLM-applikasjoner?](#hvorfor-application-insights-for-llm-applikasjoner)
|
|
- [LLM-spesifikk telemetri i Application Insights](#llm-spesifikk-telemetri-i-application-insights)
|
|
- [Custom Events for AI-interaksjoner](#custom-events-for-ai-interaksjoner)
|
|
- [Distributed Tracing for AI Pipelines](#distributed-tracing-for-ai-pipelines)
|
|
- [Performance Baselines for LLMs](#performance-baselines-for-llms)
|
|
- [Error Tracking og Alerting](#error-tracking-og-alerting)
|
|
- [Framework-integrasjoner](#framework-integrasjoner)
|
|
- [Lokal debugging med Aspire Dashboard](#lokal-debugging-med-aspire-dashboard)
|
|
- [Best Practices](#best-practices)
|
|
- [Visualisering i Azure Portal](#visualisering-i-azure-portal)
|
|
- [For Cosmo Skyberg: Application Insights for LLM Monitoring](#for-cosmo-skyberg-application-insights-for-llm-monitoring)
|
|
|
|
## Oversikt
|
|
|
|
Application Insights er Azures native observability-plattform for å overvåke LLM-applikasjoner med OpenTelemetry-kompatibel tracing. Denne guiden dekker LLM-spesifikk telemetri, custom events for AI-interaksjoner, distributed tracing for AI-pipelines, performance baselines, og error tracking.
|
|
|
|
Application Insights integrerer sømløst med Microsoft Foundry, Azure OpenAI Service, og alle større AI-rammeverk (LangChain, Semantic Kernel, Microsoft Agent Framework, OpenAI Agents SDK).
|
|
|
|
## Hvorfor Application Insights for LLM-applikasjoner?
|
|
|
|
### Utfordringer med LLM-observabilitet
|
|
|
|
LLM-applikasjoner introduserer unike overvåkingsutfordringer:
|
|
|
|
1. **Komplekse kjeder** — Agent kan kjøre 10+ steg med nøstede tool calls
|
|
2. **Varierende flows** — Execution path avhenger av user input og model reasoning
|
|
3. **Lange inputs/outputs** — Prompts og responses kan være 1000+ tokens
|
|
4. **Multi-agent orchestration** — Koordinering mellom flere agenter og tools
|
|
5. **Kostnadskontroll** — Token usage og latency må spores per operasjon
|
|
|
|
### Application Insights løsning
|
|
|
|
- **OpenTelemetry-standard** — Følger GenAI semantic conventions
|
|
- **Full trace tree** — Se nøstede spans for agent → tool → LLM calls
|
|
- **Token tracking** — Custom metrics for input/output tokens og cost
|
|
- **Performance baselines** — P50, P90, P95 latency per operasjon
|
|
- **Error correlation** — Link exceptions til spesifikk LLM call eller tool
|
|
- **Multi-framework** — Samme backend for LangChain, Semantic Kernel, etc.
|
|
|
|
## LLM-spesifikk telemetri i Application Insights
|
|
|
|
### Telemetri-typer for AI-applikasjoner
|
|
|
|
Application Insights lagrer LLM-telemetri i standardtabeller:
|
|
|
|
| Telemetri | Tabell | Bruk for LLM-applikasjoner |
|
|
|-----------|--------|----------------------------|
|
|
| **Request** | `AppRequests` | HTTP request til AI endpoint (chat completion, agent run) |
|
|
| **Dependency** | `AppDependencies` | Kall til Azure OpenAI, embeddings API, vector database |
|
|
| **Trace** | `AppTraces` | Agent reasoning steps, tool outputs, system messages |
|
|
| **Exception** | `AppExceptions` | Model errors (rate limit, content filter), tool failures |
|
|
| **Custom Event** | `AppEvents` | User feedback, agent decisions, evaluation results |
|
|
| **Custom Metric** | `AppMetrics` | Token count, cost per request, embedding dimensions |
|
|
|
|
### OpenTelemetry Spans for GenAI
|
|
|
|
Application Insights støtter OpenTelemetry Semantic Conventions for GenAI:
|
|
|
|
**Standard span-attributter:**
|
|
|
|
```json
|
|
{
|
|
"gen_ai.system": "azure_openai",
|
|
"gen_ai.request.model": "gpt-4o",
|
|
"gen_ai.request.max_tokens": 1000,
|
|
"gen_ai.request.temperature": 0.7,
|
|
"gen_ai.response.finish_reason": "stop",
|
|
"gen_ai.usage.input_tokens": 450,
|
|
"gen_ai.usage.output_tokens": 320,
|
|
"gen_ai.prompt": "[redacted]", // hvis content recording er enabled
|
|
"gen_ai.completion": "[redacted]"
|
|
}
|
|
```
|
|
|
|
**Multi-agent spans (Microsoft-utvidelse):**
|
|
|
|
| Span type | Attributt | Beskrivelse |
|
|
|-----------|-----------|-------------|
|
|
| `execute_task` | — | Task planning og event propagation |
|
|
| `invoke_agent` | `agent.name`, `agent.id` | Agent invocation |
|
|
| `agent_to_agent_interaction` | `source_agent`, `target_agent` | Inter-agent kommunikasjon |
|
|
| `agent.state.management` | `memory_type` | Memory og context management |
|
|
| `agent_planning` | `plan_steps` | Agent's internal planning |
|
|
| `execute_tool` | `tool.name`, `tool.call.arguments`, `tool.call.results` | Tool execution |
|
|
| `gen_ai.evaluation` | `evaluation.name`, `evaluation.score` | Agent performance evaluation |
|
|
|
|
## Custom Events for AI-interaksjoner
|
|
|
|
### Logg user feedback
|
|
|
|
User feedback er kritisk for å evaluere LLM-output kvalitet:
|
|
|
|
```python
|
|
from opentelemetry import trace
|
|
from opentelemetry.trace import Status, StatusCode
|
|
|
|
tracer = trace.get_tracer(__name__)
|
|
|
|
def log_user_feedback(response_id: str, rating: int, comment: str):
|
|
"""Log user feedback som OpenTelemetry event."""
|
|
with tracer.start_as_current_span("user_feedback") as span:
|
|
span.set_attribute("gen_ai.response.id", response_id)
|
|
span.set_attribute("user_feedback.rating", rating)
|
|
span.set_attribute("user_feedback.comment", comment)
|
|
|
|
# Link til parent span (LLM response)
|
|
span.set_attribute("parent_span_id", get_response_span_id(response_id))
|
|
```
|
|
|
|
**Query i Application Insights:**
|
|
|
|
```kusto
|
|
AppTraces
|
|
| where OperationName == "user_feedback"
|
|
| extend Rating = tolong(Properties.user_feedback_rating)
|
|
| summarize AvgRating = avg(Rating), Count = count() by bin(TimeGenerated, 1h)
|
|
```
|
|
|
|
### Agent decisions som events
|
|
|
|
Logg agent decisions for å forstå reasoning:
|
|
|
|
```python
|
|
def log_agent_decision(agent_name: str, decision: str, reasoning: str):
|
|
"""Log agent decision point."""
|
|
with tracer.start_as_current_span("agent_decision") as span:
|
|
span.set_attribute("agent.name", agent_name)
|
|
span.set_attribute("agent.decision", decision)
|
|
span.set_attribute("agent.reasoning", reasoning)
|
|
span.set_attribute("timestamp", datetime.utcnow().isoformat())
|
|
```
|
|
|
|
### Tool invocation tracking
|
|
|
|
Track tool usage patterns:
|
|
|
|
```python
|
|
def track_tool_usage(tool_name: str, success: bool, latency_ms: float):
|
|
"""Track tool execution metrics."""
|
|
with tracer.start_as_current_span(f"tool_{tool_name}") as span:
|
|
span.set_attribute("tool.name", tool_name)
|
|
span.set_attribute("tool.success", success)
|
|
span.set_attribute("tool.latency_ms", latency_ms)
|
|
|
|
if not success:
|
|
span.set_status(Status(StatusCode.ERROR))
|
|
```
|
|
|
|
## Distributed Tracing for AI Pipelines
|
|
|
|
### Trace hele agent execution
|
|
|
|
Application Insights viser full trace tree for agent execution:
|
|
|
|
```
|
|
[Request] POST /chat/completions (2.3s)
|
|
├─ [Span] agent_session (2.2s)
|
|
│ ├─ [Span] agent_planning (0.1s)
|
|
│ ├─ [Span] execute_task (2.0s)
|
|
│ │ ├─ [Span] invoke_agent: ResearchAgent (1.2s)
|
|
│ │ │ ├─ [Dependency] Azure OpenAI gpt-4o (0.8s)
|
|
│ │ │ └─ [Span] execute_tool: web_search (0.4s)
|
|
│ │ │ └─ [Dependency] Bing Search API (0.35s)
|
|
│ │ └─ [Span] invoke_agent: SummaryAgent (0.7s)
|
|
│ │ └─ [Dependency] Azure OpenAI gpt-4o-mini (0.6s)
|
|
│ └─ [Span] agent.state.management (0.1s)
|
|
└─ [Custom Event] user_feedback (rating: 5)
|
|
```
|
|
|
|
### Correlation IDs
|
|
|
|
Application Insights bruker W3C Trace Context for correlation:
|
|
|
|
- `operation_Id` — Unique ID for hele request (trace-id)
|
|
- `operation_ParentId` — Parent span ID (parent-id)
|
|
- `id` — Current span ID
|
|
|
|
**Query relaterte spans:**
|
|
|
|
```kusto
|
|
AppRequests
|
|
| where OperationId == "abc123..."
|
|
| union (AppDependencies | where OperationId == "abc123...")
|
|
| union (AppTraces | where OperationId == "abc123...")
|
|
| order by TimeGenerated asc
|
|
```
|
|
|
|
### Instrumentering med Microsoft Foundry SDK
|
|
|
|
**Python setup:**
|
|
|
|
```python
|
|
from azure.ai.projects import AIProjectClient
|
|
from azure.monitor.opentelemetry import configure_azure_monitor
|
|
from opentelemetry import trace
|
|
|
|
# Connect til AI Foundry project
|
|
project_client = AIProjectClient(
|
|
credential=DefaultAzureCredential(),
|
|
endpoint=os.environ["PROJECT_ENDPOINT"]
|
|
)
|
|
|
|
# Hent Application Insights connection string
|
|
connection_string = project_client.telemetry.get_application_insights_connection_string()
|
|
|
|
# Enable Azure Monitor tracing
|
|
configure_azure_monitor(connection_string=connection_string)
|
|
|
|
# Get tracer
|
|
tracer = trace.get_tracer(__name__)
|
|
|
|
# Trace agent execution
|
|
with tracer.start_as_current_span("my_agent_flow"):
|
|
agent = project_client.agents.create_agent(...)
|
|
run = project_client.agents.runs.create_and_process(...)
|
|
```
|
|
|
|
### Content recording (opt-in)
|
|
|
|
For å trace prompt/completion content (kan inneholde persondata):
|
|
|
|
```python
|
|
import os
|
|
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"
|
|
```
|
|
|
|
**Alternativt via miljøvariabel:**
|
|
|
|
```bash
|
|
# PowerShell
|
|
$env:AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED = "true"
|
|
|
|
# Bash
|
|
export AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true
|
|
```
|
|
|
|
⚠️ **Sikkerhet:** Content recording logger prompts og responses. Vurder GDPR/privacy før aktivering.
|
|
|
|
## Performance Baselines for LLMs
|
|
|
|
### Definere baselines
|
|
|
|
LLM-applikasjoner har annen performance-profil enn tradisjonelle APIs:
|
|
|
|
| Metric | Baseline (gpt-4o) | Baseline (gpt-4o-mini) |
|
|
|--------|-------------------|------------------------|
|
|
| **Latency P50** | 1.2s | 0.6s |
|
|
| **Latency P95** | 3.5s | 1.8s |
|
|
| **Tokens/sec (output)** | 40-60 | 80-120 |
|
|
| **Time to first token** | 0.3s | 0.2s |
|
|
| **Tool call overhead** | +0.5s per tool | +0.3s per tool |
|
|
|
|
### Custom metrics for LLM performance
|
|
|
|
**Track token usage:**
|
|
|
|
```python
|
|
from opentelemetry import metrics
|
|
|
|
meter = metrics.get_meter(__name__)
|
|
|
|
# Create metrics
|
|
input_tokens = meter.create_counter(
|
|
"gen_ai.input_tokens",
|
|
description="Total input tokens consumed",
|
|
unit="tokens"
|
|
)
|
|
|
|
output_tokens = meter.create_counter(
|
|
"gen_ai.output_tokens",
|
|
description="Total output tokens generated",
|
|
unit="tokens"
|
|
)
|
|
|
|
cost_metric = meter.create_counter(
|
|
"gen_ai.cost_usd",
|
|
description="Estimated cost in USD",
|
|
unit="USD"
|
|
)
|
|
|
|
# Log metrics
|
|
def track_llm_call(model: str, input_tokens: int, output_tokens: int):
|
|
input_tokens.add(input_tokens, {"model": model})
|
|
output_tokens.add(output_tokens, {"model": model})
|
|
|
|
# Calculate cost (example rates)
|
|
cost = (input_tokens * 0.000005) + (output_tokens * 0.000015)
|
|
cost_metric.add(cost, {"model": model})
|
|
```
|
|
|
|
**Query metrics i Application Insights:**
|
|
|
|
```kusto
|
|
AppMetrics
|
|
| where Name == "gen_ai.input_tokens"
|
|
| summarize TotalTokens = sum(Sum) by Model = tostring(Properties.model), bin(TimeGenerated, 1h)
|
|
| render timechart
|
|
```
|
|
|
|
### Latency percentiles
|
|
|
|
**Query latency distribution:**
|
|
|
|
```kusto
|
|
AppDependencies
|
|
| where Target contains "openai.azure.com"
|
|
| extend Model = tostring(Properties["gen_ai.request.model"])
|
|
| summarize
|
|
P50 = percentile(DurationMs, 50),
|
|
P90 = percentile(DurationMs, 90),
|
|
P95 = percentile(DurationMs, 95),
|
|
P99 = percentile(DurationMs, 99),
|
|
Count = count()
|
|
by Model, bin(TimeGenerated, 1h)
|
|
```
|
|
|
|
### Time to first token (TTFT)
|
|
|
|
TTFT er kritisk metric for user experience:
|
|
|
|
```python
|
|
import time
|
|
|
|
def track_streaming_latency(model: str):
|
|
"""Track time to first token for streaming response."""
|
|
with tracer.start_as_current_span("streaming_call") as span:
|
|
start_time = time.time()
|
|
first_token_received = False
|
|
|
|
for chunk in stream_response:
|
|
if not first_token_received:
|
|
ttft = (time.time() - start_time) * 1000 # ms
|
|
span.set_attribute("gen_ai.ttft_ms", ttft)
|
|
first_token_received = True
|
|
```
|
|
|
|
**Query TTFT:**
|
|
|
|
```kusto
|
|
AppTraces
|
|
| where OperationName == "streaming_call"
|
|
| extend TTFT = todouble(Properties.gen_ai_ttft_ms)
|
|
| summarize avg(TTFT), percentile(TTFT, 95) by bin(TimeGenerated, 1h)
|
|
```
|
|
|
|
## Error Tracking og Alerting
|
|
|
|
### LLM-spesifikke errors
|
|
|
|
**Common error patterns:**
|
|
|
|
| Error type | Årsak | Mitigering |
|
|
|------------|-------|------------|
|
|
| **RateLimitError** | 429 Too Many Requests | Implement exponential backoff, øk TPM quota |
|
|
| **ContentFilterError** | Content policy violation | Sanitize prompts, adjust content filter settings |
|
|
| **TimeoutError** | Request > 10min timeout | Chunk inputs, bruk streaming |
|
|
| **TokenLimitExceeded** | Input > model context window | Truncate history, summarize context |
|
|
| **ModelNotFound** | Deployment name feil | Validate deployment names i config |
|
|
|
|
### Exception tracking
|
|
|
|
Application Insights fanger exceptions automatisk, men legg til context:
|
|
|
|
```python
|
|
def safe_llm_call(prompt: str):
|
|
"""LLM call with exception handling."""
|
|
with tracer.start_as_current_span("llm_call") as span:
|
|
try:
|
|
response = client.chat.completions.create(...)
|
|
span.set_attribute("gen_ai.success", True)
|
|
return response
|
|
except RateLimitError as e:
|
|
span.set_status(Status(StatusCode.ERROR, "Rate limit exceeded"))
|
|
span.record_exception(e)
|
|
span.set_attribute("gen_ai.error.type", "rate_limit")
|
|
span.set_attribute("gen_ai.retry_after", e.retry_after)
|
|
raise
|
|
except ContentFilterError as e:
|
|
span.set_status(Status(StatusCode.ERROR, "Content filtered"))
|
|
span.record_exception(e)
|
|
span.set_attribute("gen_ai.error.type", "content_filter")
|
|
span.set_attribute("gen_ai.filter.category", e.category)
|
|
raise
|
|
```
|
|
|
|
**Query error rates:**
|
|
|
|
```kusto
|
|
AppExceptions
|
|
| where Properties.gen_ai_error_type == "rate_limit"
|
|
| summarize ErrorCount = count() by bin(TimeGenerated, 5m)
|
|
| render timechart
|
|
```
|
|
|
|
### Smart alerting for LLM failures
|
|
|
|
**Alert rule examples:**
|
|
|
|
1. **High error rate:**
|
|
```kusto
|
|
AppDependencies
|
|
| where Target contains "openai.azure.com"
|
|
| where Success == false
|
|
| summarize ErrorRate = (count() * 100.0) / todouble(count()) by bin(TimeGenerated, 5m)
|
|
| where ErrorRate > 10 // > 10% errors
|
|
```
|
|
|
|
2. **Latency spike:**
|
|
```kusto
|
|
AppDependencies
|
|
| where Target contains "openai.azure.com"
|
|
| summarize P95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
|
|
| where P95 > 5000 // > 5 seconds P95
|
|
```
|
|
|
|
3. **Cost spike:**
|
|
```kusto
|
|
AppMetrics
|
|
| where Name == "gen_ai.cost_usd"
|
|
| summarize TotalCost = sum(Sum) by bin(TimeGenerated, 1h)
|
|
| where TotalCost > 100 // > $100/hour
|
|
```
|
|
|
|
### Anomaly detection for token usage
|
|
|
|
Application Insights har innebygd anomaly detection:
|
|
|
|
```kusto
|
|
AppMetrics
|
|
| where Name == "gen_ai.input_tokens"
|
|
| make-series TotalTokens = sum(Sum) default=0 on TimeGenerated step 1h
|
|
| extend Anomalies = series_decompose_anomalies(TotalTokens, 1.5)
|
|
| where Anomalies > 0 // Anomali detektert
|
|
```
|
|
|
|
## Framework-integrasjoner
|
|
|
|
### LangChain / LangGraph
|
|
|
|
**Instrumentering:**
|
|
|
|
```python
|
|
from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer
|
|
|
|
azure_tracer = AzureAIOpenTelemetryTracer(
|
|
connection_string=os.environ["APPLICATION_INSIGHTS_CONNECTION_STRING"],
|
|
enable_content_recording=True,
|
|
name="My Agent",
|
|
id="agent_v1"
|
|
)
|
|
|
|
# Attach til LangChain model
|
|
llm = AzureChatOpenAI(..., callbacks=[azure_tracer])
|
|
|
|
# Eller til agent
|
|
agent = create_agent(model=llm, tools=tools, callbacks=[azure_tracer])
|
|
```
|
|
|
|
**Query LangChain traces:**
|
|
|
|
```kusto
|
|
AppTraces
|
|
| where Properties.framework == "langchain"
|
|
| extend ChainName = tostring(Properties.chain_name)
|
|
| summarize Count = count(), AvgDuration = avg(DurationMs) by ChainName
|
|
```
|
|
|
|
### Semantic Kernel
|
|
|
|
Semantic Kernel har native Application Insights støtte:
|
|
|
|
```csharp
|
|
using Microsoft.ApplicationInsights;
|
|
using Microsoft.SemanticKernel;
|
|
|
|
var telemetryClient = new TelemetryClient();
|
|
|
|
var kernel = Kernel.CreateBuilder()
|
|
.AddAzureOpenAIChatCompletion(...)
|
|
.Build();
|
|
|
|
// Tracing er automatisk enabled
|
|
var result = await kernel.InvokePromptAsync("...");
|
|
```
|
|
|
|
### Microsoft Agent Framework
|
|
|
|
Agent Framework sender automatisk telemetri til Application Insights:
|
|
|
|
```python
|
|
from azure.ai.agents.telemetry import AIAgentsInstrumentor
|
|
|
|
# Enable instrumentation
|
|
AIAgentsInstrumentor().instrument()
|
|
|
|
# All agent calls er automatisk tracet
|
|
agent = project_client.agents.create_agent(...)
|
|
```
|
|
|
|
### OpenAI Agents SDK
|
|
|
|
**Instrumentering:**
|
|
|
|
```python
|
|
from opentelemetry.instrumentation.openai_agents import OpenAIAgentsInstrumentor
|
|
|
|
# Instrument SDK
|
|
OpenAIAgentsInstrumentor().instrument(tracer_provider=trace.get_tracer_provider())
|
|
|
|
# All OpenAI agent calls er tracet
|
|
with tracer.start_as_current_span("agent_session"):
|
|
# ... run agent
|
|
pass
|
|
```
|
|
|
|
## Lokal debugging med Aspire Dashboard
|
|
|
|
For lokal utvikling uten Application Insights:
|
|
|
|
**Setup:**
|
|
|
|
```bash
|
|
pip install opentelemetry-exporter-otlp
|
|
```
|
|
|
|
**Code:**
|
|
|
|
```python
|
|
from opentelemetry import trace
|
|
from opentelemetry.sdk.trace import TracerProvider
|
|
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
|
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
|
|
|
|
# Setup OTLP exporter (Aspire Dashboard)
|
|
provider = TracerProvider()
|
|
provider.add_span_processor(
|
|
BatchSpanProcessor(
|
|
OTLPSpanExporter(endpoint="http://localhost:4317")
|
|
)
|
|
)
|
|
trace.set_tracer_provider(provider)
|
|
```
|
|
|
|
**Start Aspire Dashboard:**
|
|
|
|
```bash
|
|
docker run -p 4317:4317 -p 18888:18888 mcr.microsoft.com/dotnet/aspire-dashboard:latest
|
|
```
|
|
|
|
Åpne `http://localhost:18888` for å se traces lokalt.
|
|
|
|
## Best Practices
|
|
|
|
### 1. Service naming
|
|
|
|
Bruk `cloud_RoleName` for å skille tjenester:
|
|
|
|
```python
|
|
from opentelemetry.sdk.resources import Resource
|
|
|
|
resource = Resource.create({
|
|
"service.name": "chat-api",
|
|
"service.version": "1.2.0",
|
|
"deployment.environment": "production"
|
|
})
|
|
|
|
provider = TracerProvider(resource=resource)
|
|
```
|
|
|
|
**Query per service:**
|
|
|
|
```kusto
|
|
AppRequests
|
|
| where AppRoleName == "chat-api"
|
|
```
|
|
|
|
### 2. Redact sensitive data
|
|
|
|
**Ikke logg:**
|
|
- User PII (navn, epost, telefon)
|
|
- API keys eller secrets
|
|
- Sensitive business data
|
|
|
|
**Implementer redaction:**
|
|
|
|
```python
|
|
import re
|
|
|
|
def redact_pii(text: str) -> str:
|
|
"""Redact common PII patterns."""
|
|
# Email
|
|
text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]', text)
|
|
# Phone (US)
|
|
text = re.sub(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', '[PHONE]', text)
|
|
# SSN
|
|
text = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[SSN]', text)
|
|
return text
|
|
|
|
# Bruk før logging
|
|
span.set_attribute("gen_ai.prompt", redact_pii(original_prompt))
|
|
```
|
|
|
|
### 3. Sampling for kostnadsoptimalisering
|
|
|
|
For high-volume applikasjoner, bruk adaptive sampling:
|
|
|
|
```python
|
|
from azure.monitor.opentelemetry import configure_azure_monitor
|
|
|
|
configure_azure_monitor(
|
|
connection_string=connection_string,
|
|
enable_adaptive_sampling=True,
|
|
sampling_rate=0.1 # 10% av requests
|
|
)
|
|
```
|
|
|
|
**Aldri sample:**
|
|
- Errors og exceptions
|
|
- High-value user sessions (premium users)
|
|
- Performance anomalies
|
|
|
|
### 4. Correlation med evaluations
|
|
|
|
Link tracing til offline evaluation runs:
|
|
|
|
```python
|
|
def log_evaluation_result(trace_id: str, metric_name: str, score: float):
|
|
"""Link evaluation score til original trace."""
|
|
with tracer.start_as_current_span("evaluation_result") as span:
|
|
span.set_attribute("evaluation.trace_id", trace_id)
|
|
span.set_attribute("evaluation.metric", metric_name)
|
|
span.set_attribute("evaluation.score", score)
|
|
```
|
|
|
|
**Query:**
|
|
|
|
```kusto
|
|
AppTraces
|
|
| where OperationName == "evaluation_result"
|
|
| join kind=inner (
|
|
AppRequests
|
|
| extend TraceId = OperationId
|
|
) on $left.Properties.evaluation_trace_id == $right.TraceId
|
|
| project TimeGenerated, RequestName, EvaluationScore = todouble(Properties.evaluation_score)
|
|
```
|
|
|
|
### 5. Cost allocation per customer
|
|
|
|
Tag requests med customer ID:
|
|
|
|
```python
|
|
with tracer.start_as_current_span("customer_request") as span:
|
|
span.set_attribute("customer.id", customer_id)
|
|
span.set_attribute("customer.tier", "enterprise")
|
|
```
|
|
|
|
**Query cost per customer:**
|
|
|
|
```kusto
|
|
AppMetrics
|
|
| where Name == "gen_ai.cost_usd"
|
|
| extend CustomerId = tostring(Properties.customer_id)
|
|
| summarize TotalCost = sum(Sum) by CustomerId
|
|
| order by TotalCost desc
|
|
```
|
|
|
|
## Visualisering i Azure Portal
|
|
|
|
### Transaction details view
|
|
|
|
Application Insights **End-to-end transaction details** viser:
|
|
|
|
1. Full trace timeline med alle spans
|
|
2. Dependencies sortert etter latency
|
|
3. Exceptions linket til parent spans
|
|
4. Custom properties per span
|
|
|
|
**Navigasjon:**
|
|
- Application Insights → **Investigate** → **Performance**
|
|
- Velg en request → Klikk **View all telemetry**
|
|
|
|
### Workbooks for LLM monitoring
|
|
|
|
**Pre-built workbook template:**
|
|
|
|
1. Token usage over time (per model)
|
|
2. Cost per hour/day/month
|
|
3. Latency percentiles (P50, P95, P99)
|
|
4. Error rate by error type
|
|
5. Top 10 slowest requests
|
|
6. User feedback distribution
|
|
|
|
**Opprett workbook:**
|
|
- Application Insights → **Monitoring** → **Workbooks** → **+ New**
|
|
|
|
### Dashboards
|
|
|
|
**Key metrics dashboard:**
|
|
|
|
- Total requests (trendline)
|
|
- Average latency (by model)
|
|
- Error rate (%)
|
|
- Total cost (daily/monthly)
|
|
- Token usage (input vs output)
|
|
- Active users
|
|
|
|
## For Cosmo Skyberg: Application Insights for LLM Monitoring
|
|
|
|
**Når anbefalte:**
|
|
|
|
Application Insights er riktig valg for LLM-observabilitet når:
|
|
|
|
1. **Azure-native setup** — Kunden bruker Microsoft Foundry eller Azure OpenAI
|
|
2. **Multi-framework miljø** — LangChain, Semantic Kernel, Agent Framework i samme system
|
|
3. **Enterprise compliance** — Trenger logging i Azure subscription med RBAC
|
|
4. **Cost tracking** — Viktig å korrelere token usage med fakturering
|
|
5. **Eksisterende Azure Monitor** — Allerede bruker App Insights for web APIs
|
|
|
|
**Når vurdere alternativer:**
|
|
|
|
- **LangSmith** — Hvis kun LangChain, og trenger dataset curation
|
|
- **Weights & Biases** — Hvis ML engineering team, trenger experiment tracking
|
|
- **Elastic APM** — Hvis eksisterende Elastic stack for logging
|
|
- **Aspire Dashboard** — Lokal dev/debugging (ikke produksjon)
|
|
|
|
**Key decision factors:**
|
|
|
|
| Faktor | Application Insights | LangSmith | W&B |
|
|
|--------|----------------------|-----------|-----|
|
|
| **OpenTelemetry native** | ✅ Ja | ⚠️ Partial | ❌ Nei |
|
|
| **Cost per GB** | ~$2.76/GB | $0 (gratis tier), $39+ | $0 (gratis tier), $50+ |
|
|
| **Retention** | 90 dager default | 14 dager (gratis), 400 dager (betalt) | Ubegrenset |
|
|
| **Multi-framework** | ✅ Alle | ⚠️ LangChain best | ⚠️ Custom integration |
|
|
| **Azure integration** | ✅ Native | ❌ Nei | ❌ Nei |
|
|
| **Offline evaluation** | ⚠️ Via custom code | ✅ Built-in | ✅ Built-in |
|
|
|
|
**Arkitekturrådgiving:**
|
|
|
|
1. **Start med Application Insights** — Enkleste setup for Azure-kunder
|
|
2. **Enable content recording selektivt** — Kun for debugging, ikke produksjon
|
|
3. **Implementer custom metrics** — Token cost, latency percentiles, TTFT
|
|
4. **Sett opp alerting** — Error rate, cost spikes, latency anomalies
|
|
5. **Kombiner med prompt evaluation** — Microsoft Foundry Evaluation + App Insights tracing
|
|
|
|
**Eksempel-arkitektur:**
|
|
|
|
```
|
|
[User] → [Azure API Management]
|
|
↓ (trace-id propagation)
|
|
[Chat API] → [Application Insights]
|
|
↓
|
|
[Agent Orchestrator]
|
|
├─ [LangChain Agent] → [Azure OpenAI] → [Token metrics]
|
|
├─ [Tool: Azure AI Search] → [Dependency trace]
|
|
└─ [Tool: Bing Search] → [Dependency trace]
|
|
```
|
|
|
|
**Kostnadsestimat:**
|
|
|
|
- **Ingestion:** 1M requests/måned ≈ 10 GB ≈ $27/mnd
|
|
- **Query:** 10 GB scanned/mnd ≈ $0.50/mnd
|
|
- **Retention:** 90 dager default (inkludert i pris)
|
|
- **Total:** ~$30-50/mnd for medium-scale produksjon
|
|
|
|
**Sett opp i 10 minutter:**
|
|
|
|
```bash
|
|
# 1. Connect Application Insights til AI Foundry project
|
|
az monitor app-insights component create \
|
|
--app my-ai-app \
|
|
--location norwayeast \
|
|
--resource-group my-rg
|
|
|
|
# 2. Link til Foundry project (via portal eller CLI)
|
|
# 3. Install SDK + configure
|
|
pip install azure-ai-projects azure-monitor-opentelemetry
|
|
|
|
# 4. Skriv 5 linjer code (se "Instrumentering med Microsoft Foundry SDK")
|
|
# 5. Deploy → Se traces i portal
|
|
```
|
|
|
|
**Første queries å kjøre:**
|
|
|
|
```kusto
|
|
// 1. Top 10 slowest requests
|
|
AppRequests
|
|
| top 10 by DurationMs desc
|
|
| project TimeGenerated, Name, DurationMs, Success
|
|
|
|
// 2. Error distribution
|
|
AppExceptions
|
|
| summarize Count = count() by Type
|
|
| order by Count desc
|
|
|
|
// 3. Cost per hour
|
|
AppMetrics
|
|
| where Name == "gen_ai.cost_usd"
|
|
| summarize Cost = sum(Sum) by bin(TimeGenerated, 1h)
|
|
| render timechart
|
|
```
|
|
|
|
---
|
|
|
|
**Relaterte referanser:**
|
|
- `token-usage-tracking.md` — Token metrics og cost calculation
|
|
- `azure-monitor-integration.md` — Full Azure Monitor setup
|
|
- `llm-performance-baselines.md` — Performance benchmarks per model
|
|
- `distributed-tracing-patterns.md` — Multi-service correlation
|