ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/application-insights-llm-monitoring.md

Application Insights for LLM Monitoring

Kategori: Monitoring & Observability Dato: 2026-02-05 Status: Komplett

Oversikt

Application Insights er Azures native observability-plattform for å overvåke LLM-applikasjoner med OpenTelemetry-kompatibel tracing. Denne guiden dekker LLM-spesifikk telemetri, custom events for AI-interaksjoner, distributed tracing for AI-pipelines, performance baselines, og error tracking.

Application Insights integrerer sømløst med Microsoft Foundry, Azure OpenAI Service, og alle større AI-rammeverk (LangChain, Semantic Kernel, Microsoft Agent Framework, OpenAI Agents SDK).

Hvorfor Application Insights for LLM-applikasjoner?

Utfordringer med LLM-observabilitet

LLM-applikasjoner introduserer unike overvåkingsutfordringer:

1. Komplekse kjeder — Agent kan kjøre 10+ steg med nøstede tool calls
2. Varierende flows — Execution path avhenger av user input og model reasoning
3. Lange inputs/outputs — Prompts og responses kan være 1000+ tokens
4. Multi-agent orchestration — Koordinering mellom flere agenter og tools
5. Kostnadskontroll — Token usage og latency må spores per operasjon

Application Insights løsning

• OpenTelemetry-standard — Følger GenAI semantic conventions
• Full trace tree — Se nøstede spans for agent → tool → LLM calls
• Token tracking — Custom metrics for input/output tokens og cost
• Performance baselines — P50, P90, P95 latency per operasjon
• Error correlation — Link exceptions til spesifikk LLM call eller tool
• Multi-framework — Samme backend for LangChain, Semantic Kernel, etc.

LLM-spesifikk telemetri i Application Insights

Telemetri-typer for AI-applikasjoner

Application Insights lagrer LLM-telemetri i standardtabeller:

Telemetri	Tabell	Bruk for LLM-applikasjoner
Request	AppRequests	HTTP request til AI endpoint (chat completion, agent run)
Dependency	AppDependencies	Kall til Azure OpenAI, embeddings API, vector database
Trace	AppTraces	Agent reasoning steps, tool outputs, system messages
Exception	AppExceptions	Model errors (rate limit, content filter), tool failures
Custom Event	AppEvents	User feedback, agent decisions, evaluation results
Custom Metric	AppMetrics	Token count, cost per request, embedding dimensions

OpenTelemetry Spans for GenAI

Application Insights støtter OpenTelemetry Semantic Conventions for GenAI:

Standard span-attributter:

{
  "gen_ai.system": "azure_openai",
  "gen_ai.request.model": "gpt-4o",
  "gen_ai.request.max_tokens": 1000,
  "gen_ai.request.temperature": 0.7,
  "gen_ai.response.finish_reason": "stop",
  "gen_ai.usage.input_tokens": 450,
  "gen_ai.usage.output_tokens": 320,
  "gen_ai.prompt": "[redacted]",  // hvis content recording er enabled
  "gen_ai.completion": "[redacted]"
}

Multi-agent spans (Microsoft-utvidelse):

Span type	Attributt	Beskrivelse
execute_task	—	Task planning og event propagation
invoke_agent	agent.name, agent.id	Agent invocation
agent_to_agent_interaction	source_agent, target_agent	Inter-agent kommunikasjon
agent.state.management	memory_type	Memory og context management
agent_planning	plan_steps	Agent's internal planning
execute_tool	tool.name, tool.call.arguments, tool.call.results	Tool execution
gen_ai.evaluation	evaluation.name, evaluation.score	Agent performance evaluation

Custom Events for AI-interaksjoner

Logg user feedback

User feedback er kritisk for å evaluere LLM-output kvalitet:

from opentelemetry import trace
from opentelemetry.trace import Status, StatusCode

tracer = trace.get_tracer(__name__)

def log_user_feedback(response_id: str, rating: int, comment: str):
    """Log user feedback som OpenTelemetry event."""
    with tracer.start_as_current_span("user_feedback") as span:
        span.set_attribute("gen_ai.response.id", response_id)
        span.set_attribute("user_feedback.rating", rating)
        span.set_attribute("user_feedback.comment", comment)

        # Link til parent span (LLM response)
        span.set_attribute("parent_span_id", get_response_span_id(response_id))

Query i Application Insights:

AppTraces
| where OperationName == "user_feedback"
| extend Rating = tolong(Properties.user_feedback_rating)
| summarize AvgRating = avg(Rating), Count = count() by bin(TimeGenerated, 1h)

Agent decisions som events

Logg agent decisions for å forstå reasoning:

def log_agent_decision(agent_name: str, decision: str, reasoning: str):
    """Log agent decision point."""
    with tracer.start_as_current_span("agent_decision") as span:
        span.set_attribute("agent.name", agent_name)
        span.set_attribute("agent.decision", decision)
        span.set_attribute("agent.reasoning", reasoning)
        span.set_attribute("timestamp", datetime.utcnow().isoformat())

Tool invocation tracking

Track tool usage patterns:

def track_tool_usage(tool_name: str, success: bool, latency_ms: float):
    """Track tool execution metrics."""
    with tracer.start_as_current_span(f"tool_{tool_name}") as span:
        span.set_attribute("tool.name", tool_name)
        span.set_attribute("tool.success", success)
        span.set_attribute("tool.latency_ms", latency_ms)

        if not success:
            span.set_status(Status(StatusCode.ERROR))

Distributed Tracing for AI Pipelines

Trace hele agent execution

Application Insights viser full trace tree for agent execution:

[Request] POST /chat/completions (2.3s)
├─ [Span] agent_session (2.2s)
│  ├─ [Span] agent_planning (0.1s)
│  ├─ [Span] execute_task (2.0s)
│  │  ├─ [Span] invoke_agent: ResearchAgent (1.2s)
│  │  │  ├─ [Dependency] Azure OpenAI gpt-4o (0.8s)
│  │  │  └─ [Span] execute_tool: web_search (0.4s)
│  │  │     └─ [Dependency] Bing Search API (0.35s)
│  │  └─ [Span] invoke_agent: SummaryAgent (0.7s)
│  │     └─ [Dependency] Azure OpenAI gpt-4o-mini (0.6s)
│  └─ [Span] agent.state.management (0.1s)
└─ [Custom Event] user_feedback (rating: 5)

Correlation IDs

Application Insights bruker W3C Trace Context for correlation:

• operation_Id — Unique ID for hele request (trace-id)
• operation_ParentId — Parent span ID (parent-id)
• id — Current span ID

Query relaterte spans:

AppRequests
| where OperationId == "abc123..."
| union (AppDependencies | where OperationId == "abc123...")
| union (AppTraces | where OperationId == "abc123...")
| order by TimeGenerated asc

Instrumentering med Microsoft Foundry SDK

Python setup:

from azure.ai.projects import AIProjectClient
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry import trace

# Connect til AI Foundry project
project_client = AIProjectClient(
    credential=DefaultAzureCredential(),
    endpoint=os.environ["PROJECT_ENDPOINT"]
)

# Hent Application Insights connection string
connection_string = project_client.telemetry.get_application_insights_connection_string()

# Enable Azure Monitor tracing
configure_azure_monitor(connection_string=connection_string)

# Get tracer
tracer = trace.get_tracer(__name__)

# Trace agent execution
with tracer.start_as_current_span("my_agent_flow"):
    agent = project_client.agents.create_agent(...)
    run = project_client.agents.runs.create_and_process(...)

Content recording (opt-in)

For å trace prompt/completion content (kan inneholde persondata):

import os
os.environ["AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED"] = "true"

Alternativt via miljøvariabel:

# PowerShell
$env:AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED = "true"

# Bash
export AZURE_TRACING_GEN_AI_CONTENT_RECORDING_ENABLED=true

⚠️ Sikkerhet: Content recording logger prompts og responses. Vurder GDPR/privacy før aktivering.

Performance Baselines for LLMs

Definere baselines

LLM-applikasjoner har annen performance-profil enn tradisjonelle APIs:

Metric	Baseline (gpt-4o)	Baseline (gpt-4o-mini)
Latency P50	1.2s	0.6s
Latency P95	3.5s	1.8s
Tokens/sec (output)	40-60	80-120
Time to first token	0.3s	0.2s
Tool call overhead	+0.5s per tool	+0.3s per tool

Custom metrics for LLM performance

Track token usage:

from opentelemetry import metrics

meter = metrics.get_meter(__name__)

# Create metrics
input_tokens = meter.create_counter(
    "gen_ai.input_tokens",
    description="Total input tokens consumed",
    unit="tokens"
)

output_tokens = meter.create_counter(
    "gen_ai.output_tokens",
    description="Total output tokens generated",
    unit="tokens"
)

cost_metric = meter.create_counter(
    "gen_ai.cost_usd",
    description="Estimated cost in USD",
    unit="USD"
)

# Log metrics
def track_llm_call(model: str, input_tokens: int, output_tokens: int):
    input_tokens.add(input_tokens, {"model": model})
    output_tokens.add(output_tokens, {"model": model})

    # Calculate cost (example rates)
    cost = (input_tokens * 0.000005) + (output_tokens * 0.000015)
    cost_metric.add(cost, {"model": model})

Query metrics i Application Insights:

AppMetrics
| where Name == "gen_ai.input_tokens"
| summarize TotalTokens = sum(Sum) by Model = tostring(Properties.model), bin(TimeGenerated, 1h)
| render timechart

Latency percentiles

Query latency distribution:

AppDependencies
| where Target contains "openai.azure.com"
| extend Model = tostring(Properties["gen_ai.request.model"])
| summarize
    P50 = percentile(DurationMs, 50),
    P90 = percentile(DurationMs, 90),
    P95 = percentile(DurationMs, 95),
    P99 = percentile(DurationMs, 99),
    Count = count()
  by Model, bin(TimeGenerated, 1h)

Time to first token (TTFT)

TTFT er kritisk metric for user experience:

import time

def track_streaming_latency(model: str):
    """Track time to first token for streaming response."""
    with tracer.start_as_current_span("streaming_call") as span:
        start_time = time.time()
        first_token_received = False

        for chunk in stream_response:
            if not first_token_received:
                ttft = (time.time() - start_time) * 1000  # ms
                span.set_attribute("gen_ai.ttft_ms", ttft)
                first_token_received = True

Query TTFT:

AppTraces
| where OperationName == "streaming_call"
| extend TTFT = todouble(Properties.gen_ai_ttft_ms)
| summarize avg(TTFT), percentile(TTFT, 95) by bin(TimeGenerated, 1h)

Error Tracking og Alerting

LLM-spesifikke errors

Common error patterns:

Error type	Årsak	Mitigering
RateLimitError	429 Too Many Requests	Implement exponential backoff, øk TPM quota
ContentFilterError	Content policy violation	Sanitize prompts, adjust content filter settings
TimeoutError	Request > 10min timeout	Chunk inputs, bruk streaming
TokenLimitExceeded	Input > model context window	Truncate history, summarize context
ModelNotFound	Deployment name feil	Validate deployment names i config

Exception tracking

Application Insights fanger exceptions automatisk, men legg til context:

def safe_llm_call(prompt: str):
    """LLM call with exception handling."""
    with tracer.start_as_current_span("llm_call") as span:
        try:
            response = client.chat.completions.create(...)
            span.set_attribute("gen_ai.success", True)
            return response
        except RateLimitError as e:
            span.set_status(Status(StatusCode.ERROR, "Rate limit exceeded"))
            span.record_exception(e)
            span.set_attribute("gen_ai.error.type", "rate_limit")
            span.set_attribute("gen_ai.retry_after", e.retry_after)
            raise
        except ContentFilterError as e:
            span.set_status(Status(StatusCode.ERROR, "Content filtered"))
            span.record_exception(e)
            span.set_attribute("gen_ai.error.type", "content_filter")
            span.set_attribute("gen_ai.filter.category", e.category)
            raise

Query error rates:

AppExceptions
| where Properties.gen_ai_error_type == "rate_limit"
| summarize ErrorCount = count() by bin(TimeGenerated, 5m)
| render timechart

Smart alerting for LLM failures

Alert rule examples:

1. High error rate:

   AppDependencies
   | where Target contains "openai.azure.com"
   | where Success == false
   | summarize ErrorRate = (count() * 100.0) / todouble(count()) by bin(TimeGenerated, 5m)
   | where ErrorRate > 10  // > 10% errors
   

2. Latency spike:

   AppDependencies
   | where Target contains "openai.azure.com"
   | summarize P95 = percentile(DurationMs, 95) by bin(TimeGenerated, 5m)
   | where P95 > 5000  // > 5 seconds P95
   

3. Cost spike:

   AppMetrics
   | where Name == "gen_ai.cost_usd"
   | summarize TotalCost = sum(Sum) by bin(TimeGenerated, 1h)
   | where TotalCost > 100  // > $100/hour
   

Anomaly detection for token usage

Application Insights har innebygd anomaly detection:

AppMetrics
| where Name == "gen_ai.input_tokens"
| make-series TotalTokens = sum(Sum) default=0 on TimeGenerated step 1h
| extend Anomalies = series_decompose_anomalies(TotalTokens, 1.5)
| where Anomalies > 0  // Anomali detektert

Framework-integrasjoner

LangChain / LangGraph

Instrumentering:

from langchain_azure_ai.callbacks.tracers import AzureAIOpenTelemetryTracer

azure_tracer = AzureAIOpenTelemetryTracer(
    connection_string=os.environ["APPLICATION_INSIGHTS_CONNECTION_STRING"],
    enable_content_recording=True,
    name="My Agent",
    id="agent_v1"
)

# Attach til LangChain model
llm = AzureChatOpenAI(..., callbacks=[azure_tracer])

# Eller til agent
agent = create_agent(model=llm, tools=tools, callbacks=[azure_tracer])

Query LangChain traces:

AppTraces
| where Properties.framework == "langchain"
| extend ChainName = tostring(Properties.chain_name)
| summarize Count = count(), AvgDuration = avg(DurationMs) by ChainName

Semantic Kernel

Semantic Kernel har native Application Insights støtte:

using Microsoft.ApplicationInsights;
using Microsoft.SemanticKernel;

var telemetryClient = new TelemetryClient();

var kernel = Kernel.CreateBuilder()
    .AddAzureOpenAIChatCompletion(...)
    .Build();

// Tracing er automatisk enabled
var result = await kernel.InvokePromptAsync("...");

Microsoft Agent Framework

Agent Framework sender automatisk telemetri til Application Insights:

from azure.ai.agents.telemetry import AIAgentsInstrumentor

# Enable instrumentation
AIAgentsInstrumentor().instrument()

# All agent calls er automatisk tracet
agent = project_client.agents.create_agent(...)

OpenAI Agents SDK

Instrumentering:

from opentelemetry.instrumentation.openai_agents import OpenAIAgentsInstrumentor

# Instrument SDK
OpenAIAgentsInstrumentor().instrument(tracer_provider=trace.get_tracer_provider())

# All OpenAI agent calls er tracet
with tracer.start_as_current_span("agent_session"):
    # ... run agent
    pass

Lokal debugging med Aspire Dashboard

For lokal utvikling uten Application Insights:

Setup:

pip install opentelemetry-exporter-otlp

Code:

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Setup OTLP exporter (Aspire Dashboard)
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://localhost:4317")
    )
)
trace.set_tracer_provider(provider)

Start Aspire Dashboard:

docker run -p 4317:4317 -p 18888:18888 mcr.microsoft.com/dotnet/aspire-dashboard:latest

Åpne http://localhost:18888 for å se traces lokalt.

Best Practices

1. Service naming

Bruk cloud_RoleName for å skille tjenester:

from opentelemetry.sdk.resources import Resource

resource = Resource.create({
    "service.name": "chat-api",
    "service.version": "1.2.0",
    "deployment.environment": "production"
})

provider = TracerProvider(resource=resource)

Query per service:

AppRequests
| where AppRoleName == "chat-api"

2. Redact sensitive data

Ikke logg:

• User PII (navn, epost, telefon)
• API keys eller secrets
• Sensitive business data

Implementer redaction:

import re

def redact_pii(text: str) -> str:
    """Redact common PII patterns."""
    # Email
    text = re.sub(r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', '[EMAIL]', text)
    # Phone (US)
    text = re.sub(r'\b\d{3}[-.]?\d{3}[-.]?\d{4}\b', '[PHONE]', text)
    # SSN
    text = re.sub(r'\b\d{3}-\d{2}-\d{4}\b', '[SSN]', text)
    return text

# Bruk før logging
span.set_attribute("gen_ai.prompt", redact_pii(original_prompt))

3. Sampling for kostnadsoptimalisering

For high-volume applikasjoner, bruk adaptive sampling:

from azure.monitor.opentelemetry import configure_azure_monitor

configure_azure_monitor(
    connection_string=connection_string,
    enable_adaptive_sampling=True,
    sampling_rate=0.1  # 10% av requests
)

Aldri sample:

• Errors og exceptions
• High-value user sessions (premium users)
• Performance anomalies

4. Correlation med evaluations

Link tracing til offline evaluation runs:

def log_evaluation_result(trace_id: str, metric_name: str, score: float):
    """Link evaluation score til original trace."""
    with tracer.start_as_current_span("evaluation_result") as span:
        span.set_attribute("evaluation.trace_id", trace_id)
        span.set_attribute("evaluation.metric", metric_name)
        span.set_attribute("evaluation.score", score)

Query:

AppTraces
| where OperationName == "evaluation_result"
| join kind=inner (
    AppRequests
    | extend TraceId = OperationId
) on $left.Properties.evaluation_trace_id == $right.TraceId
| project TimeGenerated, RequestName, EvaluationScore = todouble(Properties.evaluation_score)

5. Cost allocation per customer

Tag requests med customer ID:

with tracer.start_as_current_span("customer_request") as span:
    span.set_attribute("customer.id", customer_id)
    span.set_attribute("customer.tier", "enterprise")

Query cost per customer:

AppMetrics
| where Name == "gen_ai.cost_usd"
| extend CustomerId = tostring(Properties.customer_id)
| summarize TotalCost = sum(Sum) by CustomerId
| order by TotalCost desc

Visualisering i Azure Portal

Transaction details view

Application Insights End-to-end transaction details viser:

1. Full trace timeline med alle spans
2. Dependencies sortert etter latency
3. Exceptions linket til parent spans
4. Custom properties per span

Navigasjon:

• Application Insights → Investigate → Performance
• Velg en request → Klikk View all telemetry

Workbooks for LLM monitoring

Pre-built workbook template:

1. Token usage over time (per model)
2. Cost per hour/day/month
3. Latency percentiles (P50, P95, P99)
4. Error rate by error type
5. Top 10 slowest requests
6. User feedback distribution

Opprett workbook:

• Application Insights → Monitoring → Workbooks → + New

Dashboards

Key metrics dashboard:

• Total requests (trendline)
• Average latency (by model)
• Error rate (%)
• Total cost (daily/monthly)
• Token usage (input vs output)
• Active users

For Cosmo Skyberg: Application Insights for LLM Monitoring

Når anbefalte:

Application Insights er riktig valg for LLM-observabilitet når:

1. Azure-native setup — Kunden bruker Microsoft Foundry eller Azure OpenAI
2. Multi-framework miljø — LangChain, Semantic Kernel, Agent Framework i samme system
3. Enterprise compliance — Trenger logging i Azure subscription med RBAC
4. Cost tracking — Viktig å korrelere token usage med fakturering
5. Eksisterende Azure Monitor — Allerede bruker App Insights for web APIs

Når vurdere alternativer:

• LangSmith — Hvis kun LangChain, og trenger dataset curation
• Weights & Biases — Hvis ML engineering team, trenger experiment tracking
• Elastic APM — Hvis eksisterende Elastic stack for logging
• Aspire Dashboard — Lokal dev/debugging (ikke produksjon)

Key decision factors:

Faktor	Application Insights	LangSmith	W&B
OpenTelemetry native	✅ Ja	⚠️ Partial	❌ Nei
Cost per GB	~$2.76/GB	$0 (gratis tier), $39+	$0 (gratis tier), $50+
Retention	90 dager default	14 dager (gratis), 400 dager (betalt)	Ubegrenset
Multi-framework	✅ Alle	⚠️ LangChain best	⚠️ Custom integration
Azure integration	✅ Native	❌ Nei	❌ Nei
Offline evaluation	⚠️ Via custom code	✅ Built-in	✅ Built-in

Arkitekturrådgiving:

1. Start med Application Insights — Enkleste setup for Azure-kunder
2. Enable content recording selektivt — Kun for debugging, ikke produksjon
3. Implementer custom metrics — Token cost, latency percentiles, TTFT
4. Sett opp alerting — Error rate, cost spikes, latency anomalies
5. Kombiner med prompt evaluation — Microsoft Foundry Evaluation + App Insights tracing

Eksempel-arkitektur:

[User] → [Azure API Management]
           ↓ (trace-id propagation)
       [Chat API] → [Application Insights]
           ↓
       [Agent Orchestrator]
           ├─ [LangChain Agent] → [Azure OpenAI] → [Token metrics]
           ├─ [Tool: Azure AI Search] → [Dependency trace]
           └─ [Tool: Bing Search] → [Dependency trace]

Kostnadsestimat:

• Ingestion: 1M requests/måned ≈ 10 GB ≈ $27/mnd
• Query: 10 GB scanned/mnd ≈ $0.50/mnd
• Retention: 90 dager default (inkludert i pris)
• Total: ~$30-50/mnd for medium-scale produksjon

Sett opp i 10 minutter:

# 1. Connect Application Insights til AI Foundry project
az monitor app-insights component create \
  --app my-ai-app \
  --location norwayeast \
  --resource-group my-rg

# 2. Link til Foundry project (via portal eller CLI)
# 3. Install SDK + configure
pip install azure-ai-projects azure-monitor-opentelemetry

# 4. Skriv 5 linjer code (se "Instrumentering med Microsoft Foundry SDK")
# 5. Deploy → Se traces i portal

Første queries å kjøre:

// 1. Top 10 slowest requests
AppRequests
| top 10 by DurationMs desc
| project TimeGenerated, Name, DurationMs, Success

// 2. Error distribution
AppExceptions
| summarize Count = count() by Type
| order by Count desc

// 3. Cost per hour
AppMetrics
| where Name == "gen_ai.cost_usd"
| summarize Cost = sum(Sum) by bin(TimeGenerated, 1h)
| render timechart

---

Relaterte referanser:

• token-usage-tracking.md — Token metrics og cost calculation
• azure-monitor-integration.md — Full Azure Monitor setup
• llm-performance-baselines.md — Performance benchmarks per model
• distributed-tracing-patterns.md — Multi-service correlation