ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/token-usage-tracking-attribution.md

Token Usage Tracking and Attribution

Kategori: Monitoring & Observability Dato: 2026-04-09 Versjon: 1.0

Introduksjon

Token usage tracking og cost attribution er kritiske kapabiliteter for å styre kostnader, implementere chargeback-modeller, og optimalisere ressursbruk i Microsoft AI-løsninger. Denne referansen dekker teknikker for nøyaktig token-måling, brukerattribuering, og kostnadsrapportering.

Token Counting og Logging

Basis Token Tracking

Azure OpenAI API returnerer token usage i response-objektet:

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Your prompt"}]
)

# Token-data fra response
input_tokens = response.usage.prompt_tokens
output_tokens = response.usage.completion_tokens
total_tokens = response.usage.total_tokens

Viktig: Token-telling varierer per modell og er basert på modell-spesifikk tokenizer (GPT-2 tokenizer som baseline).

Token Estimering (Pre-call)

For å estimere tokens før API-kall:

import tiktoken

class TokenEstimator(object):
    GPT2_TOKENIZER = tiktoken.get_encoding("gpt2")

    def estimate_tokens(self, text: str) -> int:
        return len(self.GPT2_TOKENIZER.encode(text))

# Bruk
estimator = TokenEstimator()
token_count = estimator.estimate_tokens(input_text)

Bruksområder:

• Pre-validering mot rate limits
• Kostnadsestimering før kall
• Budsjett-gating i applikasjoner

Azure Monitor Platform Metrics

Azure OpenAI samler automatisk token-baserte metrics:

Tilgjengelige metrics:

• TokenTransaction — Total token count (input + output)
• PromptTokens — Input tokens
• CompletionTokens — Output tokens
• ProcessedPromptTokens — Tokens faktisk prosessert (kan avvike ved caching)

Aksess via:

• Azure Portal → Azure OpenAI resource → Metrics
• Azure Monitor Metrics Explorer
• REST API (/metrics endpoint)

Dashboards: Azure OpenAI tilbyr out-of-box dashboards med "Tokens-Based Usage" kategori som viser:

• Token consumption over tid
• Breakdown per modell
• Comparison mot quota limits

Usage Attribution per Applikasjon/Bruker

Utfordring: Native Telemetri-begrensninger

Problem: Azure OpenAI logger IP-adresse med siste oktet masket (f.eks. 192.168.1.xxx), noe som gjør det vanskelig å knytte token-bruk til spesifikk applikasjon eller business unit.

Løsning: Introduser gateway-pattern for fullstendig attributering.

Gateway-basert Attribution (Azure API Management)

Arkitektur:

Client → API Management Gateway → Azure OpenAI
  ↓
  Token usage logged med:
  - Client IP (full adresse)
  - Microsoft Entra ID identity
  - Custom business unit/app identifier

Fordeler:

1. Fullstendig IP-adresse — Identifiser klient-applikasjon
2. Identity-data — Entra ID user/app principal
3. Custom metadata — Business unit, cost center, tenant ID
4. Sentralisert logging — Aggreger data fra multiple Azure OpenAI instances

Kusto Query for Usage Monitoring (APIM):

ApiManagementGatewayLogs
| where tolower(OperationId) in ('completions_create','chatcompletions_create')
| extend model = tostring(parse_json(BackendResponseBody)['model'])
| extend prompttokens = parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens']
| extend completiontokens = parse_json(parse_json(BackendResponseBody)['usage'])['completion_tokens']
| extend totaltokens = parse_json(parse_json(BackendResponseBody)['usage'])['total_tokens']
| extend ip = CallerIpAddress
| summarize
    sum(todecimal(prompttokens)),
    sum(todecimal(completiontokens)),
    sum(todecimal(totaltokens)),
    avg(todecimal(totaltokens))
    by ip, model

Output: Tabell med IP, model, sum(prompt tokens), sum(completion tokens), sum(total tokens).

Application Insights Telemetry Enrichment

For applikasjoner uten gateway, bruk Application Insights med custom telemetry:

import logging
from azure.monitor.opentelemetry import configure_azure_monitor

# Sett opp Application Insights
configure_azure_monitor()

logger = logging.getLogger(__name__)

def log_token_usage(response, user_id, business_unit):
    usage = response.usage

    # Log med custom properties for attribution
    logger.info(
        "Token usage",
        extra={
            "custom_dimensions": {
                "user_id": user_id,
                "business_unit": business_unit,
                "model": response.model,
                "prompt_tokens": usage.prompt_tokens,
                "completion_tokens": usage.completion_tokens,
                "total_tokens": usage.total_tokens
            }
        }
    )

Advarsel: Application Insights bruker sampling i high-volume scenarios, noe som ikke er egnet for nøyaktig billing/metering. For billing-data, bruk dedikert data store (Event Hubs + Stream Analytics).

Resource Tags for Attribution

For deployment-level attribution:

# Tag Azure OpenAI resource
az openai account update \
  --name myopenai \
  --resource-group myrg \
  --tags CostCenter=Finance AppName=ChatBot Environment=Prod

Bruk i Azure Cost Management: Filtrer kostnadsanalyse per tag for å allokere Azure-kostnader til business units.

Begrensning: Dette gir deployment-level attribution, ikke per-request granularitet.

Budget Monitoring og Alerts

Azure Monitor Budget Alerts

Oppsett:

1. Opprett diagnostic setting for Azure OpenAI resource:

   • Send metrics til Log Analytics workspace
   • Velg AllMetrics kategori

2. Sett opp metric alert på token usage:

   Metric: ProcessedPromptTokens
   Aggregation: Sum
   Threshold: 1000000 (1M tokens)
   Period: 1 hour
   Action: Send email / webhook
   

3. Opprett budget i Cost Management:

   • Scope: Azure OpenAI resource eller subscription
   • Budget amount: NOK 10,000/måned
   • Alert thresholds: 50%, 80%, 100%, 120%

Kusto query for budget monitoring:

AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "TokenTransaction"
| summarize TotalTokens = sum(Total) by bin(TimeGenerated, 1h), Resource
| extend EstimatedCost = TotalTokens * 0.0001 // Eksempel: $0.0001 per token
| project TimeGenerated, Resource, TotalTokens, EstimatedCost

Programmatic Budget Enforcement

API-level rate limiting:

import logging

logger = logging.getLogger(__name__)

# Konfigurasjon
MONTHLY_TOKEN_BUDGET = 10_000_000
DAILY_TOKEN_BUDGET = 500_000
ITPM_LIMIT = 100_000  # Input tokens per minute
OTPM_LIMIT = 50_000   # Output tokens per minute

def log_token_usage(response, current_usage):
    usage = response.usage

    # Log current usage
    logger.info(f"Input tokens: {usage.prompt_tokens}")
    logger.info(f"Output tokens: {usage.completion_tokens}")
    logger.info(f"Total tokens: {usage.total_tokens}")

    # Check against limits
    if usage.prompt_tokens > ITPM_LIMIT * 0.8:
        logger.warning("Approaching ITPM limit")

    if usage.completion_tokens > OTPM_LIMIT * 0.8:
        logger.warning("Approaching OTPM limit")

    # Budget enforcement
    new_usage = current_usage + usage.total_tokens
    if new_usage > DAILY_TOKEN_BUDGET:
        raise Exception("Daily token budget exceeded")

    return new_usage

Best practice: Kombiner soft limits (warnings) med hard limits (enforcement) for å balansere reliability og cost control.

Token Efficiency Metrics

Nøkkel-metrikker for Optimalisering

1. Token-to-response ratio average_tokens_per_request = total_tokens / request_count

2. Input/output ratio io_ratio = completion_tokens / prompt_tokens Høy ratio = efficient prompt design

3. Cost per request cost_per_request = (prompt_tokens * input_price + completion_tokens * output_price) / 1000

4. Tokens per user session session_tokens = sum(tokens) GROUP BY session_id

5. Prompt efficiency score efficiency = output_quality_score / total_tokens

Kusto Query for Efficiency Analysis

AzureDiagnostics
| where Category == "RequestResponse"
| extend model = tostring(parse_json(properties_s)['model'])
| extend prompt_tokens = toint(parse_json(properties_s)['usage']['prompt_tokens'])
| extend completion_tokens = toint(parse_json(properties_s)['usage']['completion_tokens'])
| extend total_tokens = toint(parse_json(properties_s)['usage']['total_tokens'])
| summarize
    AvgPromptTokens = avg(prompt_tokens),
    AvgCompletionTokens = avg(completion_tokens),
    AvgTotalTokens = avg(total_tokens),
    IOEfficiency = avg(todecimal(completion_tokens) / todecimal(prompt_tokens)),
    RequestCount = count()
    by model, bin(TimeGenerated, 1d)
| project TimeGenerated, model, AvgPromptTokens, AvgCompletionTokens, IOEfficiency, RequestCount

Chargeback Reporting

Chargeback Model Components

1. Data Collection:

• Gateway logs (APIM) eller Application Insights
• Token usage per business unit/app
• Model pricing data (per 1K tokens)

2. Cost Calculation:

# Eksempel pricing (GPT-4o, januar 2026)
INPUT_PRICE_PER_1K = 0.005   # USD
OUTPUT_PRICE_PER_1K = 0.015  # USD
NOK_EXCHANGE_RATE = 10.5

def calculate_cost(prompt_tokens, completion_tokens):
    input_cost = (prompt_tokens / 1000) * INPUT_PRICE_PER_1K
    output_cost = (completion_tokens / 1000) * OUTPUT_PRICE_PER_1K
    total_usd = input_cost + output_cost
    total_nok = total_usd * NOK_EXCHANGE_RATE
    return total_nok

3. Attribution Logic:

• Per user: Group by user_id
• Per app: Group by application_name
• Per business unit: Group by cost_center tag

4. Report Generation:

// Monthly chargeback report
ApiManagementGatewayLogs
| where TimeGenerated >= startofmonth(now())
| where tolower(OperationId) in ('completions_create','chatcompletions_create')
| extend business_unit = tostring(parse_json(RequestHeaders)['X-Business-Unit'])
| extend model = tostring(parse_json(BackendResponseBody)['model'])
| extend prompt_tokens = toint(parse_json(parse_json(BackendResponseBody)['usage'])['prompt_tokens'])
| extend completion_tokens = toint(parse_json(parse_json(BackendResponseBody)['usage'])['completion_tokens'])
| summarize
    TotalPromptTokens = sum(prompt_tokens),
    TotalCompletionTokens = sum(completion_tokens),
    RequestCount = count()
    by business_unit, model
| extend InputCostUSD = (TotalPromptTokens / 1000.0) * 0.005
| extend OutputCostUSD = (TotalCompletionTokens / 1000.0) * 0.015
| extend TotalCostUSD = InputCostUSD + OutputCostUSD
| extend TotalCostNOK = TotalCostUSD * 10.5
| project business_unit, model, TotalPromptTokens, TotalCompletionTokens,
          RequestCount, TotalCostUSD, TotalCostNOK
| order by TotalCostNOK desc

Showback vs Chargeback

Aspekt	Showback	Chargeback
Formål	Informasjon og bevisstgjøring	Faktisk fakturering
Nøyaktighet	Estimert (akseptabelt med sampling)	Høy presisjon påkrevd
Data store	Application Insights OK	Event Hubs + dedikert DB
Frekvens	Ukentlig/månedlig report	Real-time tracking
Implementering	Enklere	Mer kompleks

Anbefaling: Start med showback for å bygge kostnadsbevissthet, deretter implementer chargeback når forretningskrav og infrastruktur er på plass.

RAG-spesifikke Considerations

Token Usage i RAG-pipelines

Azure OpenAI On Your Data (RAG) gjør to LLM-kall per brukerforespørsel:

1. Intent Prompt — Reformulering av query til search intents 2. Generation Prompt — Generering av svar basert på retrieved chunks

Token breakdown:

Komponent	Beskrivelse	Token impact
Meta prompt	System instructions (inScope param avhengig)	400-4000 tokens (modell-avhengig)
User question + history	Input fra bruker	Cap: 2000 tokens
Retrieved chunks	Dokumenter fra search (5-10 chunks @ 1024 tokens)	5000-10000 tokens
Intent generation	Output fra første LLM-kall	~25 tokens
Final response	Output fra andre LLM-kall	~110 tokens

Eksempel (gpt-35-turbo-16k):

• Generation prompt: 4297 tokens
• Intent prompt: 1366 tokens
• Response output: 111 tokens
• Intent output: 25 tokens
• Total: ~5800 tokens per spørsmål

Optimaliseringstekniker:

1. Reduser retrieved_document_count (default 5)
2. Juster chunk_size (default 1024)
3. Øk strictness (filtrer irrelevante chunks)
4. Bruk inScope=True for kortere meta prompt

Monitoring RAG Token Usage

// Dedicated query for RAG scenarios
AzureDiagnostics
| where Category == "RequestResponse"
| where properties_s contains "data_sources"  // RAG indicator
| extend prompt_tokens = toint(parse_json(properties_s)['usage']['prompt_tokens'])
| extend completion_tokens = toint(parse_json(properties_s)['usage']['completion_tokens'])
| extend total_tokens = toint(parse_json(properties_s)['usage']['total_tokens'])
| extend retrieved_docs = toint(parse_json(properties_s)['data_sources'][0]['parameters']['top_n_documents'])
| summarize
    AvgPromptTokens = avg(prompt_tokens),
    AvgCompletionTokens = avg(completion_tokens),
    AvgTotalTokens = avg(total_tokens),
    AvgRetrievedDocs = avg(retrieved_docs),
    RequestCount = count()
    by bin(TimeGenerated, 1h)

Fine-tuned Models: Spesialkonsiderasjoner

Tre Kostnadskomponenter

1. Training cost — Per token i training file
2. Hosting cost — Timepris mens deployed (uansett bruk)
3. Inference cost — Per 1000 tokens (input + output)

Kritisk: Fine-tuned modeller akkumulerer hosting cost selv når de ikke brukes. Etter 15 dager inaktivitet slettes deployment automatisk (modellen bevares, kan redeployes).

Best practice:

• Monitor deployment utilization
• Slett unused deployments promptly
• Bruk automation for deployment lifecycle

Tracking Fine-tuning Costs

AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName in ("FineTuningHours", "FineTuningTokens")
| summarize
    TotalTrainingTokens = sumif(Total, MetricName == "FineTuningTokens"),
    TotalHostingHours = sumif(Total, MetricName == "FineTuningHours")
    by Resource, bin(TimeGenerated, 1d)
| extend TrainingCostUSD = TotalTrainingTokens * 0.00008  // Eksempel pricing
| extend HostingCostUSD = TotalHostingHours * 2.0         // Eksempel: $2/hour
| extend TotalCostUSD = TrainingCostUSD + HostingCostUSD

Provisioned Throughput Units (PTU): Tracking

PTU vs. Consumption-based Billing

Billing Model	Token Tracking Approach
Pay-as-you-go	Track individual tokens, calculate variable cost
PTU	Track utilization percentage against reserved capacity

PTU Metrics:

• PTUUtilization — Percentage of reserved capacity used
• ProcessedPromptTokens — Input tokens processed
• Input TPM per PTU — Model-specific (f.eks. 8450 TPM for Llama-3.3-70B)

Cost model:

• Fixed monthly cost for PTU reservation
• Cost per token = (Monthly PTU cost) / (Total tokens processed)

PTU Efficiency Monitoring

AzureMetrics
| where ResourceProvider == "MICROSOFT.COGNITIVESERVICES"
| where MetricName == "PTUUtilization"
| summarize AvgUtilization = avg(Average), MaxUtilization = max(Maximum)
    by Resource, bin(TimeGenerated, 1h)
| extend EfficiencyStatus = case(
    AvgUtilization < 50, "Underutilized",
    AvgUtilization >= 50 and AvgUtilization < 80, "Optimal",
    AvgUtilization >= 80, "Consider scaling"
)

Anbefaling: Kombiner PTU for baseline workload med pay-as-you-go for overflow traffic (via gateway pattern).

Integrasjon med FinOps Practices

FinOps Framework Alignment

1. Inform:

• Real-time dashboards med token usage
• Trend analysis og forecasting
• Anomaly detection på usage spikes

2. Optimize:

• Token efficiency metrics (se "Token Efficiency Metrics")
• Prompt optimization basert på cost/quality ratio
• Model selection guidance (GPT-4o vs. GPT-4o-mini)

3. Operate:

• Automated budget enforcement
• Chargeback/showback reporting
• Cost allocation til business units

Azure Cost Management Integration

1. Tag Strategy:

# Standardized tagging
CostCenter: <code>
BusinessUnit: <name>
Application: <name>
Environment: Prod|Dev|Test
Owner: <email>

2. Cost Analysis Views:

• Filtrer per tag dimension
• Group by resource, subscription, eller custom tag
• Sammenlign faktisk vs. budsjett

3. Budgets:

• Opprett per resource group eller subscription
• Sett alert thresholds (50%, 80%, 100%, 120%)
• Action groups for automated response (webhook, Logic App)

Best Practices

1. Data Store Selection

Use Case	Recommended Store	Rationale
Showback (informasjon)	Application Insights	Enkel, innebygd, sampling OK
Chargeback (fakturering)	Event Hubs + Synapse/Fabric	Høy presisjon, no sampling
Real-time monitoring	Stream Analytics + Power BI	Low latency, streaming dashboards
Long-term audit	Azure Storage (cold tier)	Billig, compliance-friendly

2. Attribution Hierarchy

Prioriter:

1. User-level — Mest granulær, best for interne chargeback
2. Application-level — God for multi-tenant SaaS
3. Business unit-level — Standard for enterprise showback
4. Subscription-level — Minst granulær, enklest å implementere

3. Monitoring Frequency

Metric Type	Collection Frequency	Retention
Real-time alerts	Per request	7 dager
Operational dashboards	1 minutt aggregation	30 dager
Cost reporting	1 time aggregation	1 år
Audit logs	Per request (full fidelity)	7 år (compliance)

4. Gateway Pattern Decision Matrix

Bruk gateway hvis:

• ✅ Multiple clients eller multiple Azure OpenAI instances
• ✅ Chargeback requirement (nøyaktig attribution)
• ✅ Centralized policy enforcement (rate limiting, content filtering)
• ✅ Near real-time monitoring requirement

Unngå gateway hvis:

• ❌ Single client, single Azure OpenAI instance
• ❌ Latency er kritisk (gateway adds ~10-50ms)
• ❌ Simple showback er sufficient

5. Cost Optimization Triggers

Alerts når:

• Token usage øker >20% week-over-week (anomaly)
• Cost per user > baseline + 2 standard deviations
• PTU utilization < 50% (consider downscaling)
• Fine-tuned model har 0 requests i 7 dager (delete deployment)

Relaterte Referanser

• cost-optimization/token-optimization.md — Teknikker for å redusere token consumption
• cost-optimization/ptu-vs-payg.md — Billing model selection
• monitoring-observability/azure-monitor-integration.md — Azure Monitor oppsett
• monitoring-observability/alerting-strategies.md — Alert configuration patterns
• architecture/gateway-patterns.md — API Management for AI workloads
• mlops-genaiops/evaluation-metrics.md — Quality vs. cost trade-offs

Kilder (Microsoft Learn)

1. Monitor Azure OpenAI — Official monitoring guide
2. Implement advanced monitoring for Azure OpenAI in Foundry Models through a gateway (Verified MCP 2026-04) — Gateway patterns for usage tracking. Ny brukscase dokumentert: audit av model inputs/outputs for threat detection og data exfiltration detection. Merk: gateway monitoring kan bli single point of failure — vurder redundans.
3. Plan to manage costs for Azure OpenAI — Cost management strategies
4. Token usage estimation for Azure OpenAI On Your Data — RAG-specific token calculations
5. Understanding costs associated with PTU — PTU billing model
6. Application design for AI workloads — Cost and chargeback scenarios
7. Architecture strategies for cost data — Chargeback vs. showback

For Cosmo

Når du diskuterer token usage tracking og attribution, vektlegg gateway-pattern som game-changer for chargeback-scenarioer. Mange organisasjoner undervurderer betydningen av nøyaktig attribution før de skalerer AI-løsninger til produksjon.

Key talking points:

1. Native Azure OpenAI telemetri har masket IP — ikke sufficient for per-app attribution
2. Gateway (APIM) gir full observability + centralized policy enforcement
3. Forskjellen mellom showback (informasjon) og chargeback (fakturering) krever ulik data fidelity
4. RAG-workloads har 2x token overhead (intent + generation) — må planlegges inn i budsjett
5. Fine-tuned models har hosting cost uavhengig av bruk — krev proaktiv lifecycle management

Hvis løsningen skal brukes av flere business units eller krever intern fakturering, er gateway pattern ikke optional — det er kritisk arkitektur-komponent.

Norsk offentlig sektor-vinkling: For offentlige virksomheter med krav til internprising (eks. NAV, Skatteetaten, fylkeskommuner med delte IT-tjenester), er nøyaktig cost attribution ikke bare best practice — det er governance-krav. Kombiner med Azure Cost Management tags for å oppfylle økonomiregelverket sitt krav til transparent ressursbruk.