ms-ai-architect/skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md

# Observability Patterns for Copilot Extensions and Plugins

**Last updated:** 2026-05
**Status:** GA
**Category:** Monitoring & Observability
**Type:** reference
**Source:** https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Arkitekturmønstre](#arkitekturmønstre)
- [Beslutningsveiledning](#beslutningsveiledning)
- [Integrasjon med Microsoft-stakken](#integrasjon-med-microsoft-stakken)
- [Offentlig sektor (Norge)](#offentlig-sektor-norge)
- [Kostnad og lisensiering](#kostnad-og-lisensiering)
- [For arkitekten (Cosmo)](#for-arkitekten-cosmo)
- [Kilder og verifisering](#kilder-og-verifisering)

## Introduksjon

Når organisasjoner utvider Microsoft Copilot med custom plugins, connectors og extensions, blir observability kritisk for å sikre pålitelighet, ytelse og compliance. I motsetning til standalone applikasjoner opererer Copilot-extensions i et distribuert økosystem hvor telemetri må samles fra flere lag: plugin-kjøretid, API-kall, LLM-interaksjoner og brukeropplevelse.

Microsoft tilbyr et helhetlig observability-rammeverk basert på Azure Application Insights, Copilot Studio analytics og Azure Monitor. Dette gir innsikt i plugin performance, token-forbruk, error rates, user engagement og sikkerhetshendelser. For Copilot Studio-agenter er Application Insights-integrasjon nå en out-of-the-box feature som logger incoming/outgoing messages, topic triggers og custom telemetry events.

Utfordringen ligger i å instrumentere extensions korrekt, definere relevante metrics (både system- og business-metrics), og bygge dashboards som gir actionable insights for både utviklere, data scientists og business stakeholders. I offentlig sektor må observability også dekke compliance-logging for Forvaltningsloven § 11 (journalføring av vedtak) og GDPR Article 35 (DPIA monitoring).

---

## Kjernekomponenter

### Telemetry Layers for Copilot Extensions

| Layer | Data Captured | Tool | Purpose |
|-------|---------------|------|---------|
| **Copilot Studio Agent** | Messages, topics, custom events, design mode | Application Insights | Track agent behavior, conversation flow, topic performance |
| **Plugin/Connector Runtime** | API calls, latency, errors, token usage | Application Insights SDK | Monitor external integrations, debug failures |
| **LLM Interaction** | Prompt tokens, completion tokens, model latency, groundedness | Azure OpenAI metrics | Cost tracking, performance optimization |
| **User Engagement** | Thumbs up/down, edit distance, session duration | Custom events | Measure usefulness, iterate on UX |
| **Security/Compliance** | Filtered prompts, PII detection, audit logs | Microsoft Sentinel, Purview | Governance, risk management |

### Application Insights Integration for Copilot Studio

**Configuration Steps:**
1. Navigate to **Settings → Advanced** in Copilot Studio
2. Add Application Insights **Connection string** (from Azure Portal)
3. Enable optional settings:
   - **Log activities**: Incoming/outgoing messages and events
   - **Log sensitive Activity properties**: userid, name, text, speak (vurder GDPR-implikasjoner)
   - **Log custom telemetry events**: Via "Log custom telemetry event" node in topics

**Custom Dimensions (customDimensions field):**

| Field | Description | Sample Values |
|-------|-------------|---------------|
| `type` | Activity type | `message`, `conversationUpdate`, `event`, `invoke` |
| `channelId` | Channel identifier | `emulator`, `directline`, `msteams`, `webchat` |
| `designMode` | Test canvas vs. production | `True` / `False` |
| `locale` | User locale | `en-us`, `nb-no`, `sv-se` |
| `text` | Message text (if logging enabled) | User prompt/agent response |

### Pre-Built Dashboards

**Copilot Studio Workbook (Preview)** – Tilgjengelig i Application Insights:
- **Path:** Application Insights → Monitoring → Workbooks → "Copilot Studio Dashboard"
- **Metrics:** Total conversations, latency, exceptions, tool usage, topic analytics
- **Customization:** Edit mode for adding KQL queries (e.g., track custom attributes)

---

## Arkitekturmønstre

### Pattern 1: Centralized Telemetry Hub

**Bruk:** Enterprise med mange Copilot-extensions på tvers av teams.

**Arkitektur:**
```
Copilot Studio Agent(s) → Application Insights (Workspace 1)
                      ↓
M365 Copilot Plugin(s) → Application Insights (Workspace 2) → Azure Workbook (Consolidated)
                      ↓
Power Platform Connector(s) → Application Insights (Workspace 3)
                      ↓
Microsoft Sentinel (Audit Logs via Purview)
```

**Fordeler:**
- Felles sikkerhetspolicy og RBAC (Reader role for team members)
- Cross-correlation av events på tvers av extensions
- Compliance-logging aggregert i Sentinel

**Ulemper:**
- Krever Application Insights API-tilgang for cross-workspace queries
- Høyere kostnad ved separate workspaces (vurder single workspace hvis <500GB/month)

---

### Pattern 2: Plugin-Specific Instrumentation

**Bruk:** Custom plugin/connector utviklet med pro-code (C#, TypeScript).

**Implementering:**
```csharp
// C# example - Application Insights SDK
using Microsoft.ApplicationInsights;
using Microsoft.ApplicationInsights.DataContracts;

var telemetryClient = new TelemetryClient();
telemetryClient.Context.GlobalProperties["PluginName"] = "SalesforceConnector";

// Track plugin execution
var stopwatch = Stopwatch.StartNew();
try
{
    var result = await ExecutePluginAsync(request);
    telemetryClient.TrackEvent("PluginSuccess", new Dictionary<string, string>
    {
        { "operation", request.Operation },
        { "duration_ms", stopwatch.ElapsedMilliseconds.ToString() }
    });
}
catch (Exception ex)
{
    telemetryClient.TrackException(ex);
    telemetryClient.TrackMetric("PluginErrorRate", 1);
}
```

**Fordeler:**
- Full kontroll over metrics og custom properties
- Multi-layer instrumentation (tokenize → infer → generate → detokenize)
- Granular performance debugging

**Ulemper:**
- Requires code changes for every extension
- DevOps overhead (ensure SDK updates)

---

### Pattern 3: User Feedback Loop

**Bruk:** Kontinuerlig forbedring basert på brukerrespons.

**Flow:**
1. User interacts with Copilot → Agent response
2. Thumbs up/down → Custom telemetry event: `UserFeedback`
3. Edit distance tracked → Metric: `avg_edit_distance`
4. KQL query identifies low-rated topics → Trigger re-evaluation

**KQL Example:**
```kusto
customEvents
| where name == "UserFeedback"
| extend rating = customDimensions['rating']
| where rating == "down"
| summarize count() by tostring(customDimensions['topicName'])
| order by count_ desc
```

**Fordeler:**
- Direkte input fra sluttbrukere
- Data-driven topic/prompt iteration

**Ulemper:**
- Feedback bias (users rarely rate neutral experiences)
- Privacy concerns (GDPR Article 6 – lawful basis for processing feedback)

---

## Beslutningsveiledning

### Når bruke hvilken løsning?

| Scenario | Anbefalt Tool | Reasoning |
|----------|---------------|-----------|
| Copilot Studio agent (low/no-code) | Built-in analytics + App Insights | No SDK required, out-of-the-box setup |
| Custom M365 Copilot plugin (TypeScript) | Application Insights SDK | Full control, correlation with Azure OpenAI metrics |
| Power Platform connector | Power Platform telemetry + App Insights | Hybrid (connector-level + custom events) |
| Compliance audit (Forvaltningsloven) | Microsoft Sentinel + Purview | Audit logs for decisions/actions |
| Cost tracking (Azure OpenAI) | Azure Monitor (OpenAI resource metrics) | Token-level billing data |

### Vanlige feil

| Feil | Konsekvens | Løsning |
|------|------------|---------|
| **Logger sensitive data (PII) uten consent** | GDPR Article 5 brudd, Datatilsynet-varsel | Disable "Log sensitive Activity properties" OR anonymize/hash userid/text |
| **Inkluderer test-data i production metrics** | Falske performance-trender | Filter `designMode == "False"` in all KQL queries |
| **Mangler correlation IDs** | Kan ikke tracke multi-step flows | Use `Activity.Current.RootId` (ASP.NET Core) eller custom correlation headers |
| **Ignorer latency breakdown** | Identify bottleneck i feil lag | Instrument tokenize, infer, generate, detokenize separat |
| **Ingen alerting på error spikes** | Incidents oppdages for sent | Set up Azure Monitor alerts (e.g., >5% error rate in 5 min window) |

### Røde flagg

- **Manglende telemetri i 30+ dager** → Brukere ikke adoptert extension?
- **Edit distance >50% of response length** → Agent gir irrelevante svar
- **>10% filtered prompts** → Content filtering blokkerer legitim bruk (juster policy)
- **Token cost øker 3x uten brukerøkning** → Prompt inefficiency eller token leakage

---

## Integrasjon med Microsoft-stakken

### Azure Monitor Ecosystem

```
Application Insights ← Copilot Studio, Plugins, Connectors
        ↓
Azure Monitor Workspace → Azure Copilot observability agent (preview)
        ↓
Microsoft Sentinel ← Audit logs (via Purview)
        ↓
Power BI / Azure Workbooks → Executive dashboards
```

**Key Integrations:**

| Integration | Use Case | Configuration |
|-------------|----------|---------------|
| **Azure OpenAI metrics** | Token usage, model latency, throttling | No extra config – auto-emitted to Azure Monitor |
| **Purview audit logs** | Copilot Studio events (BotCreate, BotPublish, BotShare) | Enable audit logging for Microsoft 365 license holders |
| **Sentinel analytics** | Custom detection rules (e.g., unusual token spikes) | Ingest App Insights logs, create KQL-based rules |
| **Copilot Studio Kit** | Automated testing + telemetry enrichment | Register Azure AD app, grant App Insights API permissions |

### Cross-Service Correlation

**Scenario:** M365 Copilot plugin calls Azure Function → Azure OpenAI → Cosmos DB

**Solution:**
1. **Distributed tracing**: Use `traceparent` HTTP header (W3C standard)
2. **Correlation ID**: Propagate `operation_Id` through all layers
3. **KQL join**:
```kusto
requests
| join (dependencies) on operation_Id
| join (customEvents | where name == "LLMInvocation") on operation_Id
| project timestamp, request_name, dependency_name, llm_tokens=customDimensions['tokens']
```

---

## Offentlig sektor (Norge)

### GDPR og Schrems II

**Utfordring:** Application Insights lagrer data i Azure-region (e.g., West Europe). Schrems II krever vurdering av USA-baserte sub-processors.

**Mitigering:**
- Bruk **EU Data Boundary** (Microsoft commitment per nov 2024)
- Aktivér **Data Residency** i Application Insights (Settings → Data retention)
- DPIA for logging av `text` (personopplysninger i meldinger)

### Forvaltningsloven § 11

**Krav:** Journalføring av vedtak truffet av forvaltning.

**Implementering:**
1. Custom telemetry event når Copilot-agent treffer "decision topic":
```csharp
telemetryClient.TrackEvent("DecisionMade", new Dictionary<string, string>
{
    { "decisionType", "LoanApproval" },
    { "caseId", "2026-001234" },
    { "timestamp", DateTime.UtcNow.ToString("o") }
});
```
2. Sentinels analytics rule → arkivering i case management system (e.g., ePhorte)

### AI Act (EU 2024/1689)

**Artikkel 12:** High-risk AI-systemer skal logge operations for traceability.

**Relevans:** Copilot-agent som automatiserer saksbehandling = high-risk.

**Compliance:**
- Log alle inputs (prompts), outputs (responses), intermediate steps (topic flow)
- Retention: Minimum 6 måneder (AI Act Article 12(1))
- Access control: Kun autoriserte brukere (RBAC via Application Insights)

---

## Kostnad og lisensiering

### Prismodell (Application Insights)

| Component | Pricing | Optimization Tips |
|-----------|---------|-------------------|
| **Data ingestion** | $2.76/GB (first 5GB free/month) | Use sampling (e.g., 50% for non-critical events) |
| **Data retention** | Free (90 days), $0.12/GB/month beyond | Archive to Azure Storage for long-term compliance |
| **Web tests** | $5.75 per test/month | Not required for Copilot extensions (use synthetic monitoring via Azure Functions) |

**Estimert cost for Copilot Studio agent (1000 users, 50k msgs/month):**
- Telemetry volume: ~10GB/month (hvis logging av messages enabled)
- Cost: $13.80/month (ingestion) + retention cost
- **Tip:** Disable "Log sensitive Activity properties" → reduser volume 30%

### Lisens-krav

| Feature | License Requirement |
|---------|---------------------|
| Copilot Studio analytics (built-in) | Power Virtual Agents license / Copilot Studio capacity |
| Application Insights integration | Azure subscription (free tier available) |
| Microsoft Sentinel (audit logs) | Microsoft 365 E5 OR Sentinel standalone |
| Power BI dashboards | Power BI Pro per user OR Premium capacity |

---

## For arkitekten (Cosmo)

### 5 spørsmål å stille kunden

1. **Scope:** Hvilke Copilot-extensions skal overvåkes? (Copilot Studio agents, M365 plugins, Power Platform connectors?)
2. **Compliance:** Er dette high-risk AI under AI Act? Trenger dere audit trail for Forvaltningsloven?
3. **Sensitive data:** Logger dere meldingstekst? Har dere DPIA for logging av personopplysninger?
4. **Alerting:** Hvem skal varsles ved error spikes, cost overruns eller security events?
5. **Retention:** Hvor lenge må telemetri oppbevares? (GDPR minimums vs. compliance-krav)

### Fallgruver

| Fallgruve | Impact | Hvordan unngå |
|-----------|--------|---------------|
| **Over-logging i test-fase** | Kostnadssprekk | Filter `designMode == "False"` i KQL |
| **Manglende sampling strategy** | Unødvendig detaljnivå → dyrt | 100% logging for errors, 10-50% for success events |
| **Ingen incident response plan** | Treg respons på security events | Set up Azure Monitor action groups (email, SMS, webhook til Teams/Slack) |
| **Siloed telemetry** | Kan ikke correlate plugin + LLM + backend | Bruk distributed tracing (W3C traceparent) |

### Anbefalinger per modenhetsnivå

#### Level 1: MVP (First Copilot Extension)
- Bruk Copilot Studio built-in analytics
- Enable Application Insights med basic logging (ikke sensitive properties)
- Set up 2-3 alerts (error rate >5%, response time >3s)

#### Level 2: Production Scale (5+ extensions)
- Centralized Application Insights workspace
- Custom telemetry events for business metrics (e.g., "LoanApprovalGranted")
- Pre-built dashboards (Copilot Studio Workbook + custom Azure Workbook)

#### Level 3: Enterprise/Compliance-heavy
- Microsoft Sentinel integration for audit logs
- Distributed tracing across all tiers (plugin → LLM → backend)
- Automated anomaly detection (Azure Monitor ML-based alerts)
- Quarterly compliance audit exports (GDPR, AI Act)

---

## Kilder og verifisering

**Verified (fra Microsoft Learn MCP):**
1. [Capture telemetry with Application Insights - Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry) – Full guide til App Insights setup, KQL queries, custom dimensions
2. [Observability for pro-code generative AI solutions](https://learn.microsoft.com/en-us/microsoft-cloud/dev/copilot/isv/observability-for-ai) – ISV-guidance: lifecycle phases, metrics categories, evaluation techniques
3. [Monitor operations, compliance, and capacity - Copilot Studio](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/sec-gov-phase5) – Operational monitoring, Sentinel integration, compliance auditing
4. [Application Insights telemetry with Microsoft Copilot Studio (Dynamics 365)](https://learn.microsoft.com/en-us/dynamics365/guidance/resources/copilot-studio-appinsights) – Prerequisites, custom events, topic tracking
5. [Enable Application Insights support in Copilot Studio Kit](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/kit-enable-application-insights) – Azure AD app registration, API permissions for telemetry enrichment

**Baseline (modellkunnskap, verifisert mot docs):**
- GDPR Article 5 (data minimization), Article 6 (lawful basis), Article 35 (DPIA)
- AI Act (EU 2024/1689) Article 12 (logging for high-risk AI)
- Forvaltningsloven § 11 (journalføring av vedtak)

**Konfidensnivå per seksjon:**
- Kjernekomponenter: **Verified** (App Insights docs, Copilot Studio Workbook)
- Arkitekturmønstre: **Baseline** (patterns basert på Azure Well-Architected Framework + docs)
- Offentlig sektor: **Verified** (GDPR/AI Act legal text + Microsoft EU Data Boundary docs)
- Kostnad: **Verified** (Azure pricing calculator, Application Insights pricing page)