ms-ai-architect/skills/ms-ai-security/references/ai-security-engineering/ai-threat-modeling-stride.md

# AI Threat Modeling Using STRIDE Framework

**Last updated:** 2026-05
**Status:** Established Practice
**Category:** AI Security Engineering
**Type:** reference
**Source:** https://learn.microsoft.com/ai/playbook/technology-guidance/generative-ai/mlops-in-openai/security/security-plan-llm-application

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [Arkitekturmønstre](#arkitekturmønstre)
- [Beslutningsveiledning](#beslutningsveiledning)
- [Integrasjon med Microsoft-stakken](#integrasjon-med-microsoft-stakken)
- [Offentlig sektor (Norge)](#offentlig-sektor-norge)
- [Kostnad og lisensiering](#kostnad-og-lisensiering)
- [For arkitekten (Cosmo)](#for-arkitekten-cosmo)
- [Kilder og verifisering](#kilder-og-verifisering)

## Introduksjon

Trusselmodellering for AI-systemer krever en tilpasning av etablerte sikkerhetsprinsipper til nye angrepsflater som er spesifikke for maskinlæring og generativ AI. Microsoft har utvidet det klassiske STRIDE-rammeverket (Spoofing, Tampering, Repudiation, Information Disclosure, Denial of Service, Elevation of Privilege) til å dekke AI-spesifikke trusler som datapoisoning, adversarial attacks, model inversion og prompt injection.

STRIDE for AI bygger på Microsoft Security Development Lifecycle (SDL), men introduserer nye dimensjoner: behandling av treningsdata som trust boundaries, vurdering av modellens output-integritet, og kartlegging av dependencies i ML supply chain. Rammeverket sikrer at både data scientists og security engineers kan ha strukturerte samtaler om AI-risiko uten å kreve dyp ekspertise i hverandres felt.

I norsk offentlig sektor er strukturert trusselmodellering et krav for AI-systemer som behandler personopplysninger eller støtter kritiske beslutningsprosesser. NSMs grunnprinsipper for IKT-sikkerhet må suppleres med AI-spesifikke sikkerhetskrav, og STRIDE-basert trusselmodellering gir et systematisk grunnlag for ROS-analyse og sikkerhetskontroller.

## Kjernekomponenter

### STRIDE Adaptation for AI Systems

| STRIDE Category | AI-Specific Threat | Severity | Mitigation Focus |
|-----------------|-------------------|----------|------------------|
| **Spoofing** | Neural Net Reprogramming, Malicious ML Providers | Important-Critical | Strong API authentication, access control, client-server mutual auth |
| **Tampering** | Data Poisoning (targeted/indiscriminate), Backdoored Models | Critical | Training data validation, anomaly detection, RONI defense, bagging |
| **Repudiation** | Model output manipulation, training data lineage loss | Moderate | Logging, audit trails, data provenance tracking |
| **Information Disclosure** | Model Inversion, Membership Inference, Model Stealing | Important-Critical | Rate limiting, access control, output obfuscation, differential privacy |
| **Denial of Service** | Confidence Reduction, Random Misclassification | Important | Adversarial training, feature denoising, input validation |
| **Elevation of Privilege** | Adversarial Perturbation, Excessive Agency, Physical Domain Attacks | Critical | Adversarial robustness, least privilege on plugins, input sanitization |

### Trust Boundary Shifts in AI

Tradisjonell trusselmodellering fokuserer på nettverksgrenser og applikasjonsgrenser. I AI-systemer må trust boundaries utvides til:

1. **Training Data Stores** — behandles som potensielt kompromitterte kilder (garbage-in/garbage-out)
2. **ML Supply Chain** — pre-trained models, model zoos, data providers, MLaaS-leverandører
3. **Model APIs** — query-access kan misbrukes til model extraction, inversion, membership inference
4. **Plugin/Extension Layer** — LLM-agents som kaller eksterne verktøy introduserer nye EOP-vektorer
5. **Physical Domain** — AI-beslutninger kan manifestere seg fysisk (autonomous vehicles, robotics)

### Key Questions in AI Security Reviews

**Data Integrity:**
- Hvis treningsdata er kompromittert, hvordan oppdages det?
- Brukes user-supplied inputs i trening? Hvilken validering gjøres?
- Kan modellen outputte sensitive data den ble trent på?
- Hva er lineage og provenance for treningsdata?

**Model Security:**
- Kan modellen kopieres/stjeles gjennom API-queries?
- Kan membership inference avsløre om spesifikke personer er i treningsdatasettet?
- Returnerer modellen raw confidence scores som kan misbrukes?
- Kan adversarial examples tvinge misklassifisering?

**Supply Chain:**
- Hvilke third-party models eller data providers brukes?
- Er pre-trained models verifisert for backdoors eller poisoning?
- Kan 3rd-party kunder bygge facade over API-et for skadelig bruk?

**Impact Assessment:**
- Kan modellen brukes til å forårsake fysisk skade (self-driving cars, medical diagnosis)?
- Hva er konsekvensen av false positives vs false negatives?
- Kan output brukes til trolling, bias amplification eller reputational damage?

## Arkitekturmønstre

### Pattern 1: Defense in Depth for Training Pipeline

**Scenario:** Organisasjon trener egne modeller på curated datasets kombinert med public data.

**Threat Model Approach:**
1. **Data Ingestion Boundary** — validate, sanitize, log all external data sources; implement anomaly detection on data distribution
2. **Training Environment Isolation** — segregate training from production; use private endpoints, managed identities
3. **Model Validation Gateway** — test for adversarial robustness, bias, performance drift before deployment
4. **Monitoring Layer** — track confidence scores, classification accuracy, data lineage changes

**Fordeler:**
- Reduserer risiko for data poisoning ved å isolere hver fase
- Gir audit trail for ROS-analyse og incident response
- Tillater rollback til tidligere modellversjoner ved kompromittering

**Ulemper:**
- Økt kompleksitet og kostnader
- Krever dedikert security competence i data science team

---

### Pattern 2: Zero Trust for Model APIs

**Scenario:** Eksponering av ML-modell som API for interne eller eksterne consumers.

**Threat Model Approach:**
1. **Authentication** — Entra ID managed identities, no stored credentials
2. **Authorization** — RBAC with least privilege; rate limiting per caller
3. **Input Validation** — define well-formed queries; reject malformed/adversarial inputs
4. **Output Sanitization** — round confidence scores; redact sensitive data patterns; apply content filtering
5. **Monitoring** — detect high-frequency queries (model stealing), anomalous inputs (adversarial examples)

**Fordeler:**
- Beskytter mot model extraction og inversion attacks
- Gir telemetry for sikkerhetshendelser
- Enklere å implementere compliance-kontroller (DLP, logging)

**Ulemper:**
- Rate limiting kan påvirke legitime bruksscenarioer
- Output obfuscation kan redusere nytteverdi for consumers

---

### Pattern 3: Threat Modeling for Agentic AI (LLM with Plugins)

**Scenario:** Copilot Studio agent med custom plugins som kan utføre actions (e.g., send email, update database).

**Threat Model Approach:**
1. **Identify Trust Boundaries** — user prompt → orchestrator → LLM → plugin/MCP server → external service *(Verified MCP 2026-04)*
2. **Apply STRIDE per Boundary:**
   - **User Prompt (I)** — Prompt Injection, Jailbreaking (Elevation of Privilege)
   - **Orchestrator (T)** — Intent Detection Manipulation (Tampering)
   - **LLM Output (I)** — Insecure Output Handling, Hallucinations (Information Disclosure)
   - **Plugin/MCP Layer (E)** — Excessive Agency, Unauthorized Actions; MCP server endpoints er ny angrepsflate som bør sikres via Azure API Management (Elevation of Privilege) *(Verified MCP 2026-04)*
   - **External Service (S)** — Credential Leakage, Data Exfiltration (Spoofing/Information Disclosure)
3. **Mitigation Controls:**
   - Prompt Shields (Azure AI Content Safety)
   - Least privilege for plugins (minimal scope, approval workflows)
   - Output validation and sanitization before plugin execution
   - Logging and monitoring of all plugin actions

**Fordeler:**
- Systematisk kartlegging av alle angrepsflater i kompleks agent-arkitektur
- Enklere å kommunisere risiko til non-technical stakeholders
- Grunnlag for DPIA og sikkerhetsdokumentasjon

**Ulemper:**
- Krever dyp forståelse av både LLM-sikkerhet og plugin-arkitektur
- Mitigations kan begrense agent-funksjonalitet (user experience trade-offs)

## Beslutningsveiledning

### Når Bruke STRIDE vs. MITRE ATLAS vs. OWASP Top 10 for LLM

| Framework | Best Fit | Key Advantage | Limitations |
|-----------|----------|---------------|-------------|
| **STRIDE (AI-adapted)** | Traditional ML systems, model APIs, training pipelines | Established SDL integration, broad security coverage | Mindre granularitet for LLM-specific threats (prompt injection) |
| **MITRE ATLAS** | Deep threat intelligence, red team exercises, adversarial ML focus | Comprehensive adversarial tactics, real-world attack examples | Mer teknisk, vanskelig for non-security stakeholders |
| **OWASP Top 10 for LLM** | Generative AI applications, chatbots, RAG systems | LLM-specific (prompt injection, insecure output, over-reliance) | Mindre coverage for traditional ML threats |

**Anbefaling:** Bruk STRIDE som baseline framework, supplement med MITRE ATLAS for adversarial scenarios og OWASP Top 10 for LLM-components.

### Common Mistakes in AI Threat Modeling

| Mistake | Impact | Correction |
|---------|--------|------------|
| **Treating training data as trusted** | Data poisoning går uoppdaget; modell kompromitteres permanent | Implement data provenance tracking, anomaly detection, input validation |
| **Ignoring model extraction risk** | Intellectual property loss; adversarial attacks developed offline | Apply rate limiting, output obfuscation, access control on model APIs |
| **No monitoring for adversarial inputs** | Persistent misclassification attacks | Deploy adversarial detection (feature squeezing, confidence analysis) |
| **Over-scoping plugin permissions** | LLM agent kan utføre unauthorized actions | Least privilege per plugin; require user approval for sensitive actions |
| **Missing physical domain impact assessment** | Safety-critical systems kompromittert (autonomous vehicles, medical AI) | Include physical harm scenarios in threat model; higher severity bar |

### Red Flags in AI Architecture Review

- [ ] Modellen trenes på public/uncurated data uten validering
- [ ] API returnerer raw confidence scores med høy presisjon
- [ ] Ingen rate limiting eller access control på model endpoints
- [ ] Plugin-layer har read/write til sensitive datastores uten approval workflow
- [ ] Training environment er ikke isolert fra production
- [ ] Ingen logging av model queries eller plugin actions
- [ ] Pre-trained models brukes uten source verification
- [ ] RAG-system tillater retrieval av data utenfor user's access scope

## Integrasjon med Microsoft-stakken

### Azure AI Services

**Azure AI Content Safety** — Prompt Shields for jailbreak detection, content filters for insecure output handling
```plaintext
Threat: Prompt Injection (OWASP LLM01)
Mitigation: Enable Prompt Shields, configure jailbreak detection thresholds
STRIDE Mapping: Elevation of Privilege (user manipulates system via crafted prompt)
```

**Azure OpenAI Service** — Data privacy commitments (no training on customer data), content filtering, abuse monitoring
```plaintext
Threat: Model Inversion, Membership Inference
Mitigation: Customer data not used for training; apply output redaction for PII
STRIDE Mapping: Information Disclosure
```

**Microsoft Foundry** — Secure MLOps pipelines, managed identities, private endpoints, model registry with versioning
```plaintext
Threat: Backdoored Model, ML Supply Chain Attack
Mitigation: Model provenance tracking, digital signatures, isolated training environments
STRIDE Mapping: Tampering
```

### Microsoft Defender for Cloud — AI Security Posture Management

**Capabilities:** *(Verified MCP 2026-04)*
- Automated detection of AI workloads across Azure subscriptions
- AI security posture management: automate detection and remediation of generative AI risks
- Security recommendations for AI models, data stores, network isolation

**Threat Modeling Integration:**
```plaintext
1. Run STRIDE threat model workshops
2. Map identified threats to Defender for Cloud controls
3. Enable AI threat protection in Defender
4. Monitor security posture; triage alerts in context of threat model
```

### Microsoft Threat Modeling Tool

**AI-Specific Templates:**
- ML Training Pipeline (data ingestion, training, validation, deployment)
- Model API (authentication, input validation, output sanitization)
- LLM Agent (prompt handling, orchestration, plugin execution)

**Usage:**
1. Load template matching architecture (Microsoft Foundry, Copilot Studio, custom ML)
2. Identify data flows and trust boundaries
3. Generate threats using STRIDE methodology
4. Review AI-specific threat categories (see microsoft.com/security/engineering/threat-modeling-aiml)
5. Assign mitigations and track in Azure DevOps or GitHub Issues

## Offentlig sektor (Norge)

### NSM Grunnprinsipper for IKT-Sikkerhet (AI-Tilpasning)

| NSM Prinsipp | AI Threat Modeling Tilpasning |
|--------------|-------------------------------|
| **Identifisere og kartlegge** | Inventory AI models, training data stores, ML supply chain dependencies |
| **Beskytte** | Apply STRIDE mitigations; implement access control, input validation, adversarial robustness |
| **Oppdage** | Monitor for data poisoning, adversarial inputs, model extraction attempts; log all API queries |
| **Håndtere og gjenopprette** | Incident response for AI-specific threats; rollback to previous model versions; retrain on clean data |

### ROS-Analyse for AI-Systemer

**Strukturert tilnærming:**
1. **Trussel Identifikasjon** — bruk STRIDE for AI som sjekkliste; inkluder MITRE ATLAS tactics
2. **Sannsynlighetsvurdering** — vurder angrepsvektor (remote vs. local), required expertise, attack complexity
3. **Konsekvensvurdering** — personvern (GDPR), sikkerhet (fysisk skade), omdømme (bias/diskriminering), økonomi (IP-tap)
4. **Risikoberegning** — sannsynlighet × konsekvens; prioriter høyrisiko-trusler
5. **Tiltak** — koble mitigations til identifiserte trusler; spesifiser kontroller (tekniske, organisatoriske)

### Compliance og Dokumentasjon

**DPIA (Personvernkonsekvens):**
- Threat modeling dokumentasjon brukes som input til DPIA
- Spesifikk vurdering av Information Disclosure threats (model inversion, membership inference)
- Dokumenter differential privacy eller andre privacy-enhancing technologies

**Utredningsinstruksen (AI-systemer i forvaltning):**
- Trusselmodell skal dokumentere sikkerhetskrav i alternativanalyse
- Kostnad for security controls inngår i kostnadsvurdering
- Residual risk dokumenteres i risikoanalyse-vedlegg

**Sikkerhetsloven (Kritiske AI-systemer):**
- AI-systemer i kritisk infrastruktur krever årlig ROS-analyse (inkludert threat modeling)
- Trusselbildet må oppdateres basert på nye angrepsmetoder (MITRE ATLAS, OWASP)

## Kostnad og lisensiering

### Microsoft Security Tools for AI Threat Modeling

| Tool | License/Cost | Capabilities |
|------|-------------|--------------|
| **Microsoft Threat Modeling Tool** | Free download | STRIDE automation, AI-specific templates, threat reports |
| **Microsoft Defender for Cloud (AI)** | ~$15/server/month (standard tier) | AI workload discovery, security posture management, threat detection |
| **Azure AI Content Safety** | Pay-per-use (~$1 per 1K text records) | Prompt Shields, jailbreak detection, content filtering |
| **Microsoft Purview (Data Governance)** | Starts at $0.30/GB scanned | Data classification, lineage tracking, DLP policies for AI data |

### Threat Modeling Workshop Cost Estimate (Norway Public Sector)

**Scenario:** AI-basert saksbehandlingssystem, 5 komponenter (front-end, orchestrator, LLM, RAG, database)

| Activity | Effort (hours) | Rate (NOK) | Cost (NOK) |
|----------|----------------|------------|-----------|
| Pre-workshop (architecture review, stakeholder interviews) | 8 | 1500 | 12 000 |
| STRIDE workshop facilitation (security architect + team) | 4 | 2000 | 8 000 |
| Threat documentation and mitigation mapping | 6 | 1500 | 9 000 |
| Review and approval cycle | 2 | 1500 | 3 000 |
| **Total** | **20** | | **32 000** |

**Note:** Dette er rådgivningskostnad for gjennomføring. Implementering av mitigations (e.g., Azure security controls) kommer i tillegg.

## For arkitekten (Cosmo)

### 8 Spørsmål å Stille i Arkitekturdialog

1. **Trust Boundaries:** "Hvor er trust boundaries i deres AI-arkitektur? Behandles treningsdata som potensielt kompromittert kilde?"
   - *Hvorfor:* Etablerer scope for trusselmodellering; unngår blind trust på data providers.

2. **Model Exposure:** "Hvordan eksponeres modellen? API, embedded i app, on-device? Hvem har query-access?"
   - *Hvorfor:* Model APIs er primær angrepsfelt for extraction, inversion, adversarial attacks.

3. **Supply Chain Dependencies:** "Brukes pre-trained models eller third-party data? Hvordan verifiseres integritet?"
   - *Hvorfor:* Backdoored models og data poisoning er Critical-severity trusler.

4. **Physical Domain Impact:** "Kan AI-beslutninger manifestere seg fysisk (e.g., autonomous systems, safety-critical)?"
   - *Hvorfor:* Øker severity bar; krever mer robust adversarial defenses.

5. **Sensitive Data in Training:** "Inneholder treningsdata personopplysninger eller forretningshemmeligheter? Kan disse leakes via model output?"
   - *Hvorfor:* Information Disclosure threat; krever differential privacy eller data minimization.

6. **Adversarial Robustness Testing:** "Er modellen testet mot adversarial examples? Finnes det red team plan?"
   - *Hvorfor:* Proaktiv oppdagelse av sårbarheter før deployment.

7. **Incident Response Plan:** "Hva er plan hvis modellen blir kompromittert eller data poisoning oppdages?"
   - *Hvorfor:* AI-specific incident response (rollback, retrain, forensics) må være definert.

8. **Compliance Alignment:** "Hvordan dokumenteres threat model for DPIA, ROS-analyse eller sikkerhetsgodkjenning?"
   - *Hvorfor:* Sikrer at threat modeling leverer nødvendig dokumentasjon for offentlig sektor compliance.

### Vanlige Fallgruver

**Fallgruve 1: "Vi bruker Azure OpenAI, så sikkerhet er Microsofts ansvar"**
- *Realitet:* Microsoft sikrer platform, men kunde må implementere access control, prompt injection defense, output validation, monitoring.
- *Cosmo's respons:* "Shared responsibility model gjelder også AI. Dere må threat-modellere deres bruk av Azure OpenAI, ikke selve tjenesten."

**Fallgruve 2: "Threat modeling er for traditional security, AI er annerledes"**
- *Realitet:* STRIDE er tilpasset AI; tradisjonell sikkerhet er fortsatt viktig (exploit software dependencies er AI-trussel #11).
- *Cosmo's respons:* "AI introduserer nye trusler, men fundamentet er det samme. STRIDE gir felles språk mellom security og data science."

**Fallgruve 3: "Vi gjør threat modeling én gang ved prosjektstart"**
- *Realitet:* AI-systemer evolverer (nye data sources, model updates, plugin additions); threat model må oppdateres.
- *Cosmo's respons:* "Threat model er living document. Oppdater ved hver arkitekturendring, og gjenta ved nye releases."

### Anbefalinger for Gjennomføring

1. **Involver både security og data science** — Unngå siloer; STRIDE-workshop krever begge perspektiver.
2. **Start med data flow diagram** — Visualiser alle komponenter, grenser, data flows før STRIDE-analyze.
3. **Bruk threat libraries** — MITRE ATLAS og OWASP Top 10 for LLM som supplement til STRIDE; ikke start fra scratch.
4. **Prioriter basert på severity OG feasibility** — Critical-severity trussel med lav attack complexity må fikses først.
5. **Koble til eksisterende SDL-prosess** — Threat modeling skal ikke være isolert; integrer med code review, testing, deployment pipelines.
6. **Dokumenter for compliance** — ROS-analyse, DPIA, sikkerhetsgodkjenning krever strukturert trusselmodell; bruk STRIDE som grunnlag.
7. **Test mitigations** — Ikke anta at adversarial training fungerer; red team testing er nødvendig.
8. **Oppdater threat model kontinuerlig** — Nye angrepsmetoder publiseres (MITRE ATLAS tracker real-world incidents); hold threat model current.

## Kilder og verifisering

**Microsoft Learn — Verified Sources (2026-02):**

1. [Threat Modeling AI/ML Systems and Dependencies](https://learn.microsoft.com/en-us/security/engineering/threat-modeling-aiml) — **Authoritative guide** for STRIDE adaptation to AI/ML; includes 11 threat categories with mitigations
2. [Secure AI (Cloud Adoption Framework)](https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/secure) — Integration of STRIDE, MITRE ATLAS, OWASP for comprehensive AI risk identification. Oppdatert 2026-04: inkluderer nå AI asset inventory via Azure Resource Graph, AI communication channel security med Managed Identities og Virtual Networks, APIM for sikring av MCP server-endepunkter, og Microsoft Purview Insider Risk Management for prompt-basert data exfiltration-deteksjon. *(Verified MCP 2026-04)*
3. [AI Risk Assessment for ML Engineers](https://learn.microsoft.com/en-us/security/ai-red-team/ai-risk-assessment) — Control framework for ML security assessment; incident response and business continuity
4. [Security Planning for LLM-based Applications](https://learn.microsoft.com/en-us/ai/playbook/technology-guidance/generative-ai/mlops-in-openai/security/security-plan-llm-application) — 11 LLM-specific threats mapped to STRIDE; mitigation patterns for Azure OpenAI
5. [Reference Data Flows and Threat Models for Security Evaluations (Copilot Studio)](https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/architecture/threat-models) — Agent architecture threat modeling; custom engine data flow analysis
6. [Securing the Future of AI and ML at Microsoft](https://learn.microsoft.com/en-us/security/engineering/securing-artificial-intelligence-machine-learning) — Introduction to AI-specific security pivots (Resilience, Discretion)
7. [Failure Modes in Machine Learning](https://learn.microsoft.com/en-us/security/engineering/failure-modes-in-machine-learning) — Adversarial ML threat taxonomy (foundation for STRIDE adaptation)
8. [Microsoft Threat Modeling Tool](https://learn.microsoft.com/en-us/azure/security/develop/threat-modeling-tool) — Tool documentation; AI-specific templates

**Confidence Level:** ✅ **Verified** — All content grounded in official Microsoft documentation (8 unique sources, retrieved 2026-02, re-verified 2026-04). STRIDE adaptation for AI is established practice in Microsoft SDL.

**Status:** ✅ **Current** — Threat categories and mitigations reflect 2025-2026 threat landscape (includes prompt injection, RAG vulnerabilities, agentic AI risks, MCP server endpoints). *(Verified MCP 2026-04)*

**Baseline Knowledge Integration:** Framework names (STRIDE, MITRE ATLAS, OWASP), Norwegian public sector context (NSM, ROS, DPIA, Sikkerhetsloven) derived from model knowledge and cross-referenced with retrieved sources for accuracy.