ktg-plugin-marketplace/plugins/ms-ai-architect/skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md

Security and Access Control in MLOps

Kategori: MLOps & GenAIOps Last updated: 2026-04 Dato: 2026-04-10 Confidence: HIGH — Basert på offisiell Microsoft Learn dokumentasjon (8 MCP-oppslag, 16 kilder)

---

Introduksjon

Security and access control utgjør fundamentet for enterprise-grade MLOps i Azure Machine Learning. Denne kunnskapsreferansen dekker identitetsstyring, nettverksisolasjon, datakryptering og tilgangskontroll gjennom hele ML-livssyklusen — fra treningsjobber til produksjons-endpoints.

Hvorfor dette er kritisk for MLOps:

• Beskytter treningsdata, modeller og inferens-endepunkter mot uautorisert tilgang
• Sikrer compliance med GDPR, ePrivacy-direktivet og norske personvernkrav
• Reduserer risiko for data exfiltration i delte workspace-miljøer
• Muliggjør audit trails og samsvarskontroll for regulerte virksomheter

I produksjonsmiljøer er sikkerhet ikke en tilleggsfunksjon, men en arkitekturell forutsetning.

---

Kjernekomponenter

1. Identitetshåndtering med Managed Identities

Azure Machine Learning støtter to typer managed identities for service-to-service autentisering:

System-Assigned Managed Identity (SAI)

• Livssyklus: Automatisk opprettet og slettet sammen med workspace/compute
• Bruksområde: Standard for workspace → storage/keyvault/ACR kommunikasjon
• Permissions (workspace SAI):
  • Contributor på workspace
  • Storage Blob Data Contributor på storage account
  • Full access til Key Vault keys/secrets/certificates
  • Contributor på Container Registry

# Verifiser workspace identity
az ml workspace show --name <workspace-name> \
  --resource-group <resource-group> \
  --query identity

User-Assigned Managed Identity (UAI)

• Livssyklus: Uavhengig av workspace — kan gjenbrukes på tvers av ressurser
• Bruksområde: Multi-workspace scenarios, shared resources, least-privilege access
• Fordeler:
  • Granular tilgangskontroll per compute cluster
  • Data isolation i delte storage accounts (via ABAC conditions)
  • Enklere key rotation og credential management

Oppsett av UAI for workspace:

# workspace-uai.yml
identity:
  type: user_assigned
  user_assigned_identities:
    '<UAI-resource-ID-1>': {}
    '<UAI-resource-ID-2>': {}
storage_account: <storage-account-resource-ID>
key_vault: <key-vault-resource-ID>
primary_user_assigned_identity: <UAI-resource-ID-1>

az ml workspace create -f workspace-uai.yml \
  --subscription <subscription-id> \
  --resource-group <resource-group> \
  --name <workspace-name>

RBAC-krav for UAI (minimum):

Ressurs	Rolle	Hvorfor
Workspace	Contributor	Control plane operations
Storage Account	Storage Blob Data Contributor	Data plane access (blob)
Key Vault (RBAC-modell)	Key Vault Administrator	Data plane access
Container Registry	Contributor	Image pull/push
Application Insights	Contributor	Logging og metrics

Compute Cluster Identity

Compute clusters støtter enten system-assigned eller user-assigned identities (ikke begge samtidig).

Use case: Identity-based data access i treningsjobber

# I treningsjobb — bruk compute cluster sin managed identity
import os
from azure.identity import ManagedIdentityCredential

client_id = os.environ.get('DEFAULT_IDENTITY_CLIENT_ID')
credential = ManagedIdentityCredential(client_id=client_id)
token = credential.get_token('https://storage.azure.com/')

Opprette compute cluster med UAI:

# compute-cluster-uai.yml
name: secure-cluster
type: amlcompute
size: STANDARD_D2_V2
min_instances: 0
max_instances: 4
identity:
  type: user_assigned
  user_assigned_identities:
    - resource_id: "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.ManagedIdentity/userAssignedIdentities/<identity-name>"

---

2. Role-Based Access Control (RBAC)

Azure Machine Learning bruker Azure RBAC for tilgangskontroll til workspace, data plane og compute resources.

Built-in roller

Rolle	Tilganger	Bruksområde
AzureML Data Scientist	Submit jobs, view data, manage models	Standard datavitenskapsrolle
AzureML Compute Operator	Manage compute resources	Infrastruktur-team
Reader	View workspace metadata	Audit og reporting
Contributor	Full workspace access	Workspace administrators

Custom Roles for MLOps

Eksempel: Minste privilegium for production deployment

{
  "Name": "MLOps Deployment Role",
  "Description": "Can deploy models to production endpoints",
  "Actions": [
    "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/write",
    "Microsoft.MachineLearningServices/workspaces/onlineEndpoints/deployments/write",
    "Microsoft.MachineLearningServices/workspaces/models/*/read"
  ],
  "NotActions": [],
  "AssignableScopes": [
    "/subscriptions/<subscription-id>/resourceGroups/<rg>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>"
  ]
}

az role definition create --role-definition mlops-deploy-role.json
az role assignment create --assignee <identity-id> \
  --role "MLOps Deployment Role" \
  --scope "/subscriptions/<sub-id>/resourceGroups/<rg>"

RBAC Best Practices for MLOps

1. Separate Dev/Prod permissions: Bruk forskjellige roller for utvikling og produksjon
2. Compute cluster access: Grant Storage Blob Data Reader til compute identity for datastore access
3. Endpoint authentication: Bruk Entra ID token-based auth fremfor static keys
4. Service principal rotation: Bruk managed identities fremfor service principals med secrets
5. Just-in-time access: Kombiner med Microsoft Entra PIM for privileged operations

---

3. Nettverksisolasjon

Managed Virtual Network (anbefalt for nye workspaces)

Azure ML Managed VNet tilbyr fully managed nettverksisolasjon uten manuell konfigurasjon.

Støttede compute-typer:

• Serverless compute (inkl. Spark)
• Compute cluster
• Compute instance
• Managed online endpoint
• Batch endpoint

Outbound-modi:

Modus	Beskrivelse	Use case
Allow Internet Outbound	Tillater all utgående trafikk	Dev/test miljøer
Allow Only Approved Outbound	Kun godkjente private endpoints/FQDNs	Produksjon (anbefalt)

Oppsett:

az ml workspace create --name <workspace> \
  --resource-group <rg> \
  --managed-network allow_only_approved_outbound

Private Endpoint for Workspace

Private endpoints reduserer attack surface ved å eksponere workspace kun via private IP-adresser i VNet.

Opprett private endpoint:

az network private-endpoint create \
  --name <pe-name> \
  --vnet-name <vnet-name> \
  --subnet <subnet-name> \
  --private-connection-resource-id "/subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.MachineLearningServices/workspaces/<workspace>" \
  --group-id amlworkspace \
  --connection-name workspace \
  --location <location>

DNS-konfigurasjon (påkrevd):

# Opprett private DNS zone for workspace API
az network private-dns zone create \
  --resource-group <rg> \
  --name privatelink.api.azureml.ms

az network private-dns link vnet create \
  --resource-group <rg> \
  --zone-name privatelink.api.azureml.ms \
  --name ml-dns-link \
  --virtual-network <vnet-name> \
  --registration-enabled false

az network private-endpoint dns-zone-group create \
  --resource-group <rg> \
  --endpoint-name <pe-name> \
  --name ml-zone-group \
  --private-dns-zone privatelink.api.azureml.ms \
  --zone-name privatelink.api.azureml.ms

Storage Account Private Endpoints

For å unngå data exfiltration må storage accounts også isoleres:

Påkrevde private endpoints:

• Blob (alltid)
• File (alltid)
• Queue (kun for Batch endpoints / ParallelRunStep)
• Table (kun for Batch endpoints / ParallelRunStep)

Trusted service exception:

I Storage Account firewall, velg:

• "Selected networks"
• Resource instances: Microsoft.MachineLearningServices/Workspace
• Instance name: <your-workspace>

Dette tillater workspace managed identity å kommunisere med storage selv bak firewall.

---

4. Datakryptering

Encryption at Rest

Platform-managed keys (standard):

• Storage accounts: AES-256
• Cosmos DB metadata: Microsoft-managed keys
• Compute OS disks: Microsoft-managed keys

Customer-managed keys (CMK):

CMK gir ekstra kontroll over krypteringsnøkler, spesielt viktig for:

• GDPR compliance
• Regulerte sektorer (finans, helse, offentlig sektor)
• "Bring your own key" (BYOK) policies

Ressurser som bruker CMK:

• Azure Cosmos DB (workspace metadata)
• Azure AI Search (workspace indexes)
• Azure Storage (workspace artifacts)

Oppsett av CMK-workspace:

# Opprett Key Vault med soft delete + purge protection
az keyvault create --name <kv-name> \
  --resource-group <rg> \
  --enable-soft-delete \
  --enable-purge-protection

# Opprett RSA-nøkkel (minimum 3072-bit)
az keyvault key create \
  --vault-name <kv-name> \
  --name workspace-cmk \
  --kty RSA \
  --size 3072

# Hent nøkkel-ID
KEY_ID=$(az keyvault key show --vault-name <kv-name> \
  --name workspace-cmk --query key.kid -o tsv)

# Opprett workspace med CMK
az ml workspace create --name <workspace> \
  --resource-group <rg> \
  --customer-managed-key $KEY_ID \
  --key-vault /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.KeyVault/vaults/<kv-name>

Begrensninger:

• Nøkkelen må være i samme Azure subscription som workspace
• Compute OS-disker kan ikke krypteres med CMK (kun Microsoft-managed keys)
• Temporary disks på compute: Kun kryptert hvis hbi_workspace=true

High Business Impact (HBI) Workspace

Når hbi_workspace=true:

• Lokal scratch disk på compute instance krypteres
• Temporary disk på compute cluster krypteres
• Reduserer telemetri som Microsoft samler inn
• Ekstra kryptering i Microsoft-managed environments

az ml workspace create --name <workspace> \
  --resource-group <rg> \
  --hbi-workspace true

Encryption in Transit

All kommunikasjon bruker TLS 1.2:

• Workspace ↔ Storage Account
• Workspace ↔ Compute
• Studio ↔ Workspace API
• Inference clients ↔ Online endpoints

For online endpoints: Bruk TLS/SSL certificates for custom domains.

---

5. Data Exfiltration Prevention

Risikoscenarier:

• Malicious actors med tilgang til workspace sender treningsdata til ekstern storage
• Ukonfigurerte compute resources med åpen internett-tilgang

Mitigations

1. Managed VNet med approved outbound:

az ml workspace update --name <workspace> \
  --managed-network allow_only_approved_outbound

2. Disable public network access:

az ml online-endpoint create --file endpoint.yml \
  --set public_network_access=disabled

3. Audit outbound dependencies:

Dokumenter godkjente FQDNs/Service Tags:

• AzureActiveDirectory
• AzureFrontDoor.FrontEnd
• MicrosoftContainerRegistry
• AzureMonitor

4. Private endpoints for all storage:

Kombiner workspace private endpoint med storage private endpoints for full isolation.

---

Arkitekturmønstre

Mønster 1: Zero Trust MLOps Architecture

Komponenter:

[On-premises dev environment]
          ↓ (Azure VPN Gateway / ExpressRoute)
    [Azure Virtual Network]
          ↓ (Private Endpoint)
    [Workspace (private endpoint)]
          ↓ (Managed Identity auth)
    [Storage (private endpoint)]
    [Key Vault (private endpoint)]
    [Container Registry (private endpoint)]
          ↓ (Managed VNet compute)
    [Compute Cluster (no public IP)]
          ↓ (Private endpoint)
    [Online Endpoint (public_network_access=disabled)]

Sikkerhetslag:

1. Perimeter: VPN/ExpressRoute (ingen direkte internett-tilgang)
2. Identity: Managed identities + Entra ID MFA
3. Network: Private endpoints + NSGs + Managed VNet
4. Data: CMK + encryption in transit
5. Audit: Azure Monitor + Log Analytics + Sentinel

Mønster 2: Multi-Workspace Data Isolation

For organisasjoner med flere team som deler storage/keyvault/ACR:

Enable data isolation:

az ml workspace create --name <workspace> \
  --resource-group <rg> \
  --enable-data-isolation \
  --storage-account <shared-storage-resource-id> \
  --key-vault <shared-kv-resource-id>

Effekter:

• Storage containers prefix: {workspace-guid}-azureml-blobstore
• Key Vault secrets prefix: {workspace-guid}-
• Container Registry images prefix: {workspace-guid}/
• Workspace identity får ABAC condition som kun tillater tilgang til egne containere

Default for workspace kinds:

Workspace Kind	Data Isolation Default
hub	Enabled
project	Enabled (arvet fra hub)
default	Disabled

Mønster 3: User Identity Pass-through for Training Jobs

For fine-grained tilgangskontroll hvor ulike data scientists har ulike tilganger:

Oppsett:

# training-job.yml
command: python train.py --input-data ${{inputs.data}}
inputs:
  data:
    type: uri_folder
    path: azureml://datastores/secured-data/paths/team-a/
environment: azureml://registries/azureml/environments/sklearn-1.5
compute: azureml:secure-cluster
identity:
  type: user_identity

az ml job create --file training-job.yml

Krav:

• Datastore må bruke identity-based authentication (ikke cached credentials)
• User må ha Storage Blob Data Reader på storage account
• Kun støttet via CLI/SDK v2 (ikke Studio)
• Pipeline steps må konfigureres individuelt (ikke root-level)

Fordeler:

• Audit trails viser hvilken bruker som aksesserte hvilke data
• Reuse av eksisterende storage permissions
• Segregation of duties mellom data scientists

---

Beslutningsveiledning

Når velge System-Assigned vs. User-Assigned Managed Identity?

Velg System-Assigned når:

• ✅ Enkelt workspace med dedikerte ressurser
• ✅ Prototype/dev miljøer
• ✅ Minimal administrative overhead ønskes

Velg User-Assigned når:

• ✅ Delte ressurser på tvers av workspaces
• ✅ Least-privilege access per compute cluster
• ✅ Data isolation i multi-tenant scenarios
• ✅ Enklere key rotation / credential lifecycle management

Når bruke Private Endpoints?

Alltid bruk private endpoints når:

• ✅ Produksjonsworkloads med sensitive data
• ✅ Compliance-krav (GDPR, NIS2, ISO 27001)
• ✅ Cross-premises connectivity (hybrid cloud)
• ✅ Zero-trust arkitektur implementeres

Kan utelates i:

• ❌ Development/test workspaces uten sensitive data
• ❌ Proof-of-concepts med syntetiske data

Når bruke Customer-Managed Keys?

Påkrevd for:

• ✅ Regulerte sektorer (bank, helse, offentlig sektor)
• ✅ Contractual "bring your own key" krav
• ✅ Data residency compliance (GDPR Article 44-50)

Vurder kostnad/kompleksitet:

• ⚠️ Ekstra Azure-kostnader (Cosmos DB, AI Search)
• ⚠️ Key rotation procedures må etableres
• ⚠️ Disaster recovery kompleksitet øker

---

Integrasjon med Microsoft-stakken

Azure DevOps Integration

Service connection med managed identity:

# Opprett service principal for Azure DevOps
az ad sp create-for-rbac --name "azdo-ml-connection" \
  --role Contributor \
  --scopes /subscriptions/<sub-id>/resourceGroups/<rg>

Eller bruk workload identity federation (anbefalt):

Azure DevOps → Project Settings → Service connections → Azure Resource Manager → Workload Identity federation

Pipeline secret management:

# azure-pipelines.yml
variables:
  - group: ml-production-secrets  # Hentet fra Key Vault

steps:
  - task: AzureCLI@2
    inputs:
      azureSubscription: 'ml-service-connection'
      scriptType: 'bash'
      scriptLocation: 'inlineScript'
      inlineScript: |
        az ml job create --file training-job.yml \
          --set environment_variables.STORAGE_KEY=$(storage-account-key)

GitHub Actions Integration

OIDC authentication (ingen secrets):

# .github/workflows/train-model.yml
name: Train ML Model
on: [push]

permissions:
  id-token: write
  contents: read

jobs:
  train:
    runs-on: ubuntu-latest
    steps:
      - uses: azure/login@v1
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}

      - name: Submit training job
        run: |
          az ml job create --file job.yml \
            --workspace-name ${{ vars.WORKSPACE_NAME }}

Azure Monitor & Sentinel Integration

Enable diagnostic logs:

az monitor diagnostic-settings create \
  --name workspace-diagnostics \
  --resource /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.MachineLearningServices/workspaces/<workspace> \
  --logs '[{"category":"AmlComputeClusterEvent","enabled":true}]' \
  --workspace /subscriptions/<sub-id>/resourceGroups/<rg>/providers/Microsoft.OperationalInsights/workspaces/<log-analytics>

Sentinel KQL query for anomaly detection:

AmlComputeClusterNodeEvent
| where TimeGenerated > ago(24h)
| where EventType == "NodeStateChange"
| summarize NodeChanges = count() by NodeId, bin(TimeGenerated, 1h)
| where NodeChanges > 10  // Anomali: Mer enn 10 state changes per time

---

Offentlig sektor (Norge)

Compliance-krav

Krav fra Digitaliseringsdirektoratet:

1. Logging og sporbarhet (Referansekatalogen for IT-standarder):

   • Bruk Azure Monitor med minimum 90 dagers retention
   • Integrer med Sentinel for security event monitoring
   • Implementer audit trails for alle data access operations

2. Tilgangskontroll (NSM Grunnprinsipper for IKT-sikkerhet):

   • Multifaktor autentisering for alle brukerkontoer (Entra ID MFA)
   • Principle of least privilege (RBAC custom roles)
   • Regular access reviews (Entra ID Access Reviews)

3. Datakryptering:

   • TLS 1.2/1.3 for data in transit
   • Customer-managed keys for data at rest (anbefalt for "Begrenset" og høyere)
   • Key rotation procedures (minimum årlig)

Skytjenesteleverandør-vurdering (Difis krav)

Azure Machine Learning oppfyller:

• ✅ Databehandleravtale (DPA) med Microsoft
• ✅ ISO 27001, ISO 27018, SOC 2 Type II sertifiseringer
• ✅ GDPR compliance (EU data residency)
• ✅ Norway region availability (Oslo/Norway East)

Ekstra tiltak for "Begrenset" klassifiserte data:

• Bruk customer-managed keys
• Enable data isolation for multi-tenant scenarios
• Implementer private endpoints + Managed VNet
• Document data flows i ROS-analyse

---

Kostnad og lisensiering

Kostnadsdrivere for Security Features

Feature	Ekstra kostnad	Estimat (NOK/måned)
Private Endpoint	Per endpoint	~50 kr/endpoint + inbound/outbound data
VPN Gateway (S2S)	Gateway + bandwidth	~1500-5000 kr (avhengig av SKU)
Customer-Managed Keys	Cosmos DB, AI Search	+30-50% av workspace cost
Managed VNet	Inkludert	0 kr (ingen ekstra kostnad)
Azure Monitor logs	Per GB ingested	~25 kr/GB (etter 5 GB free tier)

Lisensiering

Ingen spesielle lisenser påkrevd for security features:

• Managed identities: Inkludert i Azure-abonnement
• RBAC: Inkludert i Azure-abonnement
• Private Link: Påløper kun infrastructure costs
• Customer-managed keys: Krever Azure Key Vault (standard/premium)

Microsoft Entra ID P2 (anbefalt for enterprise):

• Privileged Identity Management (PIM)
• Conditional Access policies
• Access Reviews
• Identity Protection

---

For arkitekten (Cosmo)

Anbefalte decision points i arkitekturgesprekker

1. Identity strategy:

• "Har dere delte storage accounts eller dedikerte per team?"
  • Delt → Bruk UAI + data isolation
  • Dedikert → SAI er tilstrekkelig

2. Network isolation level:

• "Hvilken klassifisering har dataene?" (Åpen/Intern/Begrenset/Fortrolig)
  • Begrenset+ → Private endpoints obligatorisk
  • Intern → Vurder managed VNet med approved outbound

3. Compliance requirements:

• "Har dere DPA med 3rd-party data processors?"
  • Ja → Implementer CMK for "data processor independence"
  • Nei → Vurder kostnad/kompleksitet trade-off

4. User vs. compute identity for data access:

• "Trenger dere audit trails per data scientist?"
  • Ja → User identity pass-through
  • Nei → Compute managed identity (enklere)

Red flags og mitigations

🚨 Red flag: "Vi har deaktivert firewall på storage account for å unngå connectivity issues"

• Risk: Data exfiltration, unauthorized access
• Mitigation: Implementer trusted service exception + private endpoints

🚨 Red flag: "Vi bruker storage account keys i environment variables"

• Risk: Credentials leakage i logs/telemetri
• Mitigation: Bytt til identity-based data access (no cached credentials)

🚨 Red flag: "Compute clusters har public IP for SSH-tilgang"

• Risk: Brute force attacks, lateral movement
• Mitigation: Disable public IP (enableNodePublicIp=false) + use Azure Bastion for mgmt

🚨 Red flag: "Vi har én workspace for både dev og prod"

• Risk: Privilege escalation, accidental production changes
• Mitigation: Separate workspaces med ulike RBAC policies + subscription boundaries

Typical architectures — security maturity levels

Level 1 — Prototype (minimal security):

• System-assigned managed identities
• Public endpoints
• Platform-managed keys
• Default RBAC roles
• Use case: PoC, hackathons, training environments

Level 2 — Development (basic security):

• User-assigned managed identities
• Managed VNet (allow internet outbound)
• Platform-managed keys
• Custom RBAC roles
• Diagnostic logs → Log Analytics
• Use case: Development teams, non-sensitive data

Level 3 — Production (enterprise security):

• User-assigned managed identities + data isolation
• Private endpoints + Managed VNet (approved outbound only)
• Customer-managed keys
• Conditional access policies
• Azure Monitor + Sentinel integration
• Regular access reviews
• Use case: Regulated industries, sensitive data, compliance requirements

---

Kilder og verifisering

MCP Calls: 8 (microsoft-learn docs search + fetch, code samples) Primærkilder:

1. Enterprise security and governance for Azure Machine Learning
2. Set up authentication between Azure Machine Learning and other services
3. Manage access to Azure Machine Learning workspaces
4. Azure security baseline for Machine Learning Service
5. Customer-managed keys for Azure Machine Learning
6. Configure a private endpoint for an Azure Machine Learning workspace
7. Secure an Azure Machine Learning workspace with virtual networks
8. Data encryption with Azure Machine Learning (Verified MCP 2026-04 — Key updates: Azure Data Lake Storage Gen1 retired 2024-02-29; Azure Database for PostgreSQL Single Server retired 2025-03-28; Azure Database for MySQL Single Server retired 2024-09-16. Use Gen2 / Flexible Server variants.)

Sist verifisert: 2026-04-10 Neste review: Q3 2026 (ved nye identity/network features i Azure ML)

---

Confidence markers i dette dokumentet:

• ✅ HIGH confidence: Offisiell dokumentasjon + kodeeksempler fra Microsoft Learn
• ⚠️ MEDIUM confidence: Utledet fra best practices og architecture patterns
• ❓ LOW confidence: Ikke aktuelt (alle påstander er verifisert mot offisiell dokumentasjon)

Azure Machine Learning VNet Security (2026 Update)

Managed Virtual Networks (recommended approach): Azure ML handles network isolation automatically. Use az ml workspace update with managed network settings instead of manual VNet configuration.

Private Endpoint for Workspace:

• Connects workspace via private IP addresses within your VNet
• Requires securing all dependent resources: Storage, Key Vault, Container Registry
• Private endpoint alone does NOT ensure end-to-end security — all components must be secured

Storage Account Security:

• Private endpoint (recommended): Blob, File, Queue, Table subresources
• Service endpoint: Must be same VNet and subnet as compute
• Set Microsoft.MachineLearningServices/Workspace as trusted resource type

Required outbound traffic service tags:

• AzureActiveDirectory (TCP 443) — authentication
• AzureMachineLearning (TCP 443, 18881, UDP 5831)
• Storage.region (TCP 443) — data access
• MicrosoftContainerRegistry.region (TCP 443) — Docker images

Secure connectivity options: Azure VPN Gateway (Point-to-site/Site-to-site), ExpressRoute, Azure Bastion (jump box)

ACR requirements: Premium SKU required for private endpoints; ACR must be in same VNet or peered VNet.