ms-ai-architect/skills/ms-ai-security/references/performance-scalability/gpu-compute-sizing.md

# GPU and Compute Sizing for AI

**Last updated:** 2026-06-24 | Verified: MCP 2026-06
**Status:** GA
**Category:** Performance & Scalability
**Type:** reference
**Source:** https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput-billing

---

## Innhold

- [Introduksjon](#introduksjon)
- [Kjernekomponenter](#kjernekomponenter)
- [GPU Type Comparison](#gpu-type-comparison)
- [Memory Requirements](#memory-requirements)
- [Batch Size Influence](#batch-size-influence)
- [Cost-Performance Analysis](#cost-performance-analysis)
- [Azure ML Online Endpoints — oppdatert (2026-04)](#azure-ml-online-endpoints--oppdatert-2026-04)
- [Norsk offentlig sektor](#norsk-offentlig-sektor)
- [Beslutningsrammeverk](#beslutningsrammeverk)
- [Referanser](#referanser)
- [For Cosmo](#for-cosmo)

## Introduksjon

GPU- og compute-dimensjonering for AI-workloads på Azure handler om å velge riktig balanse mellom ytelse, kostnad og tilgjengelighet. For de fleste organisasjoner som bruker Azure OpenAI Service er GPU-valg abstrahert bak Provisioned Throughput Units (PTU) — du spesifiserer ønsket throughput, og Azure allokerer nødvendig GPU-kapasitet. Men for custom model hosting via Azure Machine Learning, Azure Kubernetes Service eller Azure Container Instances er eksplisitt GPU-valg nødvendig.

Azure tilbyr et bredt spekter av GPU-akselererte VM-serier: NC-serien (NVIDIA T4) for inferens, ND-serien (NVIDIA A100/H100) for trening, og NV-serien for visualisering. For AI-inferens er de viktigste faktorene GPU-minne (for modellstørrelse), compute throughput (TFLOPS), og minnebåndbredde (GB/s). For Azure OpenAI PTU-deployments håndterer Microsoft denne dimensjoneringen — din oppgave er å estimere PTU-behov basert på workload shape.

For norsk offentlig sektor er GPU-dimensjonering relevant ved deployment av open-source modeller, fine-tuned modeller som hostes on-premises eller i Azure ML, og ved evaluering av PTU vs. Standard deployments for Azure OpenAI.

## Kjernekomponenter

| Komponent | Formål | Teknologi |
|-----------|--------|-----------|
| Azure OpenAI PTU | Managed GPU-kapasitet for OpenAI-modeller | Azure OpenAI |
| NC-series VMs | NVIDIA T4 — kostnadseffektiv inferens | Azure VMs |
| ND-series VMs | NVIDIA A100/H100 — trening og stor-modell inferens | Azure VMs |
| Azure ML Endpoints | Managed inferens med GPU-akselerasjon | Azure ML |
| Azure Container Apps | GPU-støtte for containerisert AI | Azure Container Apps |
| Capacity Calculator | PTU-estimering verktøy | Microsoft Foundry |

## GPU Type Comparison

### Azure GPU VM-serier for AI

| VM-serie | GPU | GPU-minne | Use case | Pris-segment |
|----------|-----|-----------|----------|-------------|
| NC4as_T4_v3 | 1x NVIDIA T4 | 16 GB | Liten modell-inferens | Lavest |
| NC24ads_A100_v4 | 1x NVIDIA A100 | 80 GB | Medium modell-inferens/trening | Middels |
| NC96ads_A100_v4 | 4x NVIDIA A100 | 320 GB | Stor modell-trening | Høy |
| ND96asr_v4 | 8x NVIDIA A100 (40 GB) | 320 GB | LLM-trening, multi-GPU | Svært høy |
| ND96isr_H100_v5 | 8x NVIDIA H100 | 640 GB | Frontier-modell trening | Høyest |
| NC40ads_H100_v5 | 1x NVIDIA H100 | 80 GB | Stor modell-inferens | Høy |

### Modellstørrelse og GPU-krav

```python
def estimate_gpu_requirements(
    model_params_billions: float,
    precision: str = "fp16",  # fp32, fp16, int8, int4
    batch_size: int = 1,
    sequence_length: int = 4096,
    overhead_factor: float = 1.2  # 20% overhead for KV-cache etc.
) -> dict:
    """Estimate GPU memory requirements for model inference."""

    bytes_per_param = {
        "fp32": 4,
        "fp16": 2,
        "bf16": 2,
        "int8": 1,
        "int4": 0.5
    }

    if precision not in bytes_per_param:
        raise ValueError(f"Unknown precision: {precision}")

    # Modellvekter
    model_memory_gb = (
        model_params_billions * 1e9 *
        bytes_per_param[precision] / 1e9
    )

    # KV-cache (estimat)
    # KV cache ≈ 2 * num_layers * hidden_dim * seq_len * batch * bytes
    # Forenklet estimat: ~10% av modellstørrelse per 4K context
    kv_cache_gb = model_memory_gb * 0.1 * (sequence_length / 4096) * batch_size

    # Total med overhead
    total_gb = (model_memory_gb + kv_cache_gb) * overhead_factor

    # Anbefalt GPU
    gpu_options = [
        {"name": "T4", "memory_gb": 16, "tflops_fp16": 65},
        {"name": "A10G", "memory_gb": 24, "tflops_fp16": 125},
        {"name": "A100 40GB", "memory_gb": 40, "tflops_fp16": 312},
        {"name": "A100 80GB", "memory_gb": 80, "tflops_fp16": 312},
        {"name": "H100 80GB", "memory_gb": 80, "tflops_fp16": 989},
    ]

    suitable_gpus = []
    for gpu in gpu_options:
        gpus_needed = max(1, int(total_gb / gpu["memory_gb"]) + 1)
        if gpus_needed <= 8:  # Max 8 GPUs per node
            suitable_gpus.append({
                "gpu": gpu["name"],
                "gpus_needed": gpus_needed,
                "total_memory_gb": gpus_needed * gpu["memory_gb"],
                "headroom_gb": round(
                    gpus_needed * gpu["memory_gb"] - total_gb, 1)
            })

    return {
        "model_params_b": model_params_billions,
        "precision": precision,
        "model_memory_gb": round(model_memory_gb, 1),
        "kv_cache_gb": round(kv_cache_gb, 1),
        "total_required_gb": round(total_gb, 1),
        "suitable_gpus": suitable_gpus
    }

# Eksempler
print(estimate_gpu_requirements(7, "fp16"))   # Llama 3 8B — 1x T4
print(estimate_gpu_requirements(70, "int8"))  # Llama 3 70B — 1x A100 80GB
print(estimate_gpu_requirements(70, "fp16"))  # Llama 3 70B — 2x A100 80GB
```

## Memory Requirements

### GPU-minne budsjett for inferens

```
Total GPU-minne behov:
┌─────────────────────────────────────────┐
│ Modellvekter (størst posten)            │
│ ├── FP16: params * 2 bytes              │
│ ├── INT8: params * 1 byte               │
│ └── INT4: params * 0.5 bytes            │
│                                         │
│ KV-cache (vokser med context length)    │
│ ├── Per token: ~0.5-2 KB (avh. modell)  │
│ └── 128K context: kan bli flere GB      │
│                                         │
│ Aktiverings-minne (batch-avhengig)      │
│ ├── Skalerer lineært med batch size     │
│ └── Typisk 5-15% av modellstørrelse     │
│                                         │
│ Overhead (CUDA, framework)              │
│ └── Typisk 10-20% ekstra               │
└─────────────────────────────────────────┘
```

### Azure ML Deployment Sizing

```python
# Azure ML endpoint deployment med GPU
from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineDeployment,
    ManagedOnlineEndpoint
)

# Definer endpoint
endpoint = ManagedOnlineEndpoint(
    name="llama-inference",
    auth_mode="key"
)

# GPU-deployment basert på modellstørrelse
deployment_configs = {
    "small_model": {  # 7B parameters
        "instance_type": "Standard_NC4as_T4_v3",
        "instance_count": 1,
        "model_format": "int8",
        "expected_tps": 30
    },
    "medium_model": {  # 13B-34B parameters
        "instance_type": "Standard_NC24ads_A100_v4",
        "instance_count": 1,
        "model_format": "fp16",
        "expected_tps": 25
    },
    "large_model": {  # 70B parameters
        "instance_type": "Standard_NC48ads_A100_v4",
        "instance_count": 1,  # 2x A100 80GB
        "model_format": "int8",
        "expected_tps": 15
    }
}

# Deploy
deployment = ManagedOnlineDeployment(
    name="llama-70b-int8",
    endpoint_name=endpoint.name,
    model="azureml://registries/azureml-meta/models/Llama-3.3-70B-Instruct",
    instance_type="Standard_NC48ads_A100_v4",
    instance_count=2,  # For high availability
    request_settings={
        "request_timeout_ms": 120000,
        "max_concurrent_requests_per_instance": 10
    },
    liveness_probe={"initial_delay": 600},
    environment_variables={
        "TENSOR_PARALLEL_SIZE": "2",  # Shard modell over 2 GPUs
        "MAX_MODEL_LEN": "8192",
        "GPU_MEMORY_UTILIZATION": "0.9"
    }
)
```

## Batch Size Influence

### Batch size vs. throughput vs. latens

```python
def model_batch_size_tradeoff(
    gpu_memory_gb: float,
    model_memory_gb: float,
    kv_cache_per_request_gb: float,
    target_latency_ms: float
) -> dict:
    """Model the relationship between batch size and performance."""

    available_memory = gpu_memory_gb - model_memory_gb
    max_batch_from_memory = int(available_memory / kv_cache_per_request_gb)

    results = []
    for batch_size in range(1, min(max_batch_from_memory + 1, 65)):
        # Throughput øker med batch size (GPU utilization)
        # Men per-request latens øker også
        memory_used = model_memory_gb + (
            batch_size * kv_cache_per_request_gb)
        utilization = min(memory_used / gpu_memory_gb, 0.95)

        # Throughput skalerer sub-lineært med batch size
        throughput_factor = batch_size ** 0.7  # Empirisk
        latency_factor = 1 + (batch_size - 1) * 0.15  # +15% per ekstra request

        estimated_latency = target_latency_ms * latency_factor

        results.append({
            "batch_size": batch_size,
            "memory_gb": round(memory_used, 1),
            "utilization_pct": round(utilization * 100, 1),
            "relative_throughput": round(throughput_factor, 2),
            "estimated_latency_ms": round(estimated_latency)
        })

    # Finn sweet spot: beste throughput innenfor latens-krav
    acceptable = [
        r for r in results
        if r["estimated_latency_ms"] <= target_latency_ms * 3
    ]
    optimal = max(acceptable, key=lambda r: r["relative_throughput"])

    return {
        "max_batch_from_memory": max_batch_from_memory,
        "optimal_batch_size": optimal["batch_size"],
        "all_results": results[:10]  # Første 10
    }

# A100 80GB med 70B modell i INT8
result = model_batch_size_tradeoff(
    gpu_memory_gb=80,
    model_memory_gb=35,  # 70B * 0.5 bytes (INT8 ≈ 0.5)
    kv_cache_per_request_gb=0.5,  # Per 4K context
    target_latency_ms=2000
)
print(f"Optimal batch size: {result['optimal_batch_size']}")
```

## Cost-Performance Analysis

### PTU vs. Standard vs. Self-hosted

```python
def compare_deployment_options(
    monthly_input_tokens_m: float,  # Millioner input tokens
    monthly_output_tokens_m: float,
    avg_latency_requirement_ms: float = 2000,
    model: str = "gpt-4o"
) -> dict:
    """Compare cost-performance of deployment options."""

    # Priser (estimater i USD)
    pricing = {
        "gpt-4o": {
            "standard_input_per_1m": 2.50,
            "standard_output_per_1m": 10.00,
            "ptu_monthly_per_unit": 990,
            "input_tpm_per_ptu": 2500,
            "self_hosted_alternative": None
        },
        "gpt-4.1": {
            "standard_input_per_1m": 2.00,
            "standard_output_per_1m": 8.00,
            "ptu_monthly_per_unit": 990,
            "input_tpm_per_ptu": 3000,
            "self_hosted_alternative": None
        },
        "llama-70b": {
            "standard_input_per_1m": 0.00,  # Self-hosted
            "standard_output_per_1m": 0.00,
            "ptu_monthly_per_unit": 0,
            "vm_monthly_cost": 15000,  # NC48ads_A100_v4
            "self_hosted_alternative": "Standard_NC48ads_A100_v4"
        }
    }

    p = pricing.get(model, pricing["gpt-4o"])

    # Standard (pay-per-token)
    standard_cost = (
        monthly_input_tokens_m * p["standard_input_per_1m"] +
        monthly_output_tokens_m * p["standard_output_per_1m"]
    )

    # PTU
    total_tpm_needed = (
        (monthly_input_tokens_m * 1e6 + monthly_output_tokens_m * 1e6 * 4) /
        (30 * 24 * 60)  # Spread over month
    )
    ptus_needed = max(50, int(total_tpm_needed / p.get("input_tpm_per_ptu", 1)) + 1)
    ptu_cost = ptus_needed * p.get("ptu_monthly_per_unit", 0)

    return {
        "model": model,
        "standard_monthly_usd": round(standard_cost, 2),
        "standard_monthly_nok": round(standard_cost * 11, 2),
        "ptu_units": ptus_needed,
        "ptu_monthly_usd": round(ptu_cost, 2),
        "ptu_monthly_nok": round(ptu_cost * 11, 2),
        "ptu_savings_pct": round(
            (1 - ptu_cost / max(standard_cost, 1)) * 100, 1)
            if ptu_cost > 0 else "N/A",
        "recommendation": (
            "PTU" if ptu_cost < standard_cost * 0.8 else
            "Standard" if standard_cost < ptu_cost else
            "Evaluate self-hosted")
    }
```


## Azure ML Online Endpoints — oppdatert (2026-04)

Azure ML Online Endpoints har to deployment-typer:

| Type | Infrastruktur | Administrasjon | Bruksscenario |
|------|---------------|----------------|---------------|
| Managed Online Endpoint | Azure-administrert | Minimal | Raskest å komme i gang, serverless |
| Kubernetes Online Endpoint | Kundeeid K8s-kluster | Full kontroll | On-premises, hybrid, spesielle krav |

### Anbefalt arbeidsflyt: Lokal debug → Azure deploy

```python
# Steg 1: Test deployment lokalt
from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Model,
    Environment
)
from azure.identity import DefaultAzureCredential

# Lokal testing med Azure ML SDK
import subprocess
result = subprocess.run([
    "az", "ml", "online-endpoint", "create",
    "--local",
    "--name", "my-endpoint",
    "--file", "endpoint.yaml"
], capture_output=True, text=True)

# Steg 2: Deploy til Azure (ManagedOnlineDeployment)
ml_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id="...",
    resource_group_name="rg-ai",
    workspace_name="my-ml-workspace"
)

endpoint = ManagedOnlineEndpoint(
    name="my-production-endpoint",
    description="GPU-akselerert inferens",
    auth_mode="key"
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# ManagedOnlineDeployment: spesifiser instance_type for GPU
deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name="my-production-endpoint",
    model="azureml:my-model:1",
    instance_type="Standard_NC24ads_A100_v4",  # A100 GPU
    instance_count=2,
    environment="azureml:my-environment:1",
    request_settings={
        "max_concurrent_requests_per_instance": 4,
        "request_timeout_ms": 90000
    }
)
ml_client.online_deployments.begin_create_or_update(deployment).result()
```

### GPU-instanstyper for inferens (2026-04)

| SKU | GPU | VRAM | Bruksscenario |
|-----|-----|------|---------------|
| `Standard_NC6s_v3` | V100 (1x) | 16 GB | Medium modeller |
| `Standard_NC24s_v3` | V100 (4x) | 64 GB | Større modeller |
| `Standard_NC24ads_A100_v4` | A100 (1x) | 80 GB | Store modeller (7B-13B) |
| `Standard_ND96amsr_A100_v4` | A100 (8x) | 640 GB | Meget store modeller (70B+) |

## Norsk offentlig sektor

- **Anskaffelse**: GPU VM-er er kostbare — bruk Azure Reserved Instances (1-3 år) for 40-60% besparelse på forutsigbare workloads. Krever godkjenning i anskaffelsesprosess.
- **Data residency**: GPU VMs er tilgjengelige i Norway East for self-hosted modeller. Azure OpenAI PTU-deployments har regional, data zone og global variant.
- **Kapasitetsrisiko**: GPU-kapasitet i Azure kan være begrenset — bestill PTU og GPU VMs i god tid, spesielt for større deployments.
- **Open source**: For organisasjoner som ønsker full kontroll, er self-hosted Llama eller DeepSeek på Azure ML et alternativ, men krever mer operasjonelt vedlikehold.
- **Sikkerhet**: Self-hosted modeller gir full kontroll over data — ingen data sendes til tredjepart. Relevant for gradert informasjon.

## Beslutningsrammeverk

| Scenario | Anbefaling | Begrunnelse |
|----------|------------|-------------|
| Azure OpenAI, forutsigbar last | PTU deployment | Dedikert kapasitet, forutsigbar kostnad |
| Azure OpenAI, variabel last | Standard deployment | Betal per bruk, ingen forpliktelse |
| Full datakontroll krav | Self-hosted via Azure ML | Ingen data til tredjepart |
| Modell < 13B parameters | NC4as_T4_v3 (T4) | Kostnadseffektiv for små modeller |
| Modell 13B-70B parameters | NC24ads_A100_v4 | Tilstrekkelig minne og compute |
| Modell > 70B parameters | ND96asr (multi-GPU) | Krever tensor parallelism |

## Referanser

- [What is provisioned throughput?](https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput) — PTU oversikt
- [PTU costs and billing](https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput-billing) — PTU-prising per modell
- [Foundry PTU calculator](https://ai.azure.com/resource/calculator) — Kapasitetskalkulator
- [GPU optimized VM sizes](https://learn.microsoft.com/azure/virtual-machines/sizes-gpu) — Azure GPU VM-oversikt
- [Deploy models in Azure ML](https://learn.microsoft.com/azure/machine-learning/how-to-deploy-online-endpoints) — ML endpoint deployment

## For Cosmo

- **Bruk denne referansen** når kunden trenger å velge mellom PTU og Standard for Azure OpenAI, eller når de vurderer self-hosted modeller.
- For de fleste norske offentlige organisasjoner er Azure OpenAI PTU det riktige valget — unngå overhead med GPU-management med mindre datakontroll er et absolutt krav.
- PTU gir forutsigbar kostnad og ytelse — gpt-4.1-nano med 59,400 input TPM per PTU er ekstremt kostnadseffektivt for enkle oppgaver.
- Ved self-hosting: INT8 kvantisering halverer minnebehovet med minimal kvalitetstap — anbefal dette for produksjonsinferens.
- Alltid benchmark med reell workload før produksjonsdeployment — teoretiske beregninger gir bare estimater.