ms-ai-architect/skills/ms-ai-security/references/performance-scalability/gpu-compute-sizing.md

GPU and Compute Sizing for AI

Last updated: 2026-06-24 | Verified: MCP 2026-06 Status: GA Category: Performance & Scalability Type: reference Source: https://learn.microsoft.com/azure/foundry/openai/concepts/provisioned-throughput-billing

---

Innhold

• Introduksjon
• Kjernekomponenter
• GPU Type Comparison
• Memory Requirements
• Batch Size Influence
• Cost-Performance Analysis
• Azure ML Online Endpoints — oppdatert (2026-04)
• Norsk offentlig sektor
• Beslutningsrammeverk
• Referanser
• For Cosmo

Introduksjon

GPU- og compute-dimensjonering for AI-workloads på Azure handler om å velge riktig balanse mellom ytelse, kostnad og tilgjengelighet. For de fleste organisasjoner som bruker Azure OpenAI Service er GPU-valg abstrahert bak Provisioned Throughput Units (PTU) — du spesifiserer ønsket throughput, og Azure allokerer nødvendig GPU-kapasitet. Men for custom model hosting via Azure Machine Learning, Azure Kubernetes Service eller Azure Container Instances er eksplisitt GPU-valg nødvendig.

Azure tilbyr et bredt spekter av GPU-akselererte VM-serier: NC-serien (NVIDIA T4) for inferens, ND-serien (NVIDIA A100/H100) for trening, og NV-serien for visualisering. For AI-inferens er de viktigste faktorene GPU-minne (for modellstørrelse), compute throughput (TFLOPS), og minnebåndbredde (GB/s). For Azure OpenAI PTU-deployments håndterer Microsoft denne dimensjoneringen — din oppgave er å estimere PTU-behov basert på workload shape.

For norsk offentlig sektor er GPU-dimensjonering relevant ved deployment av open-source modeller, fine-tuned modeller som hostes on-premises eller i Azure ML, og ved evaluering av PTU vs. Standard deployments for Azure OpenAI.

Kjernekomponenter

Komponent	Formål	Teknologi
Azure OpenAI PTU	Managed GPU-kapasitet for OpenAI-modeller	Azure OpenAI
NC-series VMs	NVIDIA T4 — kostnadseffektiv inferens	Azure VMs
ND-series VMs	NVIDIA A100/H100 — trening og stor-modell inferens	Azure VMs
Azure ML Endpoints	Managed inferens med GPU-akselerasjon	Azure ML
Azure Container Apps	GPU-støtte for containerisert AI	Azure Container Apps
Capacity Calculator	PTU-estimering verktøy	Microsoft Foundry

GPU Type Comparison

Azure GPU VM-serier for AI

VM-serie	GPU	GPU-minne	Use case	Pris-segment
NC4as_T4_v3	1x NVIDIA T4	16 GB	Liten modell-inferens	Lavest
NC24ads_A100_v4	1x NVIDIA A100	80 GB	Medium modell-inferens/trening	Middels
NC96ads_A100_v4	4x NVIDIA A100	320 GB	Stor modell-trening	Høy
ND96asr_v4	8x NVIDIA A100 (40 GB)	320 GB	LLM-trening, multi-GPU	Svært høy
ND96isr_H100_v5	8x NVIDIA H100	640 GB	Frontier-modell trening	Høyest
NC40ads_H100_v5	1x NVIDIA H100	80 GB	Stor modell-inferens	Høy

Modellstørrelse og GPU-krav

def estimate_gpu_requirements(
    model_params_billions: float,
    precision: str = "fp16",  # fp32, fp16, int8, int4
    batch_size: int = 1,
    sequence_length: int = 4096,
    overhead_factor: float = 1.2  # 20% overhead for KV-cache etc.
) -> dict:
    """Estimate GPU memory requirements for model inference."""

    bytes_per_param = {
        "fp32": 4,
        "fp16": 2,
        "bf16": 2,
        "int8": 1,
        "int4": 0.5
    }

    if precision not in bytes_per_param:
        raise ValueError(f"Unknown precision: {precision}")

    # Modellvekter
    model_memory_gb = (
        model_params_billions * 1e9 *
        bytes_per_param[precision] / 1e9
    )

    # KV-cache (estimat)
    # KV cache ≈ 2 * num_layers * hidden_dim * seq_len * batch * bytes
    # Forenklet estimat: ~10% av modellstørrelse per 4K context
    kv_cache_gb = model_memory_gb * 0.1 * (sequence_length / 4096) * batch_size

    # Total med overhead
    total_gb = (model_memory_gb + kv_cache_gb) * overhead_factor

    # Anbefalt GPU
    gpu_options = [
        {"name": "T4", "memory_gb": 16, "tflops_fp16": 65},
        {"name": "A10G", "memory_gb": 24, "tflops_fp16": 125},
        {"name": "A100 40GB", "memory_gb": 40, "tflops_fp16": 312},
        {"name": "A100 80GB", "memory_gb": 80, "tflops_fp16": 312},
        {"name": "H100 80GB", "memory_gb": 80, "tflops_fp16": 989},
    ]

    suitable_gpus = []
    for gpu in gpu_options:
        gpus_needed = max(1, int(total_gb / gpu["memory_gb"]) + 1)
        if gpus_needed <= 8:  # Max 8 GPUs per node
            suitable_gpus.append({
                "gpu": gpu["name"],
                "gpus_needed": gpus_needed,
                "total_memory_gb": gpus_needed * gpu["memory_gb"],
                "headroom_gb": round(
                    gpus_needed * gpu["memory_gb"] - total_gb, 1)
            })

    return {
        "model_params_b": model_params_billions,
        "precision": precision,
        "model_memory_gb": round(model_memory_gb, 1),
        "kv_cache_gb": round(kv_cache_gb, 1),
        "total_required_gb": round(total_gb, 1),
        "suitable_gpus": suitable_gpus
    }

# Eksempler
print(estimate_gpu_requirements(7, "fp16"))   # Llama 3 8B — 1x T4
print(estimate_gpu_requirements(70, "int8"))  # Llama 3 70B — 1x A100 80GB
print(estimate_gpu_requirements(70, "fp16"))  # Llama 3 70B — 2x A100 80GB

Memory Requirements

GPU-minne budsjett for inferens

Total GPU-minne behov:
┌─────────────────────────────────────────┐
│ Modellvekter (størst posten)            │
│ ├── FP16: params * 2 bytes              │
│ ├── INT8: params * 1 byte               │
│ └── INT4: params * 0.5 bytes            │
│                                         │
│ KV-cache (vokser med context length)    │
│ ├── Per token: ~0.5-2 KB (avh. modell)  │
│ └── 128K context: kan bli flere GB      │
│                                         │
│ Aktiverings-minne (batch-avhengig)      │
│ ├── Skalerer lineært med batch size     │
│ └── Typisk 5-15% av modellstørrelse     │
│                                         │
│ Overhead (CUDA, framework)              │
│ └── Typisk 10-20% ekstra               │
└─────────────────────────────────────────┘

Azure ML Deployment Sizing

# Azure ML endpoint deployment med GPU
from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineDeployment,
    ManagedOnlineEndpoint
)

# Definer endpoint
endpoint = ManagedOnlineEndpoint(
    name="llama-inference",
    auth_mode="key"
)

# GPU-deployment basert på modellstørrelse
deployment_configs = {
    "small_model": {  # 7B parameters
        "instance_type": "Standard_NC4as_T4_v3",
        "instance_count": 1,
        "model_format": "int8",
        "expected_tps": 30
    },
    "medium_model": {  # 13B-34B parameters
        "instance_type": "Standard_NC24ads_A100_v4",
        "instance_count": 1,
        "model_format": "fp16",
        "expected_tps": 25
    },
    "large_model": {  # 70B parameters
        "instance_type": "Standard_NC48ads_A100_v4",
        "instance_count": 1,  # 2x A100 80GB
        "model_format": "int8",
        "expected_tps": 15
    }
}

# Deploy
deployment = ManagedOnlineDeployment(
    name="llama-70b-int8",
    endpoint_name=endpoint.name,
    model="azureml://registries/azureml-meta/models/Llama-3.3-70B-Instruct",
    instance_type="Standard_NC48ads_A100_v4",
    instance_count=2,  # For high availability
    request_settings={
        "request_timeout_ms": 120000,
        "max_concurrent_requests_per_instance": 10
    },
    liveness_probe={"initial_delay": 600},
    environment_variables={
        "TENSOR_PARALLEL_SIZE": "2",  # Shard modell over 2 GPUs
        "MAX_MODEL_LEN": "8192",
        "GPU_MEMORY_UTILIZATION": "0.9"
    }
)

Batch Size Influence

Batch size vs. throughput vs. latens

def model_batch_size_tradeoff(
    gpu_memory_gb: float,
    model_memory_gb: float,
    kv_cache_per_request_gb: float,
    target_latency_ms: float
) -> dict:
    """Model the relationship between batch size and performance."""

    available_memory = gpu_memory_gb - model_memory_gb
    max_batch_from_memory = int(available_memory / kv_cache_per_request_gb)

    results = []
    for batch_size in range(1, min(max_batch_from_memory + 1, 65)):
        # Throughput øker med batch size (GPU utilization)
        # Men per-request latens øker også
        memory_used = model_memory_gb + (
            batch_size * kv_cache_per_request_gb)
        utilization = min(memory_used / gpu_memory_gb, 0.95)

        # Throughput skalerer sub-lineært med batch size
        throughput_factor = batch_size ** 0.7  # Empirisk
        latency_factor = 1 + (batch_size - 1) * 0.15  # +15% per ekstra request

        estimated_latency = target_latency_ms * latency_factor

        results.append({
            "batch_size": batch_size,
            "memory_gb": round(memory_used, 1),
            "utilization_pct": round(utilization * 100, 1),
            "relative_throughput": round(throughput_factor, 2),
            "estimated_latency_ms": round(estimated_latency)
        })

    # Finn sweet spot: beste throughput innenfor latens-krav
    acceptable = [
        r for r in results
        if r["estimated_latency_ms"] <= target_latency_ms * 3
    ]
    optimal = max(acceptable, key=lambda r: r["relative_throughput"])

    return {
        "max_batch_from_memory": max_batch_from_memory,
        "optimal_batch_size": optimal["batch_size"],
        "all_results": results[:10]  # Første 10
    }

# A100 80GB med 70B modell i INT8
result = model_batch_size_tradeoff(
    gpu_memory_gb=80,
    model_memory_gb=35,  # 70B * 0.5 bytes (INT8 ≈ 0.5)
    kv_cache_per_request_gb=0.5,  # Per 4K context
    target_latency_ms=2000
)
print(f"Optimal batch size: {result['optimal_batch_size']}")

Cost-Performance Analysis

PTU vs. Standard vs. Self-hosted

def compare_deployment_options(
    monthly_input_tokens_m: float,  # Millioner input tokens
    monthly_output_tokens_m: float,
    avg_latency_requirement_ms: float = 2000,
    model: str = "gpt-4o"
) -> dict:
    """Compare cost-performance of deployment options."""

    # Priser (estimater i USD)
    pricing = {
        "gpt-4o": {
            "standard_input_per_1m": 2.50,
            "standard_output_per_1m": 10.00,
            "ptu_monthly_per_unit": 990,
            "input_tpm_per_ptu": 2500,
            "self_hosted_alternative": None
        },
        "gpt-4.1": {
            "standard_input_per_1m": 2.00,
            "standard_output_per_1m": 8.00,
            "ptu_monthly_per_unit": 990,
            "input_tpm_per_ptu": 3000,
            "self_hosted_alternative": None
        },
        "llama-70b": {
            "standard_input_per_1m": 0.00,  # Self-hosted
            "standard_output_per_1m": 0.00,
            "ptu_monthly_per_unit": 0,
            "vm_monthly_cost": 15000,  # NC48ads_A100_v4
            "self_hosted_alternative": "Standard_NC48ads_A100_v4"
        }
    }

    p = pricing.get(model, pricing["gpt-4o"])

    # Standard (pay-per-token)
    standard_cost = (
        monthly_input_tokens_m * p["standard_input_per_1m"] +
        monthly_output_tokens_m * p["standard_output_per_1m"]
    )

    # PTU
    total_tpm_needed = (
        (monthly_input_tokens_m * 1e6 + monthly_output_tokens_m * 1e6 * 4) /
        (30 * 24 * 60)  # Spread over month
    )
    ptus_needed = max(50, int(total_tpm_needed / p.get("input_tpm_per_ptu", 1)) + 1)
    ptu_cost = ptus_needed * p.get("ptu_monthly_per_unit", 0)

    return {
        "model": model,
        "standard_monthly_usd": round(standard_cost, 2),
        "standard_monthly_nok": round(standard_cost * 11, 2),
        "ptu_units": ptus_needed,
        "ptu_monthly_usd": round(ptu_cost, 2),
        "ptu_monthly_nok": round(ptu_cost * 11, 2),
        "ptu_savings_pct": round(
            (1 - ptu_cost / max(standard_cost, 1)) * 100, 1)
            if ptu_cost > 0 else "N/A",
        "recommendation": (
            "PTU" if ptu_cost < standard_cost * 0.8 else
            "Standard" if standard_cost < ptu_cost else
            "Evaluate self-hosted")
    }

Azure ML Online Endpoints — oppdatert (2026-04)

Azure ML Online Endpoints har to deployment-typer:

Type	Infrastruktur	Administrasjon	Bruksscenario
Managed Online Endpoint	Azure-administrert	Minimal	Raskest å komme i gang, serverless
Kubernetes Online Endpoint	Kundeeid K8s-kluster	Full kontroll	On-premises, hybrid, spesielle krav

Anbefalt arbeidsflyt: Lokal debug → Azure deploy

# Steg 1: Test deployment lokalt
from azure.ai.ml import MLClient
from azure.ai.ml.entities import (
    ManagedOnlineEndpoint,
    ManagedOnlineDeployment,
    Model,
    Environment
)
from azure.identity import DefaultAzureCredential

# Lokal testing med Azure ML SDK
import subprocess
result = subprocess.run([
    "az", "ml", "online-endpoint", "create",
    "--local",
    "--name", "my-endpoint",
    "--file", "endpoint.yaml"
], capture_output=True, text=True)

# Steg 2: Deploy til Azure (ManagedOnlineDeployment)
ml_client = MLClient(
    credential=DefaultAzureCredential(),
    subscription_id="...",
    resource_group_name="rg-ai",
    workspace_name="my-ml-workspace"
)

endpoint = ManagedOnlineEndpoint(
    name="my-production-endpoint",
    description="GPU-akselerert inferens",
    auth_mode="key"
)
ml_client.online_endpoints.begin_create_or_update(endpoint).result()

# ManagedOnlineDeployment: spesifiser instance_type for GPU
deployment = ManagedOnlineDeployment(
    name="blue",
    endpoint_name="my-production-endpoint",
    model="azureml:my-model:1",
    instance_type="Standard_NC24ads_A100_v4",  # A100 GPU
    instance_count=2,
    environment="azureml:my-environment:1",
    request_settings={
        "max_concurrent_requests_per_instance": 4,
        "request_timeout_ms": 90000
    }
)
ml_client.online_deployments.begin_create_or_update(deployment).result()

GPU-instanstyper for inferens (2026-04)

SKU	GPU	VRAM	Bruksscenario
Standard_NC6s_v3	V100 (1x)	16 GB	Medium modeller
Standard_NC24s_v3	V100 (4x)	64 GB	Større modeller
Standard_NC24ads_A100_v4	A100 (1x)	80 GB	Store modeller (7B-13B)
Standard_ND96amsr_A100_v4	A100 (8x)	640 GB	Meget store modeller (70B+)

Norsk offentlig sektor

• Anskaffelse: GPU VM-er er kostbare — bruk Azure Reserved Instances (1-3 år) for 40-60% besparelse på forutsigbare workloads. Krever godkjenning i anskaffelsesprosess.
• Data residency: GPU VMs er tilgjengelige i Norway East for self-hosted modeller. Azure OpenAI PTU-deployments har regional, data zone og global variant.
• Kapasitetsrisiko: GPU-kapasitet i Azure kan være begrenset — bestill PTU og GPU VMs i god tid, spesielt for større deployments.
• Open source: For organisasjoner som ønsker full kontroll, er self-hosted Llama eller DeepSeek på Azure ML et alternativ, men krever mer operasjonelt vedlikehold.
• Sikkerhet: Self-hosted modeller gir full kontroll over data — ingen data sendes til tredjepart. Relevant for gradert informasjon.

Beslutningsrammeverk

Scenario	Anbefaling	Begrunnelse
Azure OpenAI, forutsigbar last	PTU deployment	Dedikert kapasitet, forutsigbar kostnad
Azure OpenAI, variabel last	Standard deployment	Betal per bruk, ingen forpliktelse
Full datakontroll krav	Self-hosted via Azure ML	Ingen data til tredjepart
Modell < 13B parameters	NC4as_T4_v3 (T4)	Kostnadseffektiv for små modeller
Modell 13B-70B parameters	NC24ads_A100_v4	Tilstrekkelig minne og compute
Modell > 70B parameters	ND96asr (multi-GPU)	Krever tensor parallelism

Referanser

• What is provisioned throughput? — PTU oversikt
• PTU costs and billing — PTU-prising per modell
• Foundry PTU calculator — Kapasitetskalkulator
• GPU optimized VM sizes — Azure GPU VM-oversikt
• Deploy models in Azure ML — ML endpoint deployment

For Cosmo

• Bruk denne referansen når kunden trenger å velge mellom PTU og Standard for Azure OpenAI, eller når de vurderer self-hosted modeller.
• For de fleste norske offentlige organisasjoner er Azure OpenAI PTU det riktige valget — unngå overhead med GPU-management med mindre datakontroll er et absolutt krav.
• PTU gir forutsigbar kostnad og ytelse — gpt-4.1-nano med 59,400 input TPM per PTU er ekstremt kostnadseffektivt for enkle oppgaver.
• Ved self-hosting: INT8 kvantisering halverer minnebehovet med minimal kvalitetstap — anbefal dette for produksjonsinferens.
• Alltid benchmark med reell workload før produksjonsdeployment — teoretiske beregninger gir bare estimater.