diff --git a/skills/ms-ai-security/references/cost-optimization/request-batching-aggregation.md b/skills/ms-ai-security/references/cost-optimization/request-batching-aggregation.md index df63a71..2cde855 100644 --- a/skills/ms-ai-security/references/cost-optimization/request-batching-aggregation.md +++ b/skills/ms-ai-security/references/cost-optimization/request-batching-aggregation.md @@ -1,6 +1,6 @@ # Request Batching and Response Aggregation -**Last updated:** 2026-05 +**Last updated:** 2026-06 **Status:** GA **Category:** Cost Optimization & FinOps for AI @@ -24,10 +24,11 @@ Denne teknikken er ikke bare en optimaliseringsøvelse — den er ofte nødvendi |-----------|-------------| | **Input file (JSONL)** | JSON Lines-format med én request per linje, inkludert `custom_id` for korrelasjon | | **Global-Batch deployment** | Dedikert deployment-type med 50% lavere pris enn global standard | +| **Data Zone Batch deployment** | Batch-deployment-type (SKU `DataZoneBatch`) med samme 50% rabatt, men inferens-prosessering begrenset til Microsofts definerte datasone (EU eller US) — relevant for dataresidens | | **Enqueued token quota** | Separat quota som ikke forstyrrer online workloads | | **24-hour completion window** | Target SLA for batch processing (jobs kan ta lenger, men utløper ikke) | | **Output file** | JSONL-resultatfil med responses korrelert via `custom_id` | -| **Exponential backoff queuing** | Støtte i utvalgte regioner for automatisk retry når token quota er tilgjengelig | +| **Fail-fast token-quota-håndtering** | I utvalgte regioner feiler nye jobber raskt (fail fast) når enqueued-token-grensen overskrides, slik at klienten programmatisk kan kø-stille og retry-e med exponential backoff | ### Microsoft Graph JSON Batching @@ -230,14 +231,14 @@ def batch_processor(msgs: List[func.ServiceBusMessage]): ### Azure OpenAI Batch API **Setup:** -1. Opprett Global-Batch deployment i Azure OpenAI resource +1. Opprett Global-Batch eller Data Zone Batch deployment i Azure OpenAI resource 2. Prepare JSONL input file med standard chat completion format + `custom_id` 3. Upload file via Files API eller Azure Blob Storage 4. Create batch job med `client.batches.create()` 5. Poll job status eller subscribe til Event Grid events 6. Download output file og parse responses -**Regioner med Exponential Backoff Support:** +**Regioner med fail-fast-kø-støtte (klient-side exponential backoff-retry):** - australiaeast, eastus, eastus2, germanywestcentral, italynorth, northcentralus, polandcentral, swedencentral, switzerlandnorth, westus **Pris:** 50% rabatt vs. global standard (verifiser på Azure pricing page) @@ -323,10 +324,11 @@ def batch_processor(msgs: List[func.ServiceBusMessage]): **Risiko:** Azure OpenAI Batch API kan prosessere data i andre regioner enn der ressursen er hostet. **Mitigering:** -1. Bruk European regions (swedencentral, germanywestcentral, switzerlandnorth) -2. Evaluer TIA (Transfer Impact Assessment) for batch workloads -3. Vurder Microsoft Foundry med dedikert regional processing (når tilgjengelig) -4. Alternativt: Azure Machine Learning batch endpoints med regional compute +1. Bruk **Data Zone Batch** (SKU `DataZoneBatch`): inferens-prosessering begrenses til Microsofts definerte EU-datasone (data prosesseres innenfor EU-medlemsland), mens data-at-rest forblir i ressursens Azure-geografi +2. Bruk European regions (swedencentral, germanywestcentral, switzerlandnorth) +3. Evaluer TIA (Transfer Impact Assessment) for batch workloads +4. Vurder Microsoft Foundry med dedikert regional processing (når tilgjengelig) +5. Alternativt: Azure Machine Learning batch endpoints med regional compute ### Budsjettprosesser og Kostnadsforutsigbarhet @@ -356,6 +358,7 @@ def batch_processor(msgs: List[func.ServiceBusMessage]): |-----------------|----------------|----------| | **Global Standard** | 100% (baseline) | Online chat, real-time inference | | **Global Batch** | 50% | Bulk processing, content generation, document analysis | +| **Data Zone Batch** | 50% | Som Global Batch, men inferens innenfor EU/US-datasone (dataresidens) | | **Provisioned Throughput** | Varierer (reservation-based) | Høy throughput, forutsigbar latency | **Eksempel (GPT-4o, Januar 2026 priser — verifiser på Azure pricing page):** @@ -493,8 +496,8 @@ Bruk denne matrisen for raskt å avgjøre om batching er riktig: ### Microsoft Learn (Verified via MCP) 1. **Azure OpenAI Batch API:** - - [Getting started with Azure OpenAI batch deployments](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/batch) — **Verified 2026-02** - - Dekker: JSONL input format, Global-Batch deployment, 50% cost reduction, exponential backoff queuing + - [Getting started with Azure OpenAI batch deployments](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/batch) — **Verified 2026-06** + - Dekker: JSONL input format, Global-Batch + Data Zone Batch deployment, 50% cost reduction, fail-fast token-quota-håndtering med klient-side exponential backoff 2. **Microsoft Graph JSON Batching:** - [Combine multiple HTTP requests using JSON batching](https://learn.microsoft.com/en-us/graph/json-batching) — **Verified 2026-02** @@ -512,7 +515,7 @@ Bruk denne matrisen for raskt å avgjøre om batching er riktig: | Seksjon | Konfidens | Kilde | |---------|-----------|-------| -| Azure OpenAI Batch API | **Verified** | Microsoft Learn MCP (2026-02) | +| Azure OpenAI Batch API | **Verified** | Microsoft Learn MCP (2026-06) | | Microsoft Graph JSON Batching | **Verified** | Microsoft Learn MCP (2026-02) | | Azure ML Batch Endpoints | **Verified** | Microsoft Learn MCP (2026-02) | | Prismodell (50% rabatt) | **Verified** | Azure OpenAI pricing page (referenced in docs) |