diff --git a/skills/ms-ai-security/references/cost-optimization/token-counting-optimization.md b/skills/ms-ai-security/references/cost-optimization/token-counting-optimization.md index e08eca2..5dc246a 100644 --- a/skills/ms-ai-security/references/cost-optimization/token-counting-optimization.md +++ b/skills/ms-ai-security/references/cost-optimization/token-counting-optimization.md @@ -1,6 +1,6 @@ # Token Counting and Optimization Strategies -**Last updated:** 2026-02 +**Last updated:** 2026-06 **Status:** GA **Category:** Cost Optimization & FinOps for AI @@ -56,7 +56,7 @@ def num_tokens_from_messages(messages, model="gpt-4o"): except KeyError: encoding = tiktoken.get_encoding("o200k_base") - if model in {"gpt-4o", "gpt-4o-mini", "gpt-5", "gpt-4.1", "o1", "o3", "o4-mini"}: + if model in {"gpt-4o", "gpt-4o-mini", "gpt-5", "gpt-4.1", "o1", "o1-mini", "o3", "o3-mini", "o4-mini"}: tokens_per_message = 3 tokens_per_name = 1 @@ -97,6 +97,8 @@ string TrimToTokenLimit(string text, int maxTokens, Tokenizer tokenizer) ### Token Usage Estimation (Azure OpenAI On Your Data) +> ⚠️ **On Your Data er deprecated og pensjoneres 2026-10-14.** Migrer til Foundry Agent Service + Foundry IQ for grounding. Token-estimering-metodikken under er fortsatt illustrativ for RAG-token-budsjettering. + ```python import tiktoken @@ -127,14 +129,14 @@ token_output = TokenEstimator().estimate_tokens(input_text) |-----------|-------|--------| | Minimum prompt length | 1 024 tokens | Cache hit kan først oppnås | | Granularitet | 128 tokens | Etter første 1024 tokens, cache hit per 128 tokens | -| Cache TTL | 24 timer | Microsoft Foundry Classic | -| Cache TTL | 5-10 min idle, max 1 time | Azure AI Services | +| Cache TTL (in-memory) | 5-10 min idle, max 1 time | Alle GPT-4o eller nyere | +| Cache TTL (extended) | Opptil 24 timer (`prompt_cache_retention: "24h"`) | GPT-5-serien + gpt-4.1; default 24h for nyere modeller | | Kostnad (Standard) | 50% rabatt på cached tokens | Varierer per modell | | Kostnad (Provisioned) | Opptil 100% rabatt | Inkludert i PTU-pris | **Design-prinsipper:** 1. **Plasser repetitivt innhold først** — system messages, instructions, reference docs -2. **Bruk `prompt_cache_key`** for å påvirke routing og øke cache hit rate +2. **Bruk `prompt_cache_key`** for å påvirke routing og øke cache hit rate (NB: samme prefix + key over ~15 req/min → noen requests overflower til ekstra maskiner, og cache-effektiviteten faller) 3. **Unngå variasjon i første 1024 tokens** — én endring = cache miss ```python @@ -277,6 +279,8 @@ chunks = text_splitter.split_documents(pages) Cost = # training tokens × # epochs × price per token ``` +> **RFT (Reinforcement Fine-Tuning)** bruker en *tidsbasert* kostnadsmodell (ikke token-basert) med per-job-tak på $5 000 — se «Fine-Tuning Costs» under. + **Token validation pre-training:** ```python import json @@ -371,7 +375,7 @@ Er prompt >1024 tokens og repetitiv? ### Microsoft Foundry -**Token Usage Estimation (On Your Data):** +**Token Usage Estimation (On Your Data — deprecated, retires 2026-10-14):** - Intent prompt: ~1 366 tokens (gjennomsnitt) - Generation prompt: ~4 297 tokens (gjennomsnitt) - Response: ~111 tokens (gjennomsnitt) @@ -474,6 +478,13 @@ Image tokens (high-res) = (# tiles × 170) + 85 - Global Standard: $2 per 1M tokens (gpt-4.1, eksempel) - Developer (spot): 50% rabatt, kan bli paused/resumed +**Training (RFT — Reinforcement Fine-Tuning):** +- Tidsbasert, IKKE token-basert: `pris = treningstid (timer) × timepris + grader-inferencing-tokens (hvis model-grader brukes)` +- Timepris: ~$100/time core training for `o4-mini-2025-04-16` (eksempel; se pricing-side for gjeldende sats) +- **Per-job-tak: $5 000** — når taket nås pauses training og et deploybart checkpoint opprettes; du kan validere og resume (videre billing uten pristak) +- Ingen kostnad for kø, failed jobs eller jobber kansellert før training starter; ved kansellering belastes arbeid frem til siste checkpoint +- Kostnadskontroll: `reasoning_effort=Low`, begrens validering/`eval_samples`, velg minste grader-modell, juster `compute_multiplier` + **Hosting:** - $1.70/time per deployment (Standard/Global Standard) - Påløper selv om modellen ikke brukes @@ -570,18 +581,18 @@ def track_token_usage(prompt, completion, model="gpt-4o"): ## Kilder og verifisering **Microsoft Learn Documentation:** -1. [Prompt caching - Azure OpenAI](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching) +1. [Prompt caching - Azure OpenAI](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching) — **Verified MCP 2026-06** (in-memory + extended 24h) 2. [Work with chat completions models - Token management](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/chatgpt#manage-conversations) 3. [Plan and manage costs for Azure OpenAI](https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs) 4. [Token counting in AI - Dynamics 365 Business Central](https://learn.microsoft.com/en-us/dynamics365/business-central/dev-itpro/developer/ai-system-app-token-counting) 5. [Use Microsoft.ML.Tokenizers for text tokenization](https://learn.microsoft.com/en-us/dotnet/ai/how-to/use-tokenizers) -6. [Azure OpenAI On Your Data - Token usage estimation](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/concepts/use-your-data#token-usage-estimation-for-azure-openai-on-your-data) -7. [Cost management for fine-tuning](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/fine-tuning-cost-management) +6. [Azure OpenAI On Your Data - Token usage estimation](https://learn.microsoft.com/en-us/azure/foundry-classic/openai/concepts/use-your-data#token-usage-estimation-for-azure-openai-on-your-data) — ⚠️ *deprecated, retires 2026-10-14* +7. [Cost management for fine-tuning](https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/fine-tuning-cost-management) — **Verified MCP 2026-06** (SFT/DPO + RFT tidsbasert + $5000-tak) **OpenAI Resources:** 8. [OpenAI Cookbook - Token counting](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb) 9. [tiktoken GitHub repository](https://github.com/openai/tiktoken) -**Verification Date:** 2026-02-04 +**Verification Date:** 2026-06 **MCP Calls:** 4 (microsoft_docs_search × 3, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1) **Confidence Level:** High — all data sourced from official Microsoft Learn documentation and verified OpenAI tooling