ms-ai-architect/scripts/kb-eval/data/r73-payloads.json

[
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#1\",\n    \"claim\": \"Data drift-overvåking (Azure Machine Learning model monitoring) har status GA (generelt tilgjengelig) per 2026-04.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#2\",\n    \"claim\": \"Azure Machine Learning støtter følgende overvåkingssignaler som kjøres som scheduled jobs: Data drift | Prediction drift | Data quality | Feature attribution drift | Model performance (sistnevnte krever labeled ground truth-data).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#3\",\n    \"claim\": \"For numeriske features bruker Azure ML drift detection disse metrikkene: Jensen-Shannon Distance | Population Stability Index (PSI) | Normalized Wasserstein Distance | Two-Sample Kolmogorov-Smirnov Test.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#4\",\n    \"claim\": \"For kategoriske features bruker Azure ML drift detection disse metrikkene: Pearson's Chi-Squared Test | Euclidean Distance (beregnet på empiriske fordelinger av kategoriske kolonner).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#5\",\n    \"claim\": \"Lookback window size og lookback window offset i Azure ML model monitoring angis i ISO 8601-format, f.eks. P7D = 7 dager og P2D som offset som forskyver slutten av datavinduet fra kjøretidspunktet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#6\",\n    \"claim\": \"Data quality-signalets null value rate (andel null-verdier per feature) støtter presisjon på 0.00001.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#7\",\n    \"claim\": \"Data type error rate måler andel verdier som ikke matcher inferrert datatype fra reference data, og støtter PySpark-typer som IntegerType | DoubleType | StringType.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#8\",\n    \"claim\": \"azure.ai.ml.entities (v2 SDK) eksponerer klassene DataDriftSignal | DataQualitySignal | MonitorFeatureFilter | NumericalDriftMetrics | CategoricalDriftMetrics | DataDriftMetricThreshold | MonitorSchedule | ServerlessSparkCompute for oppsett av model monitoring.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#9\",\n    \"claim\": \"ServerlessSparkCompute for model monitoring konfigureres med runtime_version \\\"3.3\\\" (Spark runtime-versjon).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#10\",\n    \"claim\": \"Azure ML Dataset Monitors (preview, v1 SDK) er deprecated, og migrering til Model Monitor (v2 SDK) er anbefalt.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#11\",\n    \"claim\": \"API-overgangen fra v1 til v2 er: v1 bruker DataDriftDetector.create_from_datasets(), v2 bruker DataDriftSignal + MonitorSchedule.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#12\",\n    \"claim\": \"Model Monitor (v2) har flere kapabiliteter enn v1 Dataset Monitors: multi-signal | feature attribution drift | generative AI metrics.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#13\",\n    \"claim\": \"Instance-typene som brukes for Spark-basert model monitoring er standard_e4s_v3 (små til medium datasett, <100K rader/dag) og standard_e8s_v3 eller høyere (store datasett, >1M rader/dag).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#14\",\n    \"claim\": \"Data drift monitoring krever: Azure ML workspace (v2 API) | compute (serverless Spark eller managed compute cluster) | datastore for production inference data (Azure Blob Storage eller ADLS Gen2) | valgfritt Application Insights for custom metrics logging.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#15\",\n    \"claim\": \"Autentisering mot datastore for model monitoring skjer enten credential-based (credentials lagt på datastore) eller credential-less med User-Assigned Managed Identity (UAMI), der sistnevnte krever at systemDatastoresAuthMode settes til 'identity'.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#16\",\n    \"claim\": \"Event Grid-integrasjonen bruker eventet \\\"Run status changed\\\" med advanced filter på nøkkelen data.RunTags.azureml_modelmonitor_threshold_breached (operator String contains, verdi \\\"has failed due to one or more features violating metric thresholds\\\") for å trigge automatisert retraining.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-event-grid?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#17\",\n    \"claim\": \"Microsoft Foundry het tidligere Azure AI Studio.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#18\",\n    \"claim\": \"Microsoft Foundry har egen monitoring for generativ AI med generation quality metrics: groundedness | relevance | fluency, og støtter drift detection for grounding data i RAG-scenarier.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#19\",\n    \"claim\": \"Data drift monitoring er inkludert i Azure Machine Learning uten separat lisens; Azure ML workspace har ingen kostnad i seg selv, mens compute og storage faktureres separat (consumption-based).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#20\",\n    \"claim\": \"Out-of-box model monitoring-signaler for online endpoints er: Data quality | Data drift | Prediction drift | Feature attribution drift | Custom signals (brukerdefinerte metrics via Python-skript).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#21\",\n    \"claim\": \"Oppsettsalternativene for model monitoring er: Out-of-box (automatisk konfigurert for Azure ML online endpoints, ingen konfigurasjon påkrevd) | Advanced (custom monitoring for modeller deployet utenfor Azure ML, batch endpoints eller eksternt) | Azure Event Grid-integrasjon for ruting av alerts.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md#22\",\n    \"claim\": \"Statistiske metoder brukt av Azure ML model monitoring: Jensen-Shannon divergence for kategoriske features | Wasserstein distance (Earth Mover's Distance) for numeriske features | Population Stability Index (PSI) for feature-stabilitet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#1\",\n    \"claim\": \"Azure-tjenestene for produksjonsmonitorering og telemetri i feedback loops er: Azure Monitor + Application Insights (telemetri fra endpoints, latens, feilrater, token-forbruk) | Azure Machine Learning Model Monitoring (data drift, prediction drift, model performance degradation) | MLflow Tracing (sporing av hver inferens-interaksjon med inputs, outputs og mellomsteg).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#2\",\n    \"claim\": \"MLflow tilbyr API-et mlflow.log_feedback() for å logge brukerfeedback på en trace, med parameterne trace_id, span_id, name, value, source (AssessmentSource med source_type HUMAN) og rationale.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/generative-ai/agent-framework/non-conversational-agents#log-user-feedback\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#3\",\n    \"claim\": \"Foundry Agent Evaluation utfører evaluering med LLM judges for correctness | relevance | groundedness | safety.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#4\",\n    \"claim\": \"MLflow eksponerer mlflow.genai.evaluate() med parameterne data, predict_fn og scorers, og scorerne Correctness og RelevanceToQuery importeres fra mlflow.genai.scorers.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#5\",\n    \"claim\": \"Azure-tjenestene for automatisert retraining og model promotion er: Azure Machine Learning Pipelines (CI/CD for modelltrening og deployment) | Azure DevOps / GitHub Actions (automatiserte triggere ved model registration) | Azure Arc (hybrid/multicloud deployment-orkestrering).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#6\",\n    \"claim\": \"Azure ML model monitoring kan kjøre på ServerlessSparkCompute med instance_type standard_e4s_v3 og runtime_version 3.3.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#7\",\n    \"claim\": \"MLflow for GenAI definerer en 10-stegs kontinuerlig forbedringssyklus: Production App (traces) | User Feedback (thumbs up/down) | Monitor & Score (LLM judges) | Identify Issues (Trace UI) | Domain Expert Review (Review App) | Build Eval Dataset | Tune Scorers | Evaluate New Versions | Compare Results (MLflow evaluation runs) | Deploy or Iterate.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#8\",\n    \"claim\": \"Feedback loop-komponentene i Azure Machine Learning er: Data collection via inference tables på managed endpoints | Monitoring via Model Monitor | Alerting via Azure Monitor Alerts (e-post/webhook ved threshold breach) | Retraining via Azure ML Pipelines | A/B-testing via staging endpoints | Deployment via Managed Online Endpoints med blue-green deployment.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#9\",\n    \"claim\": \"Feedback loop-komponentene for GenAI i Microsoft Foundry er: MLflow Tracing (Databricks, span-nivå telemetri) | Review App (thumbs up/down og tekstlig feedback) | Agent Evaluation (LLM judges) | Microsoft Foundry Observability (dashboard for kvalitetstrender, latens, feil) | MLflow Datasets i Unity Catalog (versjonerte testsett) | AI Red Teaming Agent (adversarial testing for sikkerhet).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#10\",\n    \"claim\": \"Azure ML GenerationSafetyQualityMonitoringMetricThreshold støtter terskler for metrikkene groundedness | relevance | coherence | fluency (aggregerte pass rates).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#11\",\n    \"claim\": \"Feedback loop-komponentene i Power Platform AI er: Power Automate (ruting av low-confidence predictions til human review) | Dataverse / SharePoint (lagring av feedback-data) | AI Builder Feedback Loop (legger reviewede samples automatisk til treningssettet) | AI Builder (manuell/planlagt retraining).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/ai-builder/feedback-loop\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#12\",\n    \"claim\": \"Sweden Central og Norway East angis som low-carbon Azure-regioner for karbonbevisst deployment.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#13\",\n    \"claim\": \"Compute-SKU-ene som brukes til retraining i Azure Machine Learning er Standard_DS3_v2 (4 vCPU) for daglig retraining av tabular ML og GPU-SKU-en NC6s_v3 for ukentlig retraining av computer vision-modeller.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#14\",\n    \"claim\": \"Azure ML Enterprise er inkludert i Azure-subscription og faktureres som per-use compute pricing.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#15\",\n    \"claim\": \"Databricks Unity Catalog krever Premium tier.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/overview/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#16\",\n    \"claim\": \"Power Platform-lisensene gir følgende AI Builder-kreditter og feedback loop-støtte: Per User = 500 kreditter/mnd med feedback loop-støtte | Per App = ingen kreditter inkludert og ingen feedback loop-støtte (krever Per User) | AI Builder add-on = kreditter kjøpes ekstra, med feedback loop-støtte.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/ai-builder/feedback-loop\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#17\",\n    \"claim\": \"MLflow 3 introduserer en samlet evaluerings- og monitoreringslivssyklus for GenAI feedback loops, der samme scorers brukes både i utvikling (mlflow.genai.evaluate()) og i produksjonsmonitorering.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#18\",\n    \"claim\": \"MLflow 3 har de innebygde LLM-judge-scorerne RetrievalGroundedness | RelevanceToQuery | Safety | Guidelines(name, guidelines) | Correctness (med expected_facts).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#19\",\n    \"claim\": \"Azure ML Model Monitoring støtter signalene data quality (null-verdier, out-of-range, type mismatch) | data drift | prediction drift | feature attribution drift | custom signals via egne skript, og integrerer med Azure Event Grid for varsling ved terskelbrudd.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md#20\",\n    \"claim\": \"Versjonerte evalueringsdatasett lagres i Unity Catalog via API-et mlflow.genai.datasets.create_dataset().\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/evaluate-app\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/feedback-loops-continuous-improvement.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
    "claim_count": 17,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#1\",\n    \"claim\": \"Prompt Flow (i Microsoft Foundry og Azure Machine Learning) pensjoneres 20. april 2027 og anbefales ikke for ny utvikling; Microsoft anbefaler migrering til Microsoft Agent Framework (MAF).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/prompt-flow/migrate-prompt-flow-to-agent-framework?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#2\",\n    \"claim\": \"MLflow 3 Tracing registrerer inputs | outputs | mellomliggende steg | metadata, og støtter agentbaserte systemer og fler-tur-samtaler.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/tracing/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#3\",\n    \"claim\": \"Datapreparering for fine-tuning av Azure OpenAI-modeller bruker JSONL-format.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#4\",\n    \"claim\": \"Aktuelle embedding-modeller for RAG på Azure OpenAI er ada-002 | text-embedding-3-small | text-embedding-3-large.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#5\",\n    \"claim\": \"Retrieval-metodene i Azure AI Search er vector | full-text | hybrid | semantic ranker.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#6\",\n    \"claim\": \"Fine-tuning krever PTU (provisioned throughput) for produksjonsbruk.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#7\",\n    \"claim\": \"GPT-4o og GPT-4 Turbo støtter multimodal input (tekst + bilde).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#8\",\n    \"claim\": \"Microsofts GenAIOps Maturity Model har fire nivåer med poengintervaller: Nivå 1 Initial (0-9 poeng) | Nivå 2 Defined (10-14 poeng) | Nivå 3 Managed (15-19 poeng) | Nivå 4 Optimized (20-28 poeng).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/concept-llmops-maturity\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#9\",\n    \"claim\": \"Microsoft Foundry het tidligere Azure AI Studio.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#10\",\n    \"claim\": \"Model Catalog i Microsoft Foundry inneholder over 1 900 modeller fra leverandører som OpenAI | Meta | Mistral | Cohere.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#11\",\n    \"claim\": \"Evaluation SDK i Microsoft Foundry har innebygde evaluatorer for groundedness | relevance | coherence | fluency | safety.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#12\",\n    \"claim\": \"Content Safety utfører sanntidsfiltrering på kategoriene hate | violence | sexual | self-harm.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#13\",\n    \"claim\": \"Deployment i Microsoft Foundry skjer via Managed Online Endpoints med alternativene serverless | PTU | PAYG.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/python/api/overview/azure/ai-evaluation-readme\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#14\",\n    \"claim\": \"Monitorering i Microsoft Foundry gir signalene Generation Quality Signal | Token Statistics Signal.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/monitor-quality-safety\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#15\",\n    \"claim\": \"Mosaic AI på Azure Databricks har LLMOps-funksjonene Unity Catalog | MLflow for GenAI (Prompt Registry, LLM tracing, autologging) | Vector Search (Delta table-basert, auto-synkende indekser) | Model Serving | Mosaic AI Agent Framework | AI Gateway.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/generative-ai/guide/mosaic-ai-gen-ai-capabilities\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#16\",\n    \"claim\": \"Azure API Management har innebygde Azure OpenAI-policyer «Limit Azure OpenAI API token usage» | «Emit metrics for consumption», og er sone-redundant og multi-region.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-guide\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#17\",\n    \"claim\": \"Global og data zone deployments i Azure OpenAI distribuerer kapasitet på tvers av datasentre og utgjør i seg selv en gateway-implementasjon.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/azure-openai-gateway-guide\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md",
    "claim_count": 23,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#1\",\n    \"claim\": \"MLflow fanger automatisk metadata i fem kategorier: Parameters (hyperparametere) | Metrics | Artifacts (modeller, plots, datasets) | Code snapshots (Git commit hash) | Environment (Python-pakker, Docker-images).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow/tracking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#2\",\n    \"claim\": \"Azure Machine Learning Model Registry registrerer modeller med navn | versjon | tags | description.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#3\",\n    \"claim\": \"Azure Machine Learning Model Registry integreres med Azure Event Grid slik at hendelsene model registered | model deployed | data drift kan trigge workflows.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#4\",\n    \"claim\": \"I Databricks Unity Catalog fanges table-til-modell-lineage via mlflow.log_input fra MLflow 2.11.0 og senere, og automatisk for feature store-modeller.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/data-governance/unity-catalog/data-lineage\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#5\",\n    \"claim\": \"Med MLflow 3 kan Model Registry i Unity Catalog fange parametere og metrikker direkte på modellen og gjøre dem tilgjengelige på tvers av workspaces og eksperimenter; default registry-URI er databricks-uc.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#6\",\n    \"claim\": \"Modeller i Unity Catalog har ikke egen aktivitetslogg; audit av modelltilgang må hentes fra account-/workspace-audit-logs.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#7\",\n    \"claim\": \"Registrerte modeller i Unity Catalog er en undertype av FUNCTION-securable og tilganger gis med GRANT ON FUNCTION; deployment-status styres via model aliases (f.eks. «Champion»), og stages støttes ikke i Unity Catalog.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/machine-learning/manage-model-lifecycle/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#8\",\n    \"claim\": \"Innebygde Azure Policy-policies for Azure Machine Learning omfatter Model Registry Deployment Restrictions | Customer-Managed Key (CMK) Encryption | Private Link Only | Disable Local Auth | Idle Shutdown.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-integrate-azure-policy?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#9\",\n    \"claim\": \"Azure Activity Log kan rutes til Log Analytics | Event Hubs | Storage Account.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#10\",\n    \"claim\": \"Azure Activity Log har 90 dagers retensjon som standard, og retensjonen kan utvides.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#11\",\n    \"claim\": \"Unity Catalog-systemtabellen system.access.audit er i Public Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#12\",\n    \"claim\": \"Audit-loggene i Unity Catalog system tables (system.access.audit) har 365 dagers gratis retensjon.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#13\",\n    \"claim\": \"Lineage-systemtabellene system.access.table_lineage og system.access.column_lineage har et rullende 1-års-vindu og inneholder data fra 1. september 2024 og senere.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#14\",\n    \"claim\": \"De gamle Databricks-schemaene system.operational_data og system.lineage er avviklet og tomme.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#15\",\n    \"claim\": \"Azure API Management LLM-logger fanger prompts | responses | token usage | model deployment, og kan eksporteres til Azure Monitor.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-llm-logs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#16\",\n    \"claim\": \"Responsible AI Dashboard består av komponentene Model Fairness Assessment | Error Analysis | Interpretability (feature importance, SHAP) | Counterfactual Analysis | Causal Inference.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-dashboard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#17\",\n    \"claim\": \"Responsible AI Scorecard genereres som PDF og inneholder target metrics | fairness-mål | data insights, og kan deles med compliance-team, auditører og regulatorer.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-scorecard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#18\",\n    \"claim\": \"Microsoft Foundry AI Reports gir automatisk modelldokumentasjon med model cards | eval metrics | content safety-konfigurasjon, og kan eksporteres til PDF/SPDX for GRC-workflows.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-responsible-ai-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#19\",\n    \"claim\": \"Databricks Unity Catalog er inkludert i DBU-forbruket uten ekstra kostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#20\",\n    \"claim\": \"Databricks system tables for billable usage og pricing er gratis, og systemtabeller i Public Preview er gratis under preview, men kan bli fakturert senere.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#21\",\n    \"claim\": \"Lineage vist i Catalog Explorer beholdes på ubestemt tid, mens lineage-systemtabellene har et rullende 1-års-vindu.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/admin/system-tables/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#22\",\n    \"claim\": \"LLM-logging er inkludert i Azure API Management-lisensen.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/api-management-howto-llm-logs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/governance-audit-ml-operations.md#23\",\n    \"claim\": \"Delta Live Tables er omdøpt til Lakeflow Spark Declarative Pipelines i Databricks-dokumentasjonen.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/lakehouse-architecture/data-governance/best-practices\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/governance-audit-ml-operations.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md",
    "claim_count": 31,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#1\",\n    \"claim\": \"ONNX Runtime er cross-platform og kjører på Linux | Windows | macOS | cloud | edge.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#2\",\n    \"claim\": \"ONNX Runtime integrerer med følgende hardware-akselerasjonsbiblioteker: TensorRT (NVIDIA GPU) | OpenVINO (Intel) | DirectML (Windows).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#3\",\n    \"claim\": \"Microsoft-tjenester som bruker ONNX Runtime (Bing, Office, Azure AI) rapporterer gjennomsnittlig 2x ytelsesgevinst på CPU.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#4\",\n    \"claim\": \"Frameworks med ONNX-støtte er: TensorFlow | PyTorch | scikit-learn | Keras | Chainer | MXNet | MATLAB.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#5\",\n    \"claim\": \"AutoML-modeller fra Azure Machine Learning kan eksporteres til ONNX for image classification | object detection.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-inference-onnx-automl-image-models?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#6\",\n    \"claim\": \"Prompt caching i Azure OpenAI krever minimum 1 024 tokens i prompt-lengde, og de første 1 024 tokens må være identiske for at cachen skal treffe.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#7\",\n    \"claim\": \"Prompt caching støttes av alle Azure OpenAI-modeller som er GPT-4o eller nyere.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#8\",\n    \"claim\": \"Prompt caching gjelder operasjonstypene chat completion | completion | responses | real-time.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#9\",\n    \"claim\": \"Standard deployment gir rabatt på input token pricing for cachede tokens, mens Provisioned deployment (PTU) gir opptil 100 % rabatt på cachede input-tokens.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#10\",\n    \"claim\": \"Azure OpenAI prompt-cacher tømmes innen 24 timer og deles ikke mellom Azure-subscriptions.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#11\",\n    \"claim\": \"Mer enn 15 requests per minutt med samme prompt-prefiks kan gi cache-overflow og redusere effektiviteten av prompt caching.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/prompt-caching?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#12\",\n    \"claim\": \"Databricks disk cache konfigureres med Spark-innstillingene spark.databricks.io.cache.enabled | spark.databricks.io.cache.maxDiskUsage | spark.databricks.io.cache.maxMetaDataCache | spark.databricks.io.cache.compression.enabled.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/machine-learning/train-model/dl-best-practices\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#13\",\n    \"claim\": \"azureml-fe beregner utilization som (busy_replicas + queued_requests) / total_replicas.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-kubernetes-inference-routing-azureml-fe?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#14\",\n    \"claim\": \"azureml-fe skalerer opp raskt når utilization overstiger 70 %, mens nedskalering er konservativ og omtrent 20 ganger tregere enn oppskalering.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-kubernetes-inference-routing-azureml-fe?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#15\",\n    \"claim\": \"azureml-fe kan håndtere 5 000 requests per sekund med under 3 ms gjennomsnittlig latency og 15 ms p99-latency.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-kubernetes-inference-routing-azureml-fe?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#16\",\n    \"claim\": \"For mer enn 10 000 requests per sekund må antall azureml-fe-pods eller pods' vCPU-/minne-grenser økes.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-kubernetes-inference-routing-azureml-fe?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#17\",\n    \"claim\": \"Azure OpenAI Batch API har 50 % lavere kostnad enn standard (real-time) API.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#18\",\n    \"claim\": \"Azure OpenAI Batch API har et fullføringsvindu på 24 timer.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#19\",\n    \"claim\": \"Azure OpenAI Batch API støtter chat completions | embeddings | completions.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#20\",\n    \"claim\": \"Anbefalte compute-SKU-er for inference er Standard_DS3_v2 (små tabulære modeller, CPU) | Standard_NC6s_v3 (deep learning vision, GPU) | Standard_NC24s_v3 eller PTU (store språkmodeller).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#21\",\n    \"claim\": \"Azure Machine Learning tilbyr tre deployment-alternativer: Managed Online Endpoints | Batch Endpoints | Kubernetes Endpoints.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#22\",\n    \"claim\": \"Microsoft Foundry Model Catalog inneholder ferdigtrente modeller fra Hugging Face | Meta | Mistral.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#23\",\n    \"claim\": \"Ved Global Standard-deployments kan egendefinerte modellvekter midlertidig lagres utenfor ressursens geografi.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#24\",\n    \"claim\": \"Azure OpenAI har deployment-typene Standard (pay-per-token, regional dataresidens) | Provisioned Throughput (PTU, reservert kapasitet) | Global Standard (global routing) | Developer Tier (ingen timepris, ingen SLA).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#25\",\n    \"claim\": \"Azure SQL Edge bruker T-SQL-funksjonen PREDICT for native ONNX-scoring uten eksterne API-kall.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#26\",\n    \"claim\": \"Reserved Instances gir opptil 72 % rabatt for forutsigbare workloads, og Spot VMs gir opptil 90 % rabatt for ikke-kritiske batch-jobber.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#27\",\n    \"claim\": \"ONNX Runtime er lisensiert under MIT License og er gratis til kommersiell bruk.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#28\",\n    \"claim\": \"Windows ML er inkludert i Windows uten ekstra lisenskostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-onnx\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#29\",\n    \"claim\": \"AutoML computer vision-oppgaver med ONNX-støtte er: image classification (binær og multi-class) | object detection | instance segmentation.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-inference-onnx-automl-image-models?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#30\",\n    \"claim\": \"ONNX Runtime-API-et er tilgjengelig på tvers av språkene Python | C++ | C# | Java | JavaScript.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-onnx?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/inferencing-optimization-caching.md#31\",\n    \"claim\": \"SDK-en for AutoML image-oppgaver som genererer ONNX-modeller er azure-ai-ml v2 (current).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-inference-onnx-automl-image-models?view=azureml-api-2\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/inferencing-optimization-caching.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/infrastructure-as-code-mlops.md",
    "claim_count": 17,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/infrastructure-as-code-mlops.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/infrastructure-as-code-mlops.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#1\",\n    \"claim\": \"IaC-verktøy deles i to hovedtyper: deklarative verktøy (Bicep — Microsofts DSL for Azure som kompilerer til ARM templates | ARM templates (JSON) — Azure Resource Managers native format | Terraform — multi-cloud med Azure provider) og imperative verktøy (Azure CLI-scripts med `az`-kommandoer | PowerShell DSC for VM-konfigurasjon).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/devops/deliver/what-is-infrastructure-as-code\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#2\",\n    \"claim\": \"En Azure ML workspace krever disse tilknyttede ressursene (associated resources), med tilhørende ARM-ressurstyper: Azure ML Workspace (Microsoft.MachineLearningServices/workspaces) | Storage Account (Microsoft.Storage/storageAccounts) | Key Vault (Microsoft.KeyVault/vaults) | Application Insights (Microsoft.Insights/components) | Container Registry (Microsoft.ContainerRegistry/registries) | Compute-ressurser (compute clusters, instances, endpoints).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-hub-workspace-template\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#3\",\n    \"claim\": \"Bicep-ressursen Microsoft.MachineLearningServices/workspaces deployes med API-versjon 2024-01-01-preview.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-hub-workspace-template\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#4\",\n    \"claim\": \"Terraform støtter AWS | GCP | Azure, mens Bicep kun støtter Azure; Terraform krever en state file (remote backend), mens Bicep ikke har state file (håndteres av ARM).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-workspace-terraform\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#5\",\n    \"claim\": \"Bicep-ressursen Microsoft.Network/privateEndpoints deployes med API-versjon 2023-04-01.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#6\",\n    \"claim\": \"Et VNet-isolert Azure ML workspace krever disse private DNS-sonene: privatelink.api.azureml.ms | privatelink.notebooks.azure.net | privatelink.blob.core.windows.net | privatelink.vaultcore.azure.net.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#7\",\n    \"claim\": \"Terraform quickstart-templates for Azure ML i Azure/terraform-repoet består av 101 (basic workspace) | 201 (moderately secure, VNet-isolasjon) | 301 (hub-and-spoke med firewall).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-workspace-terraform\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#8\",\n    \"claim\": \"Terraform azurerm-provideren (hashicorp/azurerm) pinnes til versjon >= 3.0, < 4.0 for Azure ML-deployment.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-workspace-terraform\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#9\",\n    \"claim\": \"Resource providers som må registreres for Azure ML IaC: Microsoft.MachineLearningServices | Microsoft.Storage | Microsoft.KeyVault | Microsoft.ContainerRegistry | Microsoft.Insights | Microsoft.Network.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/how-to-manage-workspace-terraform\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#10\",\n    \"claim\": \"Azure-regionen norwayeast settes som location-parameter for å sikre at data lagres i norsk datasenter.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#11\",\n    \"claim\": \"Azure Well-Architected-anbefalingen OE:05 krever en standardisert IaC-tilnærming med deklarativ syntaks | konsistente stiler | hensiktsmessig modularisering | kvalitetssikring.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/well-architected/operational-excellence/infrastructure-as-code-design\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#12\",\n    \"claim\": \"Lagdelt IaC-pipeline for MLOps deles i tre lag: low-touch (networking, VNet, ACR) | medium-touch (compute clusters, storage, AKS) | high-touch (model endpoints, deployments).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#13\",\n    \"claim\": \"Microsoft Defender for Cloud tilbyr IaC vulnerability scanning som skanner IaC-repoer for secrets og feilkonfigurasjoner.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/well-architected/operational-excellence/infrastructure-as-code-design\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#14\",\n    \"claim\": \"Azure Key Vault tilbys i tier-ene standard og premium, der premium er HSM-backed.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#15\",\n    \"claim\": \"Azure Container Registry tilbys i SKU-ene Basic og Premium, der Premium brukes for geo-replikering i produksjon.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#16\",\n    \"claim\": \"Azure Hybrid Benefit aktiveres i Terraform via license_type = \\\"Windows_Server\\\" og krever eksisterende Windows Server-lisenser; kan gi opptil 40 % lavere VM-kostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/infrastructure-as-code-mlops.md#17\",\n    \"claim\": \"Den innebygde Azure-rollen Storage Blob Data Contributor har roleDefinitionId ba92f5b4-2d11-453d-a403-e96b0029c9fe.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/aks/concepts-machine-learning-ops\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/infrastructure-as-code-mlops.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/mlops-fundamentals-overview.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-fundamentals-overview.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-fundamentals-overview.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#1\",\n    \"claim\": \"Dokumentet angir status GA (generelt tilgjengelig) for MLOps-kapabilitetene i Azure Machine Learning (model management og deployment).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#2\",\n    \"claim\": \"MLOps-modenhet måles langs en 5-nivåskala med Level 0 | Level 1 | Level 2 | Level 3 | Level 4, hvor organisasjoner beveger seg fra manuelle prosesser til fullautomatisert CI/CD/CT (Continuous Training).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#3\",\n    \"claim\": \"MLOps-livssyklusens «inner loop» (data science) består av komponentene Data Collection | Data Preparation | Model Training | Model Evaluation.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#4\",\n    \"claim\": \"MLOps-livssyklusens «outer loop» (ML engineering) består av komponentene Model Packaging | Model Registration | Model Deployment | Model Monitoring | Model Retraining.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/aks/concepts-machine-learning-ops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#5\",\n    \"claim\": \"Azure Machine Learning tilbyr syv MLOps-kapabiliteter: Reproducible ML Pipelines | Reusable Environments | Model Registry | Lineage Tracking | Event-Driven Automation (Azure Event Grid) | Monitoring & Alerting | CI/CD Integration (Azure Pipelines, GitHub Actions).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#6\",\n    \"claim\": \"MLOps-arkitekturen følger fire hovedfaser: Data Estate | Administration & Setup | Model Development (Inner Loop) | Model Deployment (Outer Loop).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#7\",\n    \"claim\": \"Azure Machine Learning tilbyr som MLOps-plattform: Azure ML Pipelines (orkestrering) | Model Registry (versjonering + promotion) | Managed Endpoints med Online (sanntid) og Batch inference | Environments (conda/docker) | Compute Targets (managed CPU/GPU-klynger).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#8\",\n    \"claim\": \"Azure Machine Learning har integrasjonspunkter mot Azure DevOps (CI/CD via Azure Pipelines-extension) | GitHub Actions | Azure Event Grid (event-drevet automatisering) | Azure Monitor (sentralisert logging og alerting).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#9\",\n    \"claim\": \"For generative AI i Microsoft Foundry gjelder MLOps-prinsippene med fire tillegg: Prompt versioning | RAG pipelines (vector ingestion + indexing) | Safety monitoring (content filtering + responsible AI-metrikker) | Token cost tracking.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/genaiops-for-mlops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#10\",\n    \"claim\": \"Data må lagres i EU/Norge, og Azure-regionene Norway East og Norway West er tilgjengelige for dette.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#11\",\n    \"claim\": \"Serverless compute i Azure Machine Learning omtales som en ny funksjon, anbefalt for mindre workloads.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#12\",\n    \"claim\": \"Python >= 3.10 kreves for Azure ML SDK v2-skript i Azure DevOps-pipelines, og Azure Machine Learning-utvidelsen for Azure Pipelines må installeres fra Visual Studio Marketplace.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#13\",\n    \"claim\": \"Azure DevOps-integrasjon mot Azure ML har to autentiseringsalternativer: Azure Resource Manager service connection (anbefalt, brukes med AzureMLJobWaitTask@1 fra Azure ML-utvidelsen) | Generic service connection (brukes med InvokeRESTAPI-task som kaller REST API direkte).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#14\",\n    \"claim\": \"Ved bruk av generic service connection mot Azure ML REST API brukes api-version 2024-04-01.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#15\",\n    \"claim\": \"AzureMLJobWaitTask@1 kjører som server job (ingen agent pool-kostnad) og har maksimal ventetid på 2 dager, som er en hard grense i Azure DevOps.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#16\",\n    \"claim\": \"Utvidelsen «Machine Learning (classic)» er legacy; AzureMLJobWaitTask@1 fra Azure ML-utvidelsen skal brukes i stedet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#17\",\n    \"claim\": \"MLOps maturity model beskrives med trinnene Manual | Partial automation | Full CI/CD | Full MLOps with monitoring.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#18\",\n    \"claim\": \"Sentrale automatiseringsoperasjoner i Azure DevOps for MLOps er: infrastrukturutrulling (Terraform/Bicep) | komponentregistrering og versjonering | modelltrening på compute clusters | utrulling av online-/batch-endepunkter | produksjonsovervåkings-alerts.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-devops-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#19\",\n    \"claim\": \"Azure DevOps er gratis for 5 brukere og inkluderer 1800 pipeline-minutter per måned; Basic-planen anbefales for mindre team.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-fundamentals-overview.md#20\",\n    \"claim\": \"Azure ML-dokumentasjonen som legges til grunn gjelder API v2 (current).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/mlops-fundamentals-overview.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md",
    "claim_count": 31,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#1\",\n    \"claim\": \"Azure Machine Learning støtter to typer managed identities for service-to-service-autentisering: System-Assigned Managed Identity (SAI) | User-Assigned Managed Identity (UAI).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#2\",\n    \"claim\": \"Workspace-ets system-assigned managed identity (SAI) tildeles disse rettighetene: Contributor på workspace | Storage Blob Data Contributor på storage account | full tilgang til Key Vault keys/secrets/certificates | Contributor på Container Registry.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#3\",\n    \"claim\": \"Minimum RBAC-roller for en user-assigned managed identity (UAI) i Azure ML: Workspace = Contributor | Storage Account = Storage Blob Data Contributor | Key Vault (RBAC-modell) = Key Vault Administrator | Container Registry = Contributor | Application Insights = Contributor.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#4\",\n    \"claim\": \"Azure ML compute clusters støtter enten system-assigned eller user-assigned managed identity, men ikke begge samtidig.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#5\",\n    \"claim\": \"Innebygde Azure ML-roller for tilgangskontroll: AzureML Data Scientist (submit jobs, view data, manage models) | AzureML Compute Operator (manage compute resources) | Reader (view workspace metadata) | Contributor (full workspace access).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-assign-roles?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#6\",\n    \"claim\": \"Azure ML Managed VNet støtter disse compute-typene: Serverless compute (inkludert Spark) | Compute cluster | Compute instance | Managed online endpoint | Batch endpoint.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#7\",\n    \"claim\": \"Azure ML Managed VNet har to outbound-modi: Allow Internet Outbound (tillater all utgående trafikk) | Allow Only Approved Outbound (kun godkjente private endpoints/FQDN-er).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#8\",\n    \"claim\": \"Privat endepunkt for et Azure ML workspace opprettes med group-id amlworkspace, og krever privat DNS-sone privatelink.api.azureml.ms.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-configure-private-link?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#9\",\n    \"claim\": \"Påkrevde private endpoints for storage account i Azure ML: Blob (alltid) | File (alltid) | Queue (kun for Batch endpoints / ParallelRunStep) | Table (kun for Batch endpoints / ParallelRunStep).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#10\",\n    \"claim\": \"I storage account-brannmuren settes ressursinstans-typen Microsoft.MachineLearningServices/Workspace som trusted resource (trusted service exception) med workspace-navnet som instansnavn.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#11\",\n    \"claim\": \"Standard platform-managed kryptering i Azure ML: storage accounts bruker AES-256 | Cosmos DB-metadata bruker Microsoft-managed keys | compute OS-disker bruker Microsoft-managed keys.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#12\",\n    \"claim\": \"Ressursene som bruker customer-managed keys (CMK) i Azure ML er: Azure Cosmos DB (workspace-metadata) | Azure AI Search (workspace-indekser) | Azure Storage (workspace-artefakter).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-customer-managed-keys?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#13\",\n    \"claim\": \"CMK-nøkkelen for Azure ML workspace må være en RSA-nøkkel på minimum 3072 bit.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-customer-managed-keys?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#14\",\n    \"claim\": \"Begrensninger for CMK i Azure ML: nøkkelen må ligge i samme Azure-abonnement som workspace | compute OS-disker kan ikke krypteres med CMK (kun Microsoft-managed keys) | temporary disks på compute krypteres kun hvis hbi_workspace=true.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-customer-managed-keys?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#15\",\n    \"claim\": \"Når hbi_workspace=true settes på et Azure ML workspace: lokal scratch disk på compute instance krypteres | temporary disk på compute cluster krypteres | telemetrien Microsoft samler inn reduseres | ekstra kryptering i Microsoft-managed environments.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#16\",\n    \"claim\": \"All kommunikasjon i Azure ML bruker TLS 1.2 (workspace til storage account, workspace til compute, Studio til workspace API og inference-klienter til online endpoints).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#17\",\n    \"claim\": \"Godkjente utgående FQDN-er/Service Tags som må dokumenteres for Azure ML: AzureActiveDirectory | AzureFrontDoor.FrontEnd | MicrosoftContainerRegistry | AzureMonitor.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#18\",\n    \"claim\": \"Med data isolation aktivert i Azure ML får delte ressurser workspace-prefikser: storage containers {workspace-guid}-azureml-blobstore | Key Vault secrets {workspace-guid}- | Container Registry images {workspace-guid}/, og workspace-identiteten får en ABAC-betingelse som kun tillater tilgang til egne containere.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#19\",\n    \"claim\": \"Standardverdi for data isolation per workspace kind: hub = Enabled | project = Enabled (arvet fra hub) | default = Disabled.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#20\",\n    \"claim\": \"Det kuraterte Azure ML-miljøet azureml://registries/azureml/environments/sklearn-1.5 (scikit-learn 1.5) finnes i azureml-registret.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-enterprise-security\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#21\",\n    \"claim\": \"Krav for user identity pass-through i Azure ML-treningsjobber: datastore må bruke identity-based authentication (ikke cached credentials) | brukeren må ha Storage Blob Data Reader på storage account | kun støttet via CLI/SDK v2, ikke Studio | pipeline steps må konfigureres individuelt (ikke på root-nivå).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-identity-based-service-authentication?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#22\",\n    \"claim\": \"Azure Machine Learning er sertifisert mot ISO 27001 | ISO 27018 | SOC 2 Type II.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#23\",\n    \"claim\": \"Azure Machine Learning er tilgjengelig i norsk region (Oslo/Norway East).\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-enterprise-security?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#24\",\n    \"claim\": \"Customer-managed keys i Azure ML krever Azure Key Vault i tier standard eller premium; managed identities, RBAC og Private Link krever ingen egen lisens ut over Azure-abonnementet/infrastrukturkostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-customer-managed-keys?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#25\",\n    \"claim\": \"Microsoft Entra ID P2 gir Privileged Identity Management (PIM) | Conditional Access policies | Access Reviews | Identity Protection.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-enterprise-security\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#26\",\n    \"claim\": \"Azure Data Lake Storage Gen1 ble pensjonert (retired) 2024-02-29; Gen2 skal brukes i stedet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#27\",\n    \"claim\": \"Azure Database for PostgreSQL Single Server ble pensjonert (retired) 2025-03-28; Flexible Server skal brukes i stedet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#28\",\n    \"claim\": \"Azure Database for MySQL Single Server ble pensjonert (retired) 2024-09-16; Flexible Server skal brukes i stedet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-encryption?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#29\",\n    \"claim\": \"Påkrevde utgående service tags med porter for Azure ML: AzureActiveDirectory (TCP 443, autentisering) | AzureMachineLearning (TCP 443, 18881, UDP 5831) | Storage.region (TCP 443, datatilgang) | MicrosoftContainerRegistry.region (TCP 443, Docker-images).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#30\",\n    \"claim\": \"Alternativer for sikker tilkobling til Azure ML: Azure VPN Gateway (point-to-site/site-to-site) | ExpressRoute | Azure Bastion (jump box).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-security-access-control.md#31\",\n    \"claim\": \"Azure Container Registry krever Premium SKU for private endpoints, og ACR må ligge i samme VNet eller et peered VNet.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-secure-workspace-vnet?view=azureml-api-2\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/mlops-teams-collaboration-tools.md",
    "claim_count": 27,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-teams-collaboration-tools.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/mlops-teams-collaboration-tools.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#1\",\n    \"claim\": \"MLOps v2 definerer personaene Data Scientist og ML Engineer | Machine Learning Engineer (MLOps Engineer) | Data Engineer | Data Analyst | Model Tester | Business Stakeholders og Project Owners | Platform Technical Support, der Platform Technical Support ikke er ansvarlig for ML-modeller, pakker eller data.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#2\",\n    \"claim\": \"Azure Boards har nøkkelkapabilitetene work item management (user stories | bugs | tasks | features) | custom queries og status charts | sprint planning med velocity metrics | kanban boards for workflow-visualisering | portfolio management (epics -> features -> tasks).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#3\",\n    \"claim\": \"Azure DevOps består av tjenestene Azure Boards | Azure Repos | Azure Pipelines | Azure Test Plans | Azure Artifacts.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#4\",\n    \"claim\": \"Azure Pipelines integrerer med Azure Machine Learning via oppgaven AzureMLJobWaitTask@1.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-azureml\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#5\",\n    \"claim\": \"Azure Artifacts tilbyr pakkefeeds for NuGet | pip | conda til ML-biblioteker og delte komponenter.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#6\",\n    \"claim\": \"Azure DevOps MCP Server gir naturlig-språk-spørringer for prosjektstyring (f.eks. «Summarize sprint status», «List blocked work items», «Show pipeline success rates») og er en 2026-funksjon.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#7\",\n    \"claim\": \"For GitHub Actions mot Azure Machine Learning anbefales OIDC-autentisering uten langlevde hemmeligheter, med Entra-app eller user-assigned managed identity som identitet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-github-actions-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#8\",\n    \"claim\": \"GitHub Actions-mønsteret for Azure Machine Learning bruker azure/login@v2 sammen med az ml job create.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-github-actions-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#9\",\n    \"claim\": \"Databricks Asset Bundles heter nå Declarative Automation Bundles, og anbefales for samlet utrulling av kode og infrastruktur.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#10\",\n    \"claim\": \"Azure Pipelines-oppgaven for modellutrulling til Azure Machine Learning er AzureMLModelDeploy@1.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-setup-mlops-azureml\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#11\",\n    \"claim\": \"Azure Machine Learning workspace har nøkkelkapabilitetene delte notebooks og compute resources | serverless compute | managed environments og datasets | model registry | experiment tracking med MLflow | rollebasert tilgangskontroll (RBAC).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#12\",\n    \"claim\": \"Azure Artifacts har kapabilitetene private Python-pakkefeeds | conda-pakkehosting | Docker image registry (Azure Container Registry) | dependency security scanning.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#13\",\n    \"claim\": \"Microsofts MLOps maturity model har fem nivåer: Level 0 No MLOps | Level 1 DevOps but no MLOps | Level 2 Automated Training | Level 3 Automated Model Deployment | Level 4 Full MLOps Automated Operations.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/mlops-maturity-model\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#14\",\n    \"claim\": \"I Databricks MLOps Stacks har Data Engineers bundle-komponentene Lakeflow Pipelines YAML | cluster policies, med artefaktene etl_pipeline.yml | feature_store_job.yml.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#15\",\n    \"claim\": \"I Databricks MLOps Stacks har Data Scientists bundle-komponentene MLflow Projects | notebook workflows, med artefaktene train_model.yml | batch_inference_job.yml.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#16\",\n    \"claim\": \"I Databricks MLOps Stacks har MLOps Engineers bundle-komponentene environment variables | monitoring dashboards, med artefaktene databricks.yml | lakehouse_monitoring.yml.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#17\",\n    \"claim\": \"Anbefalt RBAC for Azure ML-workspaces: dev-workspace - data scientists har Contributor og data analysts har Reader; staging-workspace - model testers har Contributor og data scientists har Reader; produksjons-workspace - kun CI/CD-prosesser og platform support har Owner.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/machine-learning-operations-v2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#18\",\n    \"claim\": \"Azure Machine Learning har innebygde integrasjoner med Azure DevOps (via Azure ML extension tasks) | GitHub (via GitHub Actions for Azure ML) | MLflow (native tracking server) | Azure Monitor (automatisk metrikkinnsamling) | Azure Key Vault (secrets management).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#19\",\n    \"claim\": \"Delte prompt flows for team-samarbeid ligger i Azure AI Studio.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#20\",\n    \"claim\": \"Azure-løsningene for sikker fler-parts-datadeling er Azure Confidential Clean Rooms (secure multi-party data collaboration) | Delta Sharing (åpen protokoll for datadeling) | Azure Private Link (sikker tilkobling mellom organisasjoner).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#21\",\n    \"claim\": \"Azure DevOps gratis tier gir opptil 5 brukere med Basic access og ubegrenset antall stakeholders med read-only-tilgang.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#22\",\n    \"claim\": \"Azure DevOps gratis tier inkluderer 1800 minutter per måned med pipeline-kjøring på Microsoft-hosted agents.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#23\",\n    \"claim\": \"Azure DevOps har betalte nivåer Basic (for brukere ut over gratis-kvoten) og tillegg for ekstra parallelle jobber (per parallell jobb).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/user-guide/what-is-azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#24\",\n    \"claim\": \"GitHub Actions gir ubegrenset antall minutter for offentlige repositories.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#25\",\n    \"claim\": \"GitHub Actions gratis tier for private repositories gir 2000 minutter per måned og 500 MB lagring for artefakter.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#26\",\n    \"claim\": \"GitHub tilbyr de betalte lisensnivåene GitHub Team og GitHub Enterprise (pris per bruker per måned).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/overview-what-is-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/mlops-teams-collaboration-tools.md#27\",\n    \"claim\": \"Microsoft Learn-siden dev-tools/ci-cd/best-practices for Databricks er flyttet til dev-tools/ci-cd/flows, og siden dekker Declarative Automation Bundles, workload identity federation for auth samt SQL- og dashboard-CI/CD-arbeidsflyter.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/dev-tools/ci-cd/flows\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/mlops-teams-collaboration-tools.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/model-deployment-strategies-azure.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/model-deployment-strategies-azure.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/model-deployment-strategies-azure.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#1\",\n    \"claim\": \"Azure ML har to typer online endpoints: Managed Online Endpoints (Azure administrerer infrastrukturen, anbefalt) | Kubernetes Online Endpoints (kunden administrerer AKS-clusteret).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-endpoints-online?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#2\",\n    \"claim\": \"Azure DevOps pipeline-tasken KubernetesManifest@1 støtter canary-deployment via strategy: 'canary' med en percentage-parameter (f.eks. 10).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/devops/pipelines/ecosystems/kubernetes/canary-demo?view=azure-devops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#3\",\n    \"claim\": \"Traffic mirroring (shadow deployment) er ikke støttet for Kubernetes online endpoints i Azure ML.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2#test-the-deployment-with-mirrored-traffic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#4\",\n    \"claim\": \"Azure ML online endpoints tillater maksimalt 50 % mirror traffic, begrenset av endpointets bandwidth quota.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2#test-the-deployment-with-mirrored-traffic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#5\",\n    \"claim\": \"Kun én deployment kan motta mirrored traffic på et Azure ML online endpoint, og samme deployment kan ikke motta både live traffic og mirrored traffic.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-safely-rollout-online-endpoints?view=azureml-api-2#test-the-deployment-with-mirrored-traffic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#6\",\n    \"claim\": \"Trafikkallokeringen mellom deployments på et Azure ML online endpoint må summere til 100 % (eller 0 % for å deaktivere).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2#model-registration,-packaging,-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#7\",\n    \"claim\": \"En klient kan overstyre traffic routing på et Azure ML online endpoint ved å sette HTTP-headeren azureml-model-deployment: <deployment-navn>.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment?view=azureml-api-2#model-registration,-packaging,-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#8\",\n    \"claim\": \"Azure Monitor eksponerer metrikkene RequestLatency | RequestsPerSecond | CpuUtilizationPercentage for Azure ML online endpoint deployments.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#9\",\n    \"claim\": \"Modeller i et Azure ML-registry refereres ved deployment med asset-URIen azureml://registries/<registry>/models/<navn>/versions/<versjon>.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-machine-learning-registries-mlops\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#10\",\n    \"claim\": \"MLflow no-code deployment i Azure ML har innebygd støtte for scikit-learn | TensorFlow | PyTorch.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#11\",\n    \"claim\": \"Azure ML online endpoints støtter både nøkkelbasert autentisering (auth_mode=\\\"key\\\") og Microsoft Entra token-basert autentisering (aad_token); sistnevnte anbefales for produksjonsdeployments.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-endpoints?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#12\",\n    \"claim\": \"VM-typer for Azure ML managed online endpoints inkluderer Standard_DS2_v2 (2 vCPU / 7 GB RAM) | Standard_DS3_v2 (4 vCPU / 14 GB) | Standard_DS4_v2 (8 vCPU / 28 GB) | Standard_NC6s_v3 GPU (6 vCPU / 112 GB).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#13\",\n    \"claim\": \"Default bandwidth quota for et Azure ML online endpoint er 5 MBps per endpoint; overskridelse gir throttling med HTTP 429-feil.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#14\",\n    \"claim\": \"Azure Machine Learning workspace er gratis (man betaler kun for underliggende compute/storage), og alle deployment-funksjoner som blue-green, mirroring og A/B er inkludert.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#15\",\n    \"claim\": \"Azure DevOps gratis tier inkluderer 1 hosted pipeline (Microsoft-hosted agent); Basic-planen faktureres per bruker, og deployment-pipelines krever parallel jobs som koster ekstra ved mange pipelines.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#16\",\n    \"claim\": \"Azure ML online endpoints eksponeres på regionsspesifikke scoring-URLer av formen <endpoint>.<region>.inference.ml.azure.com, f.eks. prod-endpoint.norwayeast.inference.ml.azure.com i Norway East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-management-and-deployment\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#17\",\n    \"claim\": \"Progressive rollout av MLflow-modeller til Azure ML online endpoints støtter model packaging via --with-package for endpoints uten egress-tilkobling.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-deploy-mlflow-models-online-progressive?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#18\",\n    \"claim\": \"Azure Well-Architected SDP-prinsippet (OE:11) om progressive exposure omfatter modellene Canary | Blue-Green | Deployment Stamps.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/safe-deployments\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#19\",\n    \"claim\": \"Azure-tjenester som understøtter safe deployment practices: Azure Pipelines og GitHub Actions (multi-stage deployments med approval gates) | Azure App Configuration (feature flag management) | Azure Load Balancers (traffic routing og health monitoring).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/safe-deployments\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-deployment-strategies-azure.md#20\",\n    \"claim\": \"Point-in-time restore er tilgjengelig for Azure SQL | Cosmos DB | MySQL | PostgreSQL.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/operational-excellence/safe-deployments\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/model-deployment-strategies-azure.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/model-drift-performance-degradation.md",
    "claim_count": 21,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/model-drift-performance-degradation.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/model-drift-performance-degradation.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#1\",\n    \"claim\": \"Azure Machine Learning sitt modell-monitoring-rammeverk oppdager drift og degradering gjennom fem signaltyper: Data drift | Prediction drift | Data quality | Feature attribution drift | Model performance.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#2\",\n    \"claim\": \"Azure Machine Learning Model Monitoring har GA-status per februar 2026, med støtte for tabular classification- og regression-tasks.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#3\",\n    \"claim\": \"Signalene Data Drift, Prediction Drift og Data Quality har GA-status i Azure Machine Learning model monitoring.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#4\",\n    \"claim\": \"Data Drift-signalet bruker metrikkene Jensen-Shannon Distance | Population Stability Index | Normalized Wasserstein Distance | Two-Sample Kolmogorov-Smirnov Test | Pearson's Chi-Squared Test.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#5\",\n    \"claim\": \"Prediction Drift-signalet bruker metrikkene Jensen-Shannon Distance | Population Stability Index | Normalized Wasserstein Distance | Chebyshev Distance | Two-Sample Kolmogorov-Smirnov Test | Pearson's Chi-Squared Test.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#6\",\n    \"claim\": \"Data Quality-signalet bruker metrikkene Null value rate | Data type error rate | Out-of-bounds rate.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#7\",\n    \"claim\": \"Feature Attribution Drift-signalet har Preview-status og bruker metrikken Normalized Discounted Cumulative Gain.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#8\",\n    \"claim\": \"Model Performance-signalet har Preview-status og krever ground truth-data.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#9\",\n    \"claim\": \"Model Performance-signalet bruker metrikkene Accuracy | Precision | Recall for classification og MAE | MSE | RMSE for regression.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#10\",\n    \"claim\": \"Modell-monitoring støtter fire typer referansedata (baseline): Training data (anbefalt for data drift og data quality) | Validation data (anbefalt for prediction drift) | Recent past production data (rolling baseline) | Ground truth data (påkrevd for model performance).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#11\",\n    \"claim\": \"Produksjons-inferensdata kan samles inn på to måter: Azure ML Data Collector (automatisk innsamling fra online endpoints med correlationid for join) | manuell innsamling via selvregistrerte data assets (krever custom preprocessing component).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#12\",\n    \"claim\": \"Serverless Spark compute for monitoring-jobber støtter VM-typene Standard_E4s_v3 | Standard_E8s_v3 | Standard_E16s_v3 | Standard_E32s_v3 | Standard_E64s_v3.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#13\",\n    \"claim\": \"Serverless Spark for monitoring-jobber kjører med Spark runtime version 3.3 eller 3.4.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#14\",\n    \"claim\": \"Oppsett av modell-monitoring gjøres med azure-ai-ml Python SDK v2.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#15\",\n    \"claim\": \"En custom signal-komponent må ha input-signaturen production_data (mltable) og <metric>_threshold (literal), og output-signaturen signal_metrics (mltable) med kolonnene group | metric_name | metric_value | threshold_value.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#16\",\n    \"claim\": \"Event Grid-integrasjonen for Azure ML model monitoring bruker event-typen Run status changed; event-typen Dataset drift detected er deprecated (v1).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-event-grid?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#17\",\n    \"claim\": \"Støttede event handlers for Azure ML Event Grid-events er Azure Functions | Logic Apps | Event Hubs.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-event-grid?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#18\",\n    \"claim\": \"Azure ML Data Collector medfører ingen ekstra kostnad – den er inkludert i endpoint-kostnaden.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#19\",\n    \"claim\": \"Azure ML Workspace har ingen lisenskostnad (pay-per-use for compute/storage), Event Grid har ingen lisenskostnad (pay-per-event), og Azure Monitor er inkludert i Azure-subscriptions.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#20\",\n    \"claim\": \"Data drift (preview) i Azure ML v1 skal pensjoneres og erstattes av Model Monitor.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-drift-performance-degradation.md#21\",\n    \"claim\": \"MLflow 3 production monitoring gjenbruker utviklings-scorerne Groundedness og Relevance på produksjonstraces.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/model-drift-performance-degradation.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/model-evaluation-frameworks.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/model-evaluation-frameworks.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/model-evaluation-frameworks.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#1\",\n    \"claim\": \"AI-assisterte kvalitetsmetrikker i Microsoft Foundry omfatter Groundedness | Relevance | Coherence | Fluency | GPT similarity, krever en judge-modell (GPT-3.5+/GPT-4), og kun GPT similarity krever ground truth.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#2\",\n    \"claim\": \"NLP-baserte kvalitetsmetrikker i Microsoft Foundry omfatter F1 | ROUGE | BLEU | GLEU | METEOR, krever ground truth og krever ingen judge-modell.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#3\",\n    \"claim\": \"Risk & Safety-metrikker i Microsoft Foundry omfatter Self-harm | Hateful content | Violence | Sexual content | Protected material | Indirect attack, krever ikke ground truth, og kjøres av en Foundry-hostet GPT-4.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#4\",\n    \"claim\": \"Microsoft Foundry støtter tre evalueringsmål: Model | Agent | Dataset.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#5\",\n    \"claim\": \"Data mapping-krav for Foundry-evaluering: Groundedness og Relevance krever query + response + context; Coherence og Fluency krever query + response; GPT similarity krever query + response + ground truth; F1/BLEU/ROUGE/METEOR krever response + ground truth; safety-metrikker krever query + response.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#6\",\n    \"claim\": \"Azure AI Evaluation SDK konfigurerer judge-modellen mot Azure OpenAI med api_version 2024-06-01 og azure_deployment gpt-4o.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme?view=azure-python\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#7\",\n    \"claim\": \"evaluate()-funksjonen i Azure AI Evaluation SDK tar evalueringsdata i CSV- eller JSONL-format.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme?view=azure-python\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#8\",\n    \"claim\": \"MLflow 3 har fem scorer-typer: Built-in judges | Guidelines judges | Custom LLM judges | Code-based scorers | Multi-turn judges.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/concepts/scorers\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#9\",\n    \"claim\": \"MLflow 3 built-in judges omfatter Correctness | RetrievalGroundedness | Safety | RelevanceToQuery | Fluency | Equivalence.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/concepts/scorers\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#10\",\n    \"claim\": \"MLflow 3 guidelines judges omfatter Guidelines | ExpectationsGuidelines, som uttrykker egendefinerte pass/fail-regler i naturlig språk.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/concepts/scorers\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#11\",\n    \"claim\": \"MLflow 3 multi-turn judges omfatter ConversationCompleteness | UserFrustration | KnowledgeRetention | ConversationalSafety.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/concepts/scorers\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#12\",\n    \"claim\": \"Azure ML-monitorering for tradisjonell ML dekker datakvalitetssignaler (null rate | out-of-bounds | type errors) og statistisk drift (Jensen-Shannon divergence | Wasserstein distance), samt egendefinerte metrikker via Python-skript i monitoring-jobber.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#13\",\n    \"claim\": \"MLflow 3 består av hovedkomponentene Tracing | LLM Judges | Scorers | Review App | Production Monitoring.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#14\",\n    \"claim\": \"Judge-modell i MLflow 3 kan settes eksplisitt til den Databricks-hostede modellen databricks:/databricks-gpt-5-mini.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/concepts/scorers\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#15\",\n    \"claim\": \"Azure AI Evaluation tilbyr spesialiserte agent-evaluatorer: IntentResolutionEvaluator | TaskAdherenceEvaluator | ToolCallAccuracyEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme?view=azure-python\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#16\",\n    \"claim\": \"Microsoft Foundry-portalen tilbyr Evaluation page | Model Catalog med Benchmarks | Evaluator Library (med versjonering) | Synthetic data generation.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#17\",\n    \"claim\": \"Prompt Flow (Microsoft Foundry + Azure ML) pensjoneres 20. april 2027 og anbefales ikke for ny utvikling; Microsoft anbefaler migrering til Microsoft Agent Framework.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/how-to-develop-an-evaluation-flow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#18\",\n    \"claim\": \"AI-assisterte safety-metrikker er kun hostet i regionene East US 2 | France Central | UK South | Sweden Central.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#19\",\n    \"claim\": \"Azure confidential computing er ikke GA for LLM judges.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#20\",\n    \"claim\": \"Microsoft Foundry har ingen lisenskostnad for selve plattformen — pay-as-you-go der man kun betaler for compute/LLM-tokens.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#21\",\n    \"claim\": \"MLflow 3 er inkludert i Databricks-abonnement på Premium- eller Enterprise-tier.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/mlflow3/genai/eval-monitor/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-evaluation-frameworks.md#22\",\n    \"claim\": \"Azure AI Evaluation SDK er open source under MIT-lisens og gratis å bruke.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/python/api/overview/azure/ai-evaluation-readme?view=azure-python\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/model-evaluation-frameworks.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/model-versioning-registry-management.md",
    "claim_count": 23,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/model-versioning-registry-management.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/model-versioning-registry-management.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#1\",\n    \"claim\": \"Model versioning og registry management i Azure Machine Learning er merket som GA (generelt tilgjengelig).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#2\",\n    \"claim\": \"Azure Machine Learning tilbyr to primære registry-tilnærminger: workspace model registry (team-intern bruk) | Azure Machine Learning registry (tverrorganisatorisk deling), og begge støtter MLflow som standardformat.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#3\",\n    \"claim\": \"MLflow model stages i Azure ML består av: None | Staging | Production | Archived.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models-mlflow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#4\",\n    \"claim\": \"Azure ML Registry støtter deling av tre assettyper på tvers av workspaces og subscriptions: modeller | komponenter | miljøer (environments).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#5\",\n    \"claim\": \"Registry-operasjoner mot Azure ML Registry utføres med Azure ML CLI v2 og Python SDK v2.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#6\",\n    \"claim\": \"CLI v2-operasjoner mot registry omfatter: `az ml model create` (opprett modell i registry fra lokale filer) | `az ml model share` (del modell fra workspace til registry) | `az ml online-deployment create` (deploy modell fra registry til workspace).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#7\",\n    \"claim\": \"Tilgangskontroll mot Azure ML Registry er ACR token-basert, og workspace-compute får `AcrPull`-rollen via registryets managed identity.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#8\",\n    \"claim\": \"MLflow-format er påkrevd for no-code deployment med innebygd scoring server.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models-mlflow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#9\",\n    \"claim\": \"Regiondekning: Workspace Registry er bundet til workspace-regionen, mens Azure ML Registry har multi-region support.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#10\",\n    \"claim\": \"Azure ML auto-genererer inkrementelle modellversjoner (v1, v2, v3 ...) ved registrering.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models-mlflow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#11\",\n    \"claim\": \"MLflow stages er kun tilgjengelig via MLflow SDK og ikke i Azure ML Studio UI, og deployment fra en stage støttes ikke direkte i Azure ML online endpoints.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models-mlflow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#12\",\n    \"claim\": \"I Databricks Unity Catalog er MLflow stages utfaset (legacy) og erstattet av aliases (muterbare navngitte referanser, f.eks. «Champion») | tags | deployment jobs.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#13\",\n    \"claim\": \"MLflow 3 bruker `databricks-uc` som default registry.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#14\",\n    \"claim\": \"I MLflow 3 er `mlflow.evaluate` utfaset — bruk `mlflow.models.evaluate` for klassisk ML og `mlflow.genai.evaluate` for GenAI.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#15\",\n    \"claim\": \"I MLflow 3 bruker `log_model` parameteren `name=` (ikke `artifact_path`), og modell-URI-en er `models:/<model_id>`.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#16\",\n    \"claim\": \"MLflow Registry URI i Azure ML har to former: `azureml://<workspace>` for workspace-registry | `azureml://registries/<registry-name>` for Azure ML Registry.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-manage-models-mlflow?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#17\",\n    \"claim\": \"Microsoft Foundry Model Catalog bruker samme underliggende registry-infrastruktur og tilbyr to deployment-alternativer: serverless API deployments (Foundry Models) | managed compute deployments (Azure ML Registry til VM-er).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/concepts/foundry-models-overview?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#18\",\n    \"claim\": \"For production online endpoint deployments anbefaler Microsoft Entra token-basert autentisering (`aad_token`) fremfor key-based authentication.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#19\",\n    \"claim\": \"Azure ML Registry kan konfigureres i regionene Norway East og Norway West for data residency.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-share-models-pipelines-across-workspaces-with-registries?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#20\",\n    \"claim\": \"Azure Machine Learning er inkludert i Azure-abonnementet uten ekstra lisenskostnad; kun compute og storage koster.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#21\",\n    \"claim\": \"MLflow er open source under Apache 2.0-lisens og gratis.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#22\",\n    \"claim\": \"Azure DevOps er gratis for opptil 5 brukere.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/model-versioning-registry-management.md#23\",\n    \"claim\": \"GitHub Actions inkluderer 2000 gratis minutter per måned.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-model-monitoring\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/model-versioning-registry-management.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/monitoring-observability-ml-systems.md",
    "claim_count": 25,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/monitoring-observability-ml-systems.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/monitoring-observability-ml-systems.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#1\",\n    \"claim\": \"Azure Machine Learning Model Monitoring har fem innebygde monitoring-signaler: Data Drift | Prediction Drift | Data Quality | Model Performance | Feature Attribution Drift.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#2\",\n    \"claim\": \"Model Performance-signalet beregner accuracy | precision | recall for klassifisering og MAE | MSE | RMSE for regresjon, ved å sammenligne prediksjoner med ground truth.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#3\",\n    \"claim\": \"Terskelmetrikkene i Azure ML model monitoring er Jensen-Shannon distance (numeriske features) | Pearson's chi-squared test (kategoriske features) | Normalized Discounted Cumulative Gain (feature attribution drift).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#4\",\n    \"claim\": \"Datainnsamling avhenger av deployment-type: online endpoints (Azure ML Data Collector samler inputs og outputs automatisk til Azure Blob Storage) | batch endpoints og eksterne deployments (må implementeres selv og registreres som Azure ML data asset) | ground truth-data (må samles manuelt på applikasjonsnivå).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#5\",\n    \"claim\": \"Model monitoring planlegges via cron-uttrykk eller recurrence patterns med frekvensene minutt | time | dag | uke | måned.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#6\",\n    \"claim\": \"Azure ML model monitoring kjører på serverless Spark compute med instanstyper i området Standard_E4s_v3 til Standard_E64s_v3.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#7\",\n    \"claim\": \"Azure Monitor samler automatisk plattformmetrikker på workspace-nivå for Azure ML: run counts | model deployments | quota utilization.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#8\",\n    \"claim\": \"På online endpoint-nivå samler Azure Monitor metrikkene request latency (P50/P90/P95/P99) | requests per minute | network bytes.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#9\",\n    \"claim\": \"På deployment-nivå samler Azure Monitor metrikkene CPU/GPU utilization | memory utilization | disk utilization | data collection events/errors.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#10\",\n    \"claim\": \"Distribuerte treningslogger (stdout/stderr fra alle workers) samles i Application Insights-tabellen AppTraces med 90 dagers retention.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-log-search?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#11\",\n    \"claim\": \"Azure Monitor har innebygde AIOps-kapabiliteter som ikke krever ML-kunnskap: Dynamic thresholds | Predictive autoscale (for VM scale sets) | Log Analytics Workspace Insights | Observability agent.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/aiops/aiops-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#12\",\n    \"claim\": \"Event Grid-abonnement for model monitor-varsler skal filtrere på event type «Run status changed» (ikke «Dataset drift detected», som tilhører v1), med advanced filter key data.RunTags.azureml_modelmonitor_threshold_breached og operator String contains.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#13\",\n    \"claim\": \"Azure ML SDK v2 er anbefalt for model monitoring og eksponerer entitetene MonitorSchedule | MonitorDefinition | ServerlessSparkCompute | DataDriftSignal | AlertNotification i azure.ai.ml.entities.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#14\",\n    \"claim\": \"CLI v2 definerer monitoring-schedule i YAML med $schema http://azureml/sdk-2-0/Schedule.json.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#15\",\n    \"claim\": \"Alle Azure ML-metrikker ligger i metrikk-namespacet «Machine Learning Service Workspace».\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#16\",\n    \"claim\": \"Azure ML resource-logger i Log Analytics inkluderer tabellene AmlComputeJobEvent (med feltene TimeGenerated | ClusterId | EventType | ExecutionState | ToolType) og AmlOnlineEndpointTrafficLog (med feltene TimeGenerated | EndpointName | DeploymentName | ResponseCode).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#17\",\n    \"claim\": \"Anbefalte alert rules for Azure ML er Model Deploy Failed (Total > 0) | Quota Utilization (Average > 90 %) | Unusable Nodes (Total > 0).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#18\",\n    \"claim\": \"Application Insights-integrasjonen for Azure ML gir Live metrics | Transaction search | Failures | Performance analysis.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#19\",\n    \"claim\": \"Workspace-based Application Insights er standard for nye Azure ML-workspaces, og Azure Monitor Private Link brukes for VNet-isolasjon.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#20\",\n    \"claim\": \"Generation quality monitoring for LLM-apper bruker GenerationSafetyQualitySignal med metrikk-terskler for groundedness (aggregated_groundedness_pass_rate) og relevance (aggregated_relevance_pass_rate).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#21\",\n    \"claim\": \"GenerationTokenStatisticsSignal sporer token-forbruk automatisk og krever ingen terskler (informasjonell metrikk).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#22\",\n    \"claim\": \"Log Analytics støtter retention-policyer på tabellnivå med logplanene Basic og Analytics.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#23\",\n    \"claim\": \"Azure Activity Log oppbevarer hendelser i 90 dager; lengre oppbevaring krever eksport til Storage.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#24\",\n    \"claim\": \"RBAC-roller for tilgangskontroll på Log Analytics workspace: Reader | Contributor | Log Analytics Reader | Monitoring Metrics Publisher.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/monitoring-observability-ml-systems.md#25\",\n    \"claim\": \"Azure ML Model Monitoring v3 er på roadmap for Q2 2026.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/monitoring-observability-ml-systems.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/mlops-genaiops/responsible-ai-mlops-integration.md",
    "claim_count": 15,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/mlops-genaiops/responsible-ai-mlops-integration.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/mlops-genaiops/responsible-ai-mlops-integration.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#1\",\n    \"claim\": \"Responsible AI dashboard i Azure Machine Learning består av komponentene Error Analysis | Model Fairness | Model Interpretability | Data Analysis | Counterfactual What-If | Causal Inference.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-dashboard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#2\",\n    \"claim\": \"Responsible AI Scorecard i Azure Machine Learning er i public preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-scorecard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#3\",\n    \"claim\": \"Responsible AI Scorecard bruker Azure SDK v2 og CLI v2.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-responsible-ai-insights-sdk-cli?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#4\",\n    \"claim\": \"Responsible AI Scorecard genereres som et PDF-dokument som oppsummerer RAI-innsiktene fra dashbordet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-scorecard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#5\",\n    \"claim\": \"Responsible AI Scorecard inneholder Model summary med performance metrics og target values | Data characteristics (distribusjon, representasjon) | Fairness assessment på tvers av sensitive grupper | Top important features (global interpretability) | Error cohort analysis | Causal insights.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-scorecard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#6\",\n    \"claim\": \"Azure Machine Learning tilbyr RAI-komponenter for pipeline jobs: microsoft_azureml_rai_tabular_insight_constructor | microsoft_azureml_rai_tabular_explanation | microsoft_azureml_rai_tabular_erroranalysis | microsoft_azureml_rai_tabular_causal | microsoft_azureml_rai_tabular_counterfactual | microsoft_azureml_rai_tabular_insight_gather.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-responsible-ai-insights-sdk-cli?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#7\",\n    \"claim\": \"Azure ML Model Monitoring sporer data drift | prediction drift | feature attribution drift.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#8\",\n    \"claim\": \"Azure ML DataCollector i ManagedOnlineDeployment støtter innsamling av collections model_inputs | model_outputs, med konfigurerbar sampling_rate.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-data-collection\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#9\",\n    \"claim\": \"Responsible AI dashboard støtter kun tabulære, strukturerte data (regresjon og klassifisering).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-dashboard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#10\",\n    \"claim\": \"Prompt Flow avvikles 2027-04-20, og evaluerings-/orkestreringslogikk må migreres til Microsoft Agent Framework (MAF).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-responsible-ai-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#11\",\n    \"claim\": \"Model Registry er inkludert i Azure Machine Learning-workspace uten ekstrakostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-responsible-ai-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#12\",\n    \"claim\": \"Test-datasettet for RAI dashboard må begrenses til 5000 rader, som er maksgrensen for RAI dashboard-UI-et.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-dashboard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#13\",\n    \"claim\": \"Responsible AI Dashboard er open source og bygger på InterpretML | Fairlearn | ErrorAnalysis | DiCE, og er inkludert i Azure ML workspace.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-responsible-ai-dashboard?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#14\",\n    \"claim\": \"Azure DevOps lisensieres per bruker, og Basic Plan er gratis for inntil 5 brukere.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-responsible-ai-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/mlops-genaiops/responsible-ai-mlops-integration.md#15\",\n    \"claim\": \"GitHub Actions er gratis for offentlige repositorier og betalt for private repositorier.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/machine-learning/concept-responsible-ai-dashboard\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/mlops-genaiops/responsible-ai-mlops-integration.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/agentic-rag-patterns.md",
    "claim_count": 21,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/agentic-rag-patterns.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/agentic-rag-patterns.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#1\",\n    \"claim\": \"Semantic Kernel-baserte agentic RAG-funksjoner har status GA (generelt tilgjengelig).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#2\",\n    \"claim\": \"Azure AI Search agentic retrieval er delvis GA via REST API-versjon `2026-04-01`; GA-overflaten returnerer ekstraktiv grounding-data (`activity` + `references`), ikke syntetiserte svar.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#3\",\n    \"claim\": \"GA-kildetypene (knowledge sources) for Azure AI Search agentic retrieval er: searchIndex | azureBlob | indexedOneLake | web.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#4\",\n    \"claim\": \"Answer synthesis, ikke-minimal reasoning effort (LLM query planning på low/medium) og multi-turn messages forblir preview og krever REST API-versjon `2026-05-01-preview`.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#5\",\n    \"claim\": \"Azure-portalen og Foundry-portalen gir kun preview-tilgang til agentic retrieval.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#6\",\n    \"claim\": \"Microsoft tilbyr tre primære implementeringsveier for agentic RAG: Semantic Kernel (code-first RAG med TextSearchProvider) | Microsoft Agent Framework (produksjonsklart, merged fra AutoGen + Semantic Kernel) | Azure AI Search agentic retrieval (managed service med automatisk query decomposition).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#7\",\n    \"claim\": \"Semantic Kernel TextSearchProvider har to modi for retrieval timing: BeforeAIInvoke (default — automatisk søk før hver agent-invokasjon) | OnDemandFunctionCalling (agenten bestemmer selv når den søker).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#8\",\n    \"claim\": \"Azure AI Search agentic retrieval kjører en 4-stegs prosess: Workflow initiation | Query planning | Query execution | Result synthesis.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#9\",\n    \"claim\": \"Reasoning effort i Azure AI Search agentic retrieval har nivåene minimal | low | medium; ved low/medium dekomponerer LLM-en komplekse queries i fokuserte subqueries, ved minimal hoppes query planning over og queries sendes direkte til knowledge sources, og default er low.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#10\",\n    \"claim\": \"I result synthesis returneres merged content alltid, mens source references og activity log er valgfrie.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#11\",\n    \"claim\": \"Semantic Kernel TextSearchProvider/TextSearchStore støtter vector stores: Azure AI Search | Qdrant | Pinecone | Redis.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#12\",\n    \"claim\": \"Semantic Kernel RAG med TextSearchProvider er eksperimentell (subject to change).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#13\",\n    \"claim\": \"Azure AI Search managed agentic retrieval returnerer en 3-delt respons: grounding data | citations (source references) | activity plan.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#14\",\n    \"claim\": \"Azure AI Search agentic retrieval med knowledge bases (som kan samle flere knowledge sources, indexed eller remote) er kun tilgjengelig i utvalgte regioner, og grensene varierer med pristier og reasoning effort.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#15\",\n    \"claim\": \"Azure AI Search agentic retrieval har to planer: Free plan (default — månedlig gratis token-kvote) og Standard plan (pay-as-you-go etter at gratis-kvoten er brukt opp); Azure OpenAI faktureres separat som pay-as-you-go for query planning og answer synthesis.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#16\",\n    \"claim\": \"Semantic Kernel har fire agent orchestration patterns: Sequential | Concurrent | Handoff | Group Chat.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-orchestration/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#17\",\n    \"claim\": \"Prompt Flow i Microsoft Foundry pensjoneres 2027-04-20 og erstattes av Microsoft Agent Framework.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#18\",\n    \"claim\": \"Azure OpenAI (function calling) er tilgjengelig i Sweden Central, med data i EU/EØS.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#19\",\n    \"claim\": \"LLM-en som bryter ned komplekse queries til fokuserte subqueries i Azure AI Search agentic retrieval er gpt-4o / 4.1 / 5-serien.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#20\",\n    \"claim\": \"Arkitekturen for Azure AI Search agentic retrieval består av: Knowledge Base | Knowledge Source(s) | Azure OpenAI LLM | Azure AI Search index med semantic configuration.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/agentic-retrieval-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/agentic-rag-patterns.md#21\",\n    \"claim\": \"AI Agent Design Patterns (Azure Architecture Center) omfatter mønstrene: sequential (pipeline) | parallel fanout | supervisor | autonomous loop.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/ai-agent-design-patterns\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/agentic-rag-patterns.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/chunking-strategies.md",
    "claim_count": 27,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/chunking-strategies.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/chunking-strategies.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#1\",\n    \"claim\": \"Chunking-funksjonaliteten som beskrives er GA for kjernefunksjonene, mens token chunking er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#2\",\n    \"claim\": \"Azure AI Search tilbyr fem innebygde chunking-strategier gjennom integrated vectorization: fixed-size (Text Split skill) | variable-size (sentences mode) | document layout-basert (Document Intelligence) | semantisk (Azure Content Understanding) | document parsing (native formatstøtte).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#3\",\n    \"claim\": \"Alle seks chunking-metodene i Azure AI Search har status GA: Text Split skill (`pages` mode) | Text Split skill (`sentences` mode) | Document Intelligence Layout skill | Azure Content Understanding skill | Indexer parsing modes | Custom Web API skill.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#4\",\n    \"claim\": \"Document parsing gjøres via indexer parsing modes som støtter Markdown | JSON.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#5\",\n    \"claim\": \"Text Split skill-parameteren `textSplitMode` har default `pages` og støtter verdiene `pages` (multi-sentence chunks) | `sentences` (enkeltsetninger).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#6\",\n    \"claim\": \"Text Split skill-parameteren `maximumPageLength` har default 2000 tegn, som tilsvarer omtrent 512 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#7\",\n    \"claim\": \"Text Split skill-parameteren `pageOverlapLength` har default 0, altså ingen overlapp mellom chunks.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#8\",\n    \"claim\": \"Text Split skill-parameteren `defaultLanguageCode` har default `en` og settes til `no` for norsk setningsdeteksjon.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#9\",\n    \"claim\": \"Text Split skill-parameteren `maximumPagesToTake` har default 0, der 0 betyr at alle chunks tas med.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#10\",\n    \"claim\": \"Text Split skill identifiseres i skillset-definisjonen med @odata.type `#Microsoft.Skills.Text.SplitSkill`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#11\",\n    \"claim\": \"Document Layout skill identifiseres med @odata.type `#Microsoft.Skills.Util.DocumentIntelligenceLayoutSkill` og konfigureres blant annet med `outputMode` (oneToMany) og `markdownHeaderDepth` (h3).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#12\",\n    \"claim\": \"Document Layout skill har begrenset portalstøtte og er tilgjengelig i regionene East US | West Europe | North Central US.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#13\",\n    \"claim\": \"Document Layout skill krever filbasert datakilde med formater som PDF | DOCX | PPTX.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#14\",\n    \"claim\": \"Index projections gir sporbarhet fra chunk til kildedokument via feltet `text_parent_id`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#15\",\n    \"claim\": \"Azure Content Understanding er GA og er anbefalt skill for nye skillsets.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#16\",\n    \"claim\": \"Azure Content Understanding skill produserer Markdown-output med LaTeX | HTML-tabeller | headings.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#17\",\n    \"claim\": \"Azure-begrensning: overlapp over 50 % av chunk-størrelsen gir faktisk ingen overlapp; anbefalt overlapp er høyst 25-30 %.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#18\",\n    \"claim\": \"Embedding-modellen trunkerer input over token-grensen, og chunks må holdes under 8191 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#19\",\n    \"claim\": \"Embedding-generering i Azure OpenAI for denne pipelinen bruker modellene text-embedding-3-small og text-embedding-3-large.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#20\",\n    \"claim\": \"Semantic Kernel tilbyr en TextChunker-klasse for kode-basert chunking.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#21\",\n    \"claim\": \"Azure AI Search er tilgjengelig i Norway East, slik at data forblir i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#22\",\n    \"claim\": \"Document Intelligence er tilgjengelig i West Europe, altså i EU, men ikke i Norge spesifikt.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#23\",\n    \"claim\": \"Document Intelligence Layout støtter norsk tekst i OCR og strukturgjenkjenning.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#24\",\n    \"claim\": \"Embedding-modellene i text-embedding-3-serien støtter norsk.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#25\",\n    \"claim\": \"Text Split skill er inkludert i Azure AI Search og medfører ingen ekstra tjenestekostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#26\",\n    \"claim\": \"Document Layout skill faktureres etter Document Intelligence Standard-tier per side.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/document-intelligence/concept/retrieval-augmented-generation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/chunking-strategies.md#27\",\n    \"claim\": \"Token chunking er i preview og tilgjengelig i API-versjonen 2026-05-01-preview (Skillsets - Create or Update).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/chunking-strategies.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/contextual-retrieval.md",
    "claim_count": 15,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/contextual-retrieval.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/contextual-retrieval.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#1\",\n    \"claim\": \"Contextual Retrieval har status GA for custom skill-mønsteret og Preview for agentic retrieval.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#2\",\n    \"claim\": \"Kontekstuell prefiks-generering består av fire komponenter: Custom Web API Skill (Azure Function som mottar chunk + metadata og returnerer kontekstuell prefiks) | LLM (GPT-4o) | Text Merge Skill | Embedding Skill.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#3\",\n    \"claim\": \"Azure OpenAI-klienten i custom skill-eksempelet bruker api_version «2024-10-01-preview».\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#4\",\n    \"claim\": \"Contextual retrieval i Microsoft-stakken integrerer med: Azure AI Search (skillset-pipeline) | Azure OpenAI | Azure Functions | Azure AI Document Intelligence (Document Layout skill) | Semantic Kernel (TextChunker) | Application Insights.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#5\",\n    \"claim\": \"Azure OpenAI leverer GPT-4o til kontekst-generering og text-embedding-3-large til embedding.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#6\",\n    \"claim\": \"Custom skill hostes på Azure Functions consumption plan (Python/C#).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#7\",\n    \"claim\": \"Den fullstendige skillset-pipelinen består av fem trinn: Document Layout skill (struktur til Markdown) | Text Split skill | Custom Context Generation skill (GPT-4o) | Text Merge skill | Azure OpenAI Embedding skill (text-embedding-3-large).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#8\",\n    \"claim\": \"Azure AI Search er tilgjengelig i Norway East, slik at data ligger i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#9\",\n    \"claim\": \"Azure Functions er tilgjengelig i Norway East, slik at compute skjer i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#10\",\n    \"claim\": \"Azure OpenAI må kjøres i Sweden Central som nærmeste region med GPT-4o; data forblir i EU/EØS.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#11\",\n    \"claim\": \"Custom skills integreres i Azure AI Search enrichment pipeline via skilltypen #Microsoft.Skills.Custom.WebApiSkill.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#12\",\n    \"claim\": \"Et custom skill må eksponere et HTTPS-endepunkt, hostet som Azure Functions | containere | annen Azure-hostet tjeneste.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#13\",\n    \"claim\": \"Custom skill-grensesnittet krever JSON-batch inn på formen {\\\"values\\\": [{\\\"recordId\\\": ..., \\\"data\\\": {...}}, ...]} og JSON-batch ut med feltene recordId | data | errors | warnings.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#14\",\n    \"claim\": \"Timeout for custom skill er 30 sekunder som standard og maksimalt 230 sekunder (PT230S).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/contextual-retrieval.md#15\",\n    \"claim\": \"Autentisering mot custom skill skjer enten med API-nøkkel i URI/header eller med managed identity via authResourceId.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-custom-skill-interface\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/contextual-retrieval.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/embedding-models-selection.md",
    "claim_count": 25,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/embedding-models-selection.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/embedding-models-selection.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#1\",\n    \"claim\": \"Filen angir status GA for Azure OpenAI og Azure AI Search, og Preview for Multilingual E5 og Custom embeddings.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#2\",\n    \"claim\": \"Azure OpenAI Service tilbyr embedding-modellene text-embedding-ada-002 | text-embedding-3-small | text-embedding-3-large.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#3\",\n    \"claim\": \"text-embedding-ada-002, text-embedding-3-small og text-embedding-3-large har alle status GA i Azure OpenAI.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#4\",\n    \"claim\": \"text-embedding-ada-002 har 1536 dimensjoner og maks 8191 tokens per input.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#5\",\n    \"claim\": \"text-embedding-3-small har 1536 dimensjoner (kan reduseres til 512) og maks 8191 tokens per input.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#6\",\n    \"claim\": \"text-embedding-3-large har 3072 dimensjoner (kan reduseres til 256-1024) og maks 8191 tokens per input.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#7\",\n    \"claim\": \"multilingual-e5-small har 384 dimensjoner og maks 512 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#8\",\n    \"claim\": \"multilingual-e5-small og multilingual-e5-large har status Preview og tilbys via Azure AI.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#9\",\n    \"claim\": \"multilingual-e5-large har 1024 dimensjoner og maks 512 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#10\",\n    \"claim\": \"Custom embeddings for domene-spesifikk fine-tuning har status Announced, med variabel dimensjonalitet og variabel maks token-kapasitet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#11\",\n    \"claim\": \"OpenAI-embeddingmodellene håndterer 100+ språk, men med fallende kvalitet utenfor engelsk, mens E5-modellene er optimalisert for flerspråklig kvalitet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#12\",\n    \"claim\": \"text-embedding-3-modellene støtter Matryoshka-dimensjonalitet, altså dynamisk reduksjon av dimensjoner (f.eks. fra 3072 til 512) uten retraining.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#13\",\n    \"claim\": \"Azure OpenAI embeddings-kall i eksempelet bruker api_version 2024-02-01.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#14\",\n    \"claim\": \"Azure AI Search integrated vectorization bruker skill-typen #Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill, konfigurert med feltene resourceUri | deploymentId | modelName | dimensions | inputs | outputs.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#15\",\n    \"claim\": \"Copilot Studio kan kobles til Azure AI Search som knowledge source; embeddings genereres automatisk ved opplasting, og standardmodellen (typisk ada-002 eller text-embedding-3-small) kan ikke endres direkte i UI.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#16\",\n    \"claim\": \"Azure OpenAI-embeddings prosesseres i Azure-regionen modellen er deployet i (typisk West Europe eller North Europe), og no-logging-policyen gir ikke garanti for EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#17\",\n    \"claim\": \"Custom embeddings kan self-hostes i Azure Norway-regioner, noe som anbefales for data klassifisert Høy/Kritisk.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#18\",\n    \"claim\": \"Azure AI Search støtter sletting av dokumenter, men ikke selektiv sletting av embeddings uten reindeksering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#19\",\n    \"claim\": \"Et batch-kall til embeddings-API-et kan sende ca. 100 dokumenter per API-kall, med maks 8191 tokens totalt.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#20\",\n    \"claim\": \"Azure OpenAI Service krever Azure-abonnement, og det finnes ingen separate lisenser for embedding-modellene.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#21\",\n    \"claim\": \"Azure AI Search Basic tier gir 15 GB storage per partisjon for nye tjenester (2 GB for eldre tjenester), og Standard S1 gir 160 GB per partisjon (25 GB for eldre tjenester).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#22\",\n    \"claim\": \"Microsoft Foundry støtter fine-tuning av embedding-modeller via Custom Models, som er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#23\",\n    \"claim\": \"Fine-tuning av embeddings i Microsoft Foundry støtter modellene text-embedding-3-small | text-embedding-3-large, med treningsdata i JSON Lines-format bestående av query-document pairs.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#24\",\n    \"claim\": \"Fine-tuning av embedding-modeller krever minimum 100 positive query-document-par, med 1000+ som anbefalt antall.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/embedding-models-selection.md#25\",\n    \"claim\": \"Evaluering av fine-tunede embedding-modeller i Microsoft Foundry skjer med metrikkene Recall@k | NDCG | MRR mot valideringssett.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/vector-search-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/embedding-models-selection.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/graphrag-knowledge-graphs.md",
    "claim_count": 16,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/graphrag-knowledge-graphs.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/graphrag-knowledge-graphs.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#1\",\n    \"claim\": \"Entity extraction i GraphRAG gjøres med Azure AI Language Service Named Entity Recognition (NER) v3 og GenAI Prompt skill.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-entity-recognition-v3\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#2\",\n    \"claim\": \"Microsoft Fabric Graph er i Public Preview og bruker Labeled Property Graph (LPG)-modellen.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/graph/graph-data-models\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#3\",\n    \"claim\": \"Entity Recognition (v3) i Azure AI Search ekstraherer 14 kategorier, blant dem Person | Organization | Location | Quantity | DateTime | URL | Email.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-entity-recognition-v3\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#4\",\n    \"claim\": \"Entity normalization legger til Wikipedia IDs | Bing IDs | confidence scores for datakvalitet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-entity-recognition-v3\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#5\",\n    \"claim\": \"I Labeled Property Graph (LPG)-modellen har nodes labels (typer) | properties (attributter) | unique IDs, og edges har types | properties | retning.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/graph/graph-data-models\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#6\",\n    \"claim\": \"I Microsoft Fabric Graph krever schema-endringer per 2026 at man oppretter en ny graph model og laster data på nytt.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/graph/graph-data-models\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#7\",\n    \"claim\": \"Resource Description Framework (RDF) er ikke støttet i Microsoft Fabric per 2026.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/graph/graph-data-models\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#8\",\n    \"claim\": \"CosmosAIGraph er en multi-model database som lagrer document | vector | graph i samme container.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/cosmos-ai-graph\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#9\",\n    \"claim\": \"CosmosAIGraph eksponerer Gremlin API (graph traversal) | SQL API (document queries).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/cosmos-ai-graph\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#10\",\n    \"claim\": \"Knowledge base API i Azure AI Search er en preview-feature for agentic retrieval.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/responsible-ai/search/transparency-note\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#11\",\n    \"claim\": \"Azure OpenAI-modellen text-embedding-3-large brukes som embedding-modell for vector-komponenten i hybrid GraphRAG.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#12\",\n    \"claim\": \"GPT-4 og Opus anbefales for kompleks graph-resonnering (path explanations, multi-hop inferenser).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-solution-design-and-evaluation-guide\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#13\",\n    \"claim\": \"Microsoft Fabric og Azure Data Explorer støtter KQL graph operators: make-graph | graph-match | graph-shortest-paths | graph-to-table.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/kusto/query/graph-semantics-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#14\",\n    \"claim\": \"GraphRAG-data lagret i Azure Norway (oslo-region) oppfyller EU data residency-kravene under Schrems II.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/kusto/query/graph-semantics-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#15\",\n    \"claim\": \"Semantic ranking i Azure AI Search har en tier på 1000 queries per måned.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/kusto/query/graph-semantics-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/graphrag-knowledge-graphs.md#16\",\n    \"claim\": \"CosmosAIGraph er en GitHub-basert løsning i preview, ikke en fullt integrert Azure-tjeneste, per februar 2026.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/cosmos-ai-graph\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/graphrag-knowledge-graphs.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/hierarchical-rag-patterns.md",
    "claim_count": 15,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/hierarchical-rag-patterns.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/hierarchical-rag-patterns.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#1\",\n    \"claim\": \"Index projections er GA, mens agentic retrieval er i Preview (per 2026-06-19).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#2\",\n    \"claim\": \"Azure AI Search tilbyr tre arkitekturmønstre for parent-child indeksering: Single index, repeating parent fields | Single index, mixed document shapes | Separate parent-child indexes.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#3\",\n    \"claim\": \"Index projections konfigureres i skillsettet under indexProjections med selectors (targetIndexName | parentKeyFieldName | sourceContext | mappings) og parameters (projectionMode).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#4\",\n    \"claim\": \"Parameteren projectionMode har verdiene skipIndexingParentDocuments | includeIndexingParentDocuments, som styrer om kun chunks eller også parent-dokumentene indekseres.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#5\",\n    \"claim\": \"Azure AI Search genererer chunk-IDer automatisk fra parent-IDen etter mønsteret parentId_pages_N, f.eks. aa1b22c33_pages_0 | aa1b22c33_pages_1, og hash-komponenten endres ved parent-oppdatering slik at change tracking sikres.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#6\",\n    \"claim\": \"Index-schemaet for chunk-indeksen bruker felttypene Edm.String | Collection(Edm.Single) | Edm.Int32 med feltattributtene key | filterable | searchable.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-howto-complex-data-types\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#7\",\n    \"claim\": \"Azure AI Search-datakilder støtter dataDeletionDetectionPolicy, som gjør det mulig å konfigurere kaskade-sletting slik at chunks slettes når kildedokumentet slettes.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#8\",\n    \"claim\": \"Azure AI Document Intelligence Layout-skill støtter parameteren markdownHeaderDepth med verdien h3.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#9\",\n    \"claim\": \"Semantic Kernel tilbyr TextSearchProvider med namespace-filtrering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#10\",\n    \"claim\": \"Microsoft anbefaler per 2026-06 Azure Content Understanding-skill (#Microsoft.Skills.Util.ContentUnderstandingSkill) for nye skillsets i stedet for Document Layout-skill; Document Layout-skill er fortsatt støttet og dokumentert for eksisterende pipelines.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#11\",\n    \"claim\": \"Document Layout-skill identifiseres med @odata.type #Microsoft.Skills.Util.DocumentIntelligenceLayoutSkill og konfigureres med outputMode oneToMany, input file_data og output markdown_document.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-semantic-chunking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#12\",\n    \"claim\": \"Azure AI Search er tilgjengelig i Norway East, slik at den hierarkiske indeksen forblir i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#13\",\n    \"claim\": \"Azure AI Document Intelligence kjøres i West Europe, slik at dokument-parsing skjer i EU.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#14\",\n    \"claim\": \"Index projections er inkludert i Azure AI Search uten ekstra kostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hierarchical-rag-patterns.md#15\",\n    \"claim\": \"Vektorfelt i Azure AI Search kan settes med stored: false for å spare lagringsplass.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/hierarchical-rag-patterns.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/hybrid-search-configuration.md",
    "claim_count": 18,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/hybrid-search-configuration.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/hybrid-search-configuration.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#1\",\n    \"claim\": \"Hybrid search i Azure AI Search har vært GA siden 2023.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#2\",\n    \"claim\": \"Score-egenskaper og algoritmer i Azure AI Search: full-text = `@search.score` med BM25 (område 0 til ubegrenset) | vector = `@search.score` med HNSW/KNN (område 0.333–1.00 for cosine) | hybrid = `@search.score` med RRF (område 0 til ~1/k per query) | semantic ranking = `@search.rerankerScore` med ML comprehension (område 0.00–4.00).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-relevance-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#3\",\n    \"claim\": \"Vector queries i Azure AI Search støtter en `weight`-parameter med default-verdi 1.0, der 0.5 halverer og 2.0 dobler vektorens bidrag til RRF-scoren.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#4\",\n    \"claim\": \"`maxTextRecallSize` kontrollerer hvor mange BM25-resultater som mates inn i RRF, og har default-verdi 1000.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#5\",\n    \"claim\": \"Semantic ranker i Azure AI Search rangerer de 50 øverste resultatene fra RRF-fusjonen.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-relevance-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#6\",\n    \"claim\": \"Semantic ranking gir 1000 gratis spørringer per måned; utover dette påløper ekstra kostnad per query.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-relevance-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#7\",\n    \"claim\": \"Integrasjoner mot Microsoft-stakken: Azure OpenAI (embedding-modeller som text-embedding-3-large for vektordelen) | Microsoft Foundry (integrert vektorisering via skills i indekserings-pipeline) | Copilot Studio (Azure AI Search som grounding-kilde for Copilot-agenter) | Semantic Kernel (`AzureAISearchVectorStore`-connector for hybrid queries) | Power Platform (AI Builder kan bruke Azure AI Search via custom connectors).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#8\",\n    \"claim\": \"Azure AI Search er tilgjengelig i regionene Norway East og Norway West.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#9\",\n    \"claim\": \"All indeksdata i Azure AI Search forblir i valgt region, og Microsofts EU Data Boundary gjelder for norske deployments.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#10\",\n    \"claim\": \"Sikkerhetsfunksjoner i Azure AI Search: Azure Private Link (isolerer search service fra offentlig internett) | Managed Identity med autentisering via Entra ID (uten API-nøkler) | customer-managed keys i Azure Key Vault | dokumentnivå-sikkerhet (filtrering per brukeridentitet) | RBAC for indeks- og spørringsoperasjoner.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#11\",\n    \"claim\": \"Minimumstier i Azure AI Search: hybrid search (BM25 + vektor) krever Basic | scoring profiles er tilgjengelig på alle tier | semantic ranking krever S1 eller høyere (1000 gratis per måned) | integrert vektorisering krever Basic eller høyere.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#12\",\n    \"claim\": \"Scalar/binary quantization i Azure AI Search er i preview og reduserer vektorlagring med opptil 50 %.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#13\",\n    \"claim\": \"API-kallet støtter `debug`-parameteren med verdiene `vector` og `semantic` for å pakke ut subscores per rankingkomponent.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#14\",\n    \"claim\": \"`maxTextRecallSize`, som kontrollerer BM25-bidraget til RRF, er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#15\",\n    \"claim\": \"`maxTextRecallSize` kan settes til maksimalt 10000.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#16\",\n    \"claim\": \"Filter-moduser i hybrid search: `preFilter` (default, filtrerer før query for begge subqueries) | `postFilter` (filtrerer etter query) | `strictPostFilter` (preview) | `filterOverride` (preview, per-vectorQuery-filter som overstyrer globalt filter).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#17\",\n    \"claim\": \"SDK-støtte for hybrid search: Python (`azure-search-documents`) | C# (`Azure.Search.Documents`) | Java | JavaScript.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/hybrid-search-configuration.md#18\",\n    \"claim\": \"API-versjoner: `2026-04-01` (stable) og `2026-05-01-preview` (som inneholder maxTextRecallSize, filterOverride, strictPostFilter m.m.).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-how-to-query\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/hybrid-search-configuration.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/late-chunking-patterns.md",
    "claim_count": 13,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/late-chunking-patterns.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/late-chunking-patterns.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#1\",\n    \"claim\": \"Late chunking-støtte er GA i Jina Embeddings og i Preview via Azure Marketplace.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#2\",\n    \"claim\": \"Jina Embeddings v3 og v4 er tilgjengelig på Azure via Azure Marketplace.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#3\",\n    \"claim\": \"Azure OpenAI sine embedding-modeller (text-embedding-3) eksponerer per i dag ikke token-level embeddings.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#4\",\n    \"claim\": \"text-embedding-3-large har kontekstlengde 8 191 tokens og 3 072 dimensjoner.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#5\",\n    \"claim\": \"text-embedding-3-large er GA via Azure OpenAI.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#6\",\n    \"claim\": \"text-embedding-3-small har kontekstlengde 8 191 tokens og 1 536 dimensjoner.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#7\",\n    \"claim\": \"text-embedding-3-small er GA via Azure OpenAI.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#8\",\n    \"claim\": \"Azure AI Search støtter vektorindeks med egendefinert embedding via push API | custom skill.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#9\",\n    \"claim\": \"Jina Embeddings fra Azure Marketplace kan deployes i Norway East, slik at data forblir i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#10\",\n    \"claim\": \"Azure OpenAI Embeddings kjøres i Sweden Central, med data i EU/EØS.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#11\",\n    \"claim\": \"Azure AI Search er tilgjengelig i Norway East, slik at indeksen ligger i Norge.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#12\",\n    \"claim\": \"Jina Embeddings på Azure deployes som Azure Container Instance med consumption-basert prismodell.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/late-chunking-patterns.md#13\",\n    \"claim\": \"Bruk av Jina Embeddings på Azure krever ingen Azure OpenAI-kvote.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/tutorials/embeddings\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/late-chunking-patterns.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/metadata-management-filtering.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/metadata-management-filtering.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/metadata-management-filtering.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#1\",\n    \"claim\": \"Referansen angir status GA for metadata-håndtering og filtrert søk i Azure AI Search (OData-filtre og faceted navigation).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#2\",\n    \"claim\": \"Azure AI Search støtter OData v4-syntaks for filtre, og filtrene fungerer sammen med keyword search, vector search og hybrid search.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-odata-filter-orderby-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#3\",\n    \"claim\": \"Metadata-felttyper i Azure AI Search med egenskaper: Edm.String (filterable, facetable, sortable) | Collection(Edm.String) (filterable, facetable) | Edm.Int32/Edm.Int64 (filterable, facetable, sortable) | Edm.DateTimeOffset (filterable, facetable, sortable) | Edm.Boolean (filterable, facetable) | Edm.GeographyPoint (filterable kun via geo.distance, ikke facetable).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-create-search-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#4\",\n    \"claim\": \"I REST API-et er felttypene String | DateTime | Boolean | numeriske typer filterable og facetable som standard (default attributes).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-create-search-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#5\",\n    \"claim\": \"Facet-parametere i Azure AI Search er count | sort | values | interval | timeoffset.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-faceted-navigation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#6\",\n    \"claim\": \"Facet-parameteren count angir maks antall facet-verdier som returneres, med standardverdi 10, og count:0 betyr ubegrenset.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-faceted-navigation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#7\",\n    \"claim\": \"Facet-parameteren sort støtter verdiene count | -count (etter frekvens) og value | -value (alfabetisk).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-faceted-navigation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#8\",\n    \"claim\": \"Hierarkisk faceted navigation (hierarchical facets med delimiter og levels) er en preview-funksjon i Azure AI Search, ikke GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-faceted-navigation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#9\",\n    \"claim\": \"Azure AI Search vector search har filtermodusene preFilter (filter FØR vector search) | postFilter (filter ETTER vector search) | strictPostFilter (preview).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#10\",\n    \"claim\": \"strictPostFilter — filter etter vector search med re-ranking — er en preview-funksjon.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#11\",\n    \"claim\": \"Azure AI Search har en grense for filter-kompleksitet på om lag 1000 klausuler.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-odata-filter\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#12\",\n    \"claim\": \"SharePoint-kolonnetyper mappes til Azure AI Search-typer slik: Single line of text → Edm.String | Choice → Edm.String | Managed Metadata → Collection(Edm.String) | Date and Time → Edm.DateTimeOffset | Person or Group → Collection(Edm.String), alle filterable og facetable.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#13\",\n    \"claim\": \"Dataverse-typer mappes til Azure AI Search-typer slik: Choice → Edm.String | Choices (multi-select) → Collection(Edm.String) | Lookup → Edm.String (ID eller Name) | DateTime → Edm.DateTimeOffset.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#14\",\n    \"claim\": \"Azure AI Search-indexer for Azure Blob Storage eksponerer blob index tags i feltet metadata_storage_blob_index_tags (Edm.String, filterable).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#15\",\n    \"claim\": \"Azure AI Search tilbys i tjenestenivåene Basic | S1 | S2 | S3/S3HD.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#16\",\n    \"claim\": \"Maks indeksstørrelse per tier i Azure AI Search: Basic 2 GB | S1 25 GB per partisjon | S2 100 GB per partisjon | S3/S3HD 200 GB per partisjon.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#17\",\n    \"claim\": \"Maks antall felt per indeks er 1000 for Basic, S1 og S2, og 3000 for S3/S3HD.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#18\",\n    \"claim\": \"OData-filteroperatorer i Azure AI Search: sammenligning eq | ne | gt | lt | ge | le; logisk and | or | not; samling any() | all(); geo-spatial geo.distance() | geo.intersects(); fulltekst search.ismatch() | search.ismatchscoring(); liste-matching search.in().\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-odata-filter-orderby-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#19\",\n    \"claim\": \"Presedensrekkefølgen for OData-filteroperatorer er not > sammenligningsoperatorer > and > or.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-odata-filter-orderby-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#20\",\n    \"claim\": \"search.in() teller som én klausul mot filter size limits i Azure AI Search, i motsetning til lange OR-kjeder.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-odata-filter\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#21\",\n    \"claim\": \"preFilter er standardmodus for filter i vector-/hybrid-søk i Azure AI Search og anvendes FØR query execution.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/metadata-management-filtering.md#22\",\n    \"claim\": \"filterOverride — per-vectorQuery filter som overstyrer det globale filteret — er en preview-funksjon.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-what-is-an-index\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/metadata-management-filtering.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/multi-index-federation.md",
    "claim_count": 13,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/multi-index-federation.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/multi-index-federation.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#1\",\n    \"claim\": \"Multi-index federation i Azure AI Search: enkelt-indeks-søk er GA, mens native cross-index-søk ikke er støttet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#2\",\n    \"claim\": \"Azure AI Search støtter ikke native multi-index federation — en spørring er alltid avgrenset til én enkelt indeks.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-faq-frequently-asked-questions\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#3\",\n    \"claim\": \"Azure AI Search er en single-region service; høyere reliability oppnås ved å deploye identiske search services i flere regioner.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#4\",\n    \"claim\": \"Synkronisering av data mellom Azure AI Search-services i flere regioner skjer via to API-tilnærminger: push-API | pull-API (indexer).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#5\",\n    \"claim\": \"Load balancing mellom Azure AI Search-services i flere regioner håndteres av Azure Front Door | Azure Traffic Manager | Azure Application Gateway.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#6\",\n    \"claim\": \"Data residency i Azure AI Search: innhold lagres i regionen du velger, uten kryssregional dataflyt uten eksplisitt autorisasjon.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#7\",\n    \"claim\": \"Azure AI Search støtter multiple vektorer i ett dokumentfelt via Collection(Edm.ComplexType); funksjonen (multi-vector fields) er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-multi-vector-fields\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#8\",\n    \"claim\": \"Multi-vector fields i Azure AI Search støtter opp til 100 vektorer per dokument.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-multi-vector-fields\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#9\",\n    \"claim\": \"Parameteren perDocumentVectorLimit kontrollerer antall matchende vektorer per dokument i query-resultater i Azure AI Search.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-multi-vector-fields\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#10\",\n    \"claim\": \"Multi-vector field support i Azure AI Search er GA (2025).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-multi-vector-fields\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#11\",\n    \"claim\": \"Agentic Retrieval i Azure AI Search er i Preview og gir LLM-assistert query planning, men er fremdeles single-index.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#12\",\n    \"claim\": \"Targeted vector filters i Azure AI Search er i Preview — filtre spesifikt for vektor-subqueries.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-multi-region\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multi-index-federation.md#13\",\n    \"claim\": \"Eksempelet for LLM-basert query routing bruker modellen gpt-4o-mini via chat.completions-API-et.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-multi-region\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/multi-index-federation.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/multimodal-rag.md",
    "claim_count": 23,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/multimodal-rag.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/multimodal-rag.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#1\",\n    \"claim\": \"Document Intelligence og Content Understanding er GA, mens multimodale embeddings er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#2\",\n    \"claim\": \"Azure-stakken tilbyr tre komplementære tilnærminger til multimodal RAG: Image verbalization (GPT-4o/4v konverterer bilder til tekst) | direkte multimodale embeddings (Azure Vision genererer vektorer for bilder og tekst i samme vektorrom) | Azure Content Understanding (konverterer komplekse dokumenter til Markdown).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#3\",\n    \"claim\": \"Azure Content Understanding ble GA i november 2025 og konverterer komplekse dokumenter til Markdown med LaTeX-ligninger, HTML-tabeller og semantic chunking.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#4\",\n    \"claim\": \"Document Extraction-skillen støtter kun PDF, ekstraherer bilder, men ikke tekst og ikke tabeller.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/tutorial-document-extraction-multimodal-embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#5\",\n    \"claim\": \"Document Layout-skillen dekker: tekst Ja | bilder Ja | tabeller Nei | kryss-sideenheter kun innen side | flere formater støttet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#6\",\n    \"claim\": \"Content Understanding-skillen dekker: tekst Ja | bilder Ja | tabeller Ja (også kryss-side) | kryss-sideenheter Ja | formater PDF, DOCX, XLSX, PPTX.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#7\",\n    \"claim\": \"Azure Vision multimodal embedding via Microsoft Foundry gir 1024 dimensjoner per vektor, for både tekst og bilde.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/tutorial-document-extraction-multimodal-embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#8\",\n    \"claim\": \"Text embedding-skillen i verbaliseringsmønsteret bruker modellen text-embedding-3-large.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#9\",\n    \"claim\": \"Azure Vision Vectorize-skillen (@odata.type #Microsoft.Skills.Vision.VectorizeSkill) konfigureres med modelVersion 2023-04-15.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/tutorial-document-extraction-multimodal-embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#10\",\n    \"claim\": \"Content Understanding konverterer dokumenter til GitHub Flavored Markdown.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#11\",\n    \"claim\": \"Markdown-representasjonen i Content Understanding: Tabeller = HTML-markup med rowspan/colspan | Ligninger = LaTeX | Diagrammer = Chart.js JSON eller Mermaid | Bilder = markdown-bildesyntaks med valgfri beskrivelse og analyse | Sidemetadata = HTML-kommentarer (f.eks. PageNumber).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#12\",\n    \"claim\": \"Konfigurasjonsparametere for Content Understanding i RAG-pipelines: outputContentFormat=markdown | enableFigureAnalysis=true | enableAnnotation=true | chartFormat=markdown.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/content-understanding/document/markdown\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#13\",\n    \"claim\": \"GPT-4v har modellvarianten vision-preview og anbefales til bildeberikelse og summary-generering.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#14\",\n    \"claim\": \"Bildeekstraksjon i indexeren aktiveres med innstillingen imageAction: generateNormalizedImages.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#15\",\n    \"claim\": \"Free tier i Azure AI Search støtter ikke multimodal; minimum Basic tier kreves.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#16\",\n    \"claim\": \"Azure Vision brukes i West Europe slik at bildeprosessering skjer i EU.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#17\",\n    \"claim\": \"Azure AI Search-indeksen kan ligge i Norway East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#18\",\n    \"claim\": \"Microsoft Foundry-ressurs kreves for Vision multimodale embeddinger, og tilgjengeligheten er regionbegrenset.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/tutorial-document-extraction-multimodal-embeddings\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#19\",\n    \"claim\": \"Azure AI Search multimodal pipeline er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#20\",\n    \"claim\": \"Den multimodale pipelinen i Azure AI Search består av fem steg: Ekstraksjon | Tekst-chunking (Text Split skill) | Bildebeskriving (GenAI Prompt skill) | Embedding | Bildelagring i knowledge store.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#21\",\n    \"claim\": \"Embedding-steget i den multimodale pipelinen kan bruke Azure OpenAI | Microsoft Foundry | Azure Vision embedding.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#22\",\n    \"claim\": \"Hybride queries i Azure AI Search kombinerer full-text search | vector search | semantic ranking.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/multimodal-rag.md#23\",\n    \"claim\": \"GenAI Prompt skill-baserte pipelines støtter hybrid queries over tekst og verbaliserte bilder, mens bilde-til-vektor-queries krever Azure Vision multimodal embedding skill med en tilsvarende vectorizer.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/multimodal-search-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/multimodal-rag.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md",
    "claim_count": 21,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#1\",\n    \"claim\": \"Multi-layer caching i RAG dekker fire nivåer: result caching (hele LLM-responser) | retrieval caching (knowledge fragments fra vektorsøk) | embedding caching (forhåndsberegnede vektorrepresentasjoner) | semantic caching (semantisk like prompts).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/ai/application-design#implement-multi-layer-caching-strategies\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#2\",\n    \"claim\": \"Microsoft-stakken tilbyr disse cache-tjenestene for AI-workloads: Azure Cache for Redis (tradisjonell og semantisk caching) | Azure Cosmos DB (semantisk cache med vektorsøk) | Azure AI Search (innebygd caching av søkeresultater) | Azure API Management (semantisk caching for LLM-API-er).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#3\",\n    \"claim\": \"Multi-layer caching-tabellen lister fire cache-lag: Result caching (cache hele LLM-responser) | Retrieval caching (cache knowledge fragments fra vector search) | Embedding caching (cache forhåndsberegnede embeddings) | Model output caching (cache intermediate model outputs).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/ai/application-design#implement-multi-layer-caching-strategies\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#4\",\n    \"claim\": \"Effektive cache keys må inkludere fem komponenter: tenant/user identity (multi-tenant security) | policy context (RBAC og data access policies) | model version | prompt version | context window (chat history).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/ai/application-design#implement-multi-layer-caching-strategies\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#5\",\n    \"claim\": \"Cache invalidation utløses av fire triggere: data updates (webhook-triggered ved endring i kildedata) | model changes (ved model deployment/retraining) | prompt modifications (ved endring av prompt-template) | manual purge (admin-utløst for compliance eller testing).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/well-architected/ai/application-design#implement-multi-layer-caching-strategies\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#6\",\n    \"claim\": \"Semantisk caching krever en vector-capable cache: Azure Cosmos DB | Redis med RediSearch.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/semantic-cache\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#7\",\n    \"claim\": \"Azure Cache for Redis-tiers: Premium tier (99,9 % SLA, opptil 120 GB per shard) | Enterprise tier (99,99 % SLA, active-active geo-replikering, Flash-storage-støtte) | Enterprise Flash tier (opptil 13 TB cache-størrelse, 20 % RAM + 80 % NVMe Flash).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#8\",\n    \"claim\": \"Flash-tier egner seg for read-heavy workloads (høyt read/write-forhold) | hot/cold-aksessmønstre (ofte aksessert subset) | store verdier (nøkler i RAM, verdier i Flash), og egner seg ikke for write-heavy workloads | uniforme aksessmønstre | lange nøkkelnavn med små verdier.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#9\",\n    \"claim\": \"Semantisk caching i Azure API Management gjelder LLM-API-er: Azure OpenAI | Model Inference API.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#10\",\n    \"claim\": \"Forutsetninger for semantisk caching i APIM: Azure Managed Redis med RediSearch-modul aktivert | deployment av Embeddings API (for vektorisering) | deployment av Chat Completion API (for brukerforespørsler).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#11\",\n    \"claim\": \"APIM-policyene for semantisk cache er azure-openai-semantic-cache-lookup (inbound, med attributtene score-threshold, embeddings-backend-id, embeddings-backend-auth, ignore-system-messages, max-message-count og underelementet vary-by) og azure-openai-semantic-cache-store (outbound, med attributtet duration).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#12\",\n    \"claim\": \"APIM-attributtet score-threshold er en distanse der lavere verdi gir strengere matching og krever høyere semantisk likhet: 0,1–0,2 = strict matching | 0,3–0,5 = balanced | 0,6–0,8 = liberal matching.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/azure-openai-enable-semantic-caching\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#13\",\n    \"claim\": \"Azure Cosmos DB for NoSQL støtter systemfunksjonen VectorDistance() i SQL-queries (med SELECT TOP og ORDER BY) for oppslag mot semantisk cache.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/semantic-cache\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#14\",\n    \"claim\": \"Azure Cosmos DB som semantisk cache gir: global distribusjon med multi-region writes | automatisk indeksering av vektorer | 99,999 % SLA med multi-region-oppsett | innebygd TTL-støtte.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cosmos-db/gen-ai/semantic-cache\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#15\",\n    \"claim\": \"Høyere Azure AI Search-tiers (S2, S3) gir mer minne og dermed større cache-kapasitet.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-performance-tips\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#16\",\n    \"claim\": \"Azure Cache for Redis i Premium- og Enterprise-tier har automatisk kryptering at rest.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#17\",\n    \"claim\": \"TLS 1.2 eller nyere er påkrevd for alle cache-tilkoblinger (in transit).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#18\",\n    \"claim\": \"Microsoft Entra ID-autentisering for Redis er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/python-get-started#code-to-connect-to-a-redis-cache\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#19\",\n    \"claim\": \"Azure Cache for Redis SKU-er med kapasitet: Basic C0 (250 MB, ingen SLA) | Standard C1 (1 GB, 2 replicas, 99,9 % SLA) | Premium P1 (6 GB, clustering, geo-replikering) | Enterprise E10 (12 GB, active-active, 99,99 % SLA) | Enterprise Flash F300 (345 GB, 20 % RAM + 80 % Flash).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#20\",\n    \"claim\": \"Azure Managed Redis (basert på Redis Enterprise) skiller seg fra Azure Cache for Redis (community edition) på: threading (multi-threaded vs single-threaded) | arkitektur (flere shards per node med distribuerte primaries vs primary + replica med 2 noder) | skalering (nær-lineær med vCPU-er vs begrenset av én tråd) | clustering (alltid aktivert vs valgfritt) | active geo-replication (ja vs nei).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-caching-optimization.md#21\",\n    \"claim\": \"Azure Managed Redis har tre cluster policies: OSS policy (klienten kobles direkte til shards, laveste latency og best throughput) | Enterprise policy (ett endpoint, bakoverkompatibelt, påkrevd for RediSearch) | Non-Clustered (kun ≤25 GB, for migrering fra ikke-shardede miljøer).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/redis/architecture#flash-optimized-tier\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-caching-optimization.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-core-patterns.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-core-patterns.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-core-patterns.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#1\",\n    \"claim\": \"Azure OpenAI leverer embedding-generering for RAG med modellene text-embedding-3-large | text-embedding-ada-002.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#2\",\n    \"claim\": \"Vector store for RAG i Microsoft-stakken dekkes av Azure AI Search | Azure Cosmos DB (MongoDB vCore).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#3\",\n    \"claim\": \"Azure AI Search støtter tre retrieval-modi for RAG: vector search | hybrid search | semantic search.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#4\",\n    \"claim\": \"Generering i RAG gjøres med Azure OpenAI Service-modellene GPT-4 | GPT-4o.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#5\",\n    \"claim\": \"GPT-4 Turbo er en long-context-modell med 128K tokens kontekstvindu.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#6\",\n    \"claim\": \"Azure AI Search tilbyr disse funksjonene for RAG: Vector search (cosine similarity, HNSW-indeksering) | Hybrid search (vector + BM25) | Semantic Ranker (L2-reranking basert på cross-encoder) | Metadata filtering | Skillset API.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#7\",\n    \"claim\": \"Skillset API i Azure AI Search utfører document cracking | OCR | entity extraction før indeksering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#8\",\n    \"claim\": \"Prompt flow i Microsoft Foundry pensjoneres 2027-04-20, og migrering anbefales til Microsoft Agent Framework (MAF).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-studio/how-to/flow-develop\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#9\",\n    \"claim\": \"Microsoft Foundry har innebygde evalueringsmetrikker: groundedness | relevance | coherence.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-studio/how-to/flow-develop\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#10\",\n    \"claim\": \"Semantic Kernel memory connectors gir abstraksjon over Azure AI Search | Cosmos DB | Qdrant.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/memories/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#11\",\n    \"claim\": \"Copilot Studio tilbyr for RAG: generative answers (low-code RAG med Azure AI Search + SharePoint) | knowledge sources (drag-and-drop indeksering av dokumenter og nettsteder) | conversation boosting (automatisk fallback til RAG når intent ikke matches).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/nlu-boost-conversations\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#12\",\n    \"claim\": \"Azure har norske regioner Norway East og Norway West som kan brukes for dataresidens i RAG-løsninger.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#13\",\n    \"claim\": \"Azure AI Search S1-tier er SKU-en angitt for en RAG-løsning med 10M vektorer, og Semantic Ranker kommer som et separat tillegg.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#14\",\n    \"claim\": \"Hybrid search i Azure AI Search er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#15\",\n    \"claim\": \"Semantic ranking i Azure AI Search er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#16\",\n    \"claim\": \"Integrert vektorisering (Azure OpenAI integrert med Azure AI Search) er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-integrated-vectorization-ai-studio\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#17\",\n    \"claim\": \"Generative answers i Copilot Studio er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/nlu-boost-conversations\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#18\",\n    \"claim\": \"Semantic Kernel memory og embeddings er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/memories/\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#19\",\n    \"claim\": \"Prompt flow for RAG i Microsoft Foundry er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-studio/how-to/flow-develop\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#20\",\n    \"claim\": \"Microsoft Agent Framework har preview-status.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#21\",\n    \"claim\": \"Hybrid search i Azure AI Search bruker en RRF-merger som normaliserer scores fra BM25 og HNSW/eKNN.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-core-patterns.md#22\",\n    \"claim\": \"Semantic ranker (L2) i Azure AI Search re-ranker opptil 50 resultater.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-core-patterns.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-cost-optimization.md",
    "claim_count": 18,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-cost-optimization.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-cost-optimization.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#1\",\n    \"claim\": \"Azure AI Search tilbys i pricing-tierene Free | Basic | S1 | S2 | S3 HD | L1/L2.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#2\",\n    \"claim\": \"Lagringskapasitet per Azure AI Search-tier: Free 50 MB | S1 160 GB per partisjon | S2 512 GB per partisjon | S3 HD 1 024 GB per partisjon | L1 2 TB | L2 4 TB.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#3\",\n    \"claim\": \"Basic-tier i Azure AI Search har 15 GB lagring per partisjon for tjenester opprettet etter april 2024; eldre tjenester har 2 GB.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#4\",\n    \"claim\": \"Tier switching er støttet i Azure AI Search: du kan bytte direkte mellom Basic og Standard (S1/S2/S3), forutsatt at konfigurasjonen ikke overstiger target-tier og at regionen har kapasitet.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#5\",\n    \"claim\": \"Azure AI Search har to prismodeller: Dedicated (fast pris per Search Unit) og Serverless, som per juni 2026 er i preview (Serverless Developer), uten SLA og uten støtte for migrering til eller fra Dedicated.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#6\",\n    \"claim\": \"Serverless Developer (preview) i Azure AI Search er kun tilgjengelig i regionene West Central US | Switzerland North | Japan East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#7\",\n    \"claim\": \"Azure OpenAI Batch API gir 50 % rabatt sammenlignet med standard API.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/batch\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#8\",\n    \"claim\": \"Azure OpenAI Batch API har 24-timers SLA og egen separat quota, uten påvirkning på online workloads.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/batch\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#9\",\n    \"claim\": \"Embedding-modeller og antall dimensjoner: text-embedding-ada-002 1536 | text-embedding-3-small 512-1536 | text-embedding-3-large 1024-3072 | Multilingual-e5-large 1024.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-generate-embeddings#understand-embedding-economics\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#10\",\n    \"claim\": \"text-embedding-ada-002 er legacy og deprecated.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-generate-embeddings#understand-embedding-economics\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#11\",\n    \"claim\": \"text-embedding-3-modellene (small og large) støtter dimensionality reduction (redusert antall dimensjoner).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-generate-embeddings#understand-embedding-economics\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#12\",\n    \"claim\": \"Scalar quantization reduserer vektorlagring med 75 % (float32 → int8).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#13\",\n    \"claim\": \"Binary quantization gir 96,875 % reduksjon i vektorlagring.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#14\",\n    \"claim\": \"Azure AI Search har innebygd vektorkomprimering som gir opptil 92,5 % kostnadsreduksjon.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#15\",\n    \"claim\": \"Azure AI Search støtter query-typene Vector only | Keyword only | Hybrid | Semantic ranking.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/retrieval-augmented-generation#cost-and-latency-considerations\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#16\",\n    \"claim\": \"Semantic ranker er en premium-funksjon i Azure AI Search som faktureres per query.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#17\",\n    \"claim\": \"Basic-tier i Azure AI Search kan skaleres til 3 replicas for høy tilgjengelighet.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-tier\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-cost-optimization.md#18\",\n    \"claim\": \"Search Units (SU) per tier i Azure AI Search: Free 1 SU med grense på 1 indeks | Basic 1-3 SU | S1, S2, S3, S3 HD, L1 og L2 1-36 SU.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-sku-manage-costs\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-cost-optimization.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-enterprise-scale.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-enterprise-scale.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-enterprise-scale.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#1\",\n    \"claim\": \"Azure AI Search-funksjonaliteten dokumentet dekker (indexers, replicas/partitions for indeksering og serving) har status GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#2\",\n    \"claim\": \"Azure AI Search har to skaleringsdimensjoner: replicas (serving og high availability) | partitions (storage og indexing throughput), og kombinasjonen utgjør search units (SU).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-capacity-planning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#3\",\n    \"claim\": \"Microsoft tilbyr to grunnleggende indekseringstilnærminger i Azure AI Search: push model (programmatisk opplasting via API) | pull model (indexers som henter data fra støttede datakilder).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#4\",\n    \"claim\": \"Én batch-request til Azure AI Search kan maksimalt inneholde 1000 dokumenter eller 16 MB.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#5\",\n    \"claim\": \"Azure.Search.Documents SDK er på versjon v11.7.0, med v11.8.0-beta.1 tilgjengelig som beta.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dotnet/api/azure.search.documents.models.indexdocumentsbatch\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#6\",\n    \"claim\": \"Standardverdien for indexerens batchSize varierer per datakilde: 1000 for SQL/Cosmos DB | 10 for Blob Storage.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-large-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#7\",\n    \"claim\": \"Change detection på en Azure AI Search data source støtter High Water Mark | Integrated Change Tracking | SQL Change Tracking.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#8\",\n    \"claim\": \"Indexer-schedule i Azure AI Search kan settes til intervaller som hvert 2., 5. og 15. minutt, hver time og hver dag.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#9\",\n    \"claim\": \"En indexer-kjøring har et vindu på 2 timer, og indexeren resumerer automatisk fra siste kjente stopppunkt hvis prosesseringen tar lengre tid enn vinduet.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-large-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#10\",\n    \"claim\": \"Antall parallelle indexers begrenses av antall search units: 1 SU = 1 samtidig indexer.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-how-to-large-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#11\",\n    \"claim\": \"SLA i Azure AI Search etter replica-antall: 1 replica = ingen SLA | 2 replicas = read-only SLA (99.9%) | 3+ replicas = read/write SLA (99.9%).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-capacity-planning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#12\",\n    \"claim\": \"Azure AI Search har ingen innebygd cross-region replication.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-capacity-planning\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#13\",\n    \"claim\": \"Pull model (indexers) i Azure AI Search støtter datakildene Azure Blob | SQL | Cosmos DB | SharePoint | OneLake.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#14\",\n    \"claim\": \"Storage per partition per tier i Azure AI Search: Basic 15 GB (services opprettet etter april 2024; eldre services 2 GB) | Standard S1 25 GB | Standard S2 100 GB | Standard S3 200 GB | Storage Optimized L1 1 TB.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#15\",\n    \"claim\": \"Basic-services opprettet etter 3. april 2024 støtter opptil 3 partitions × 3 replicas (maks 9 SU); eldre Basic-services var begrenset til 1 partition × 3 replicas.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity#service-limits\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#16\",\n    \"claim\": \"Azure AI Search tilbyr ved siden av Dedicated (replicas × partitions = Search Units) en ny forbruksbasert prismodell Serverless, som per juni 2026 er i Preview, uten SLA og uten migrering til/fra Dedicated.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#17\",\n    \"claim\": \"Serverless (Preview) for Azure AI Search er per juni 2026 kun tilgjengelig i West Central US | Switzerland North | Japan East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#18\",\n    \"claim\": \"Azure Monitor eksponerer metrikkene Queries per Second (QPS) | Indexing Failed Documents | Search Latency (p95) | Throttled Requests for Azure AI Search.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#19\",\n    \"claim\": \"Azure AI Search er tilgjengelig i de norske regionene Norway East (Oslo) | Norway West (Stavanger).\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/search-indexer-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-enterprise-scale.md#20\",\n    \"claim\": \"QPS-estimat per tier i Azure AI Search: Basic ~15 | Standard S1 ~15 | Standard S2 ~60 | Standard S3 ~120.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-capacity-planning\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-enterprise-scale.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-evaluation-frameworks.md",
    "claim_count": 17,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-evaluation-frameworks.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-evaluation-frameworks.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#1\",\n    \"claim\": \"Azure AI Evaluation SDK har status GA, mens agentiske evaluatorer og Groundedness Pro er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#2\",\n    \"claim\": \"Microsoft Foundry tilbyr et evalueringsrammeverk med over 30 (30+) innebygde evaluatorer.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#3\",\n    \"claim\": \"De RAG-spesifikke evaluatorene i Azure AI er: GroundednessEvaluator | GroundednessProEvaluator | RelevanceEvaluator | ResponseCompletenessEvaluator | RetrievalEvaluator | DocumentRetrievalEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#4\",\n    \"claim\": \"GroundednessProEvaluator bruker Azure AI Content Safety som metode (ikke LLM-judge) og gir forbedret groundedness.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#5\",\n    \"claim\": \"Evaluatorene for tekstuell likhet er: SimilarityEvaluator | F1ScoreEvaluator | BleuScoreEvaluator | RougeScoreEvaluator | MeteorScoreEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#6\",\n    \"claim\": \"Evaluatorene for generell kvalitet er: CoherenceEvaluator | FluencyEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#7\",\n    \"claim\": \"De agentiske evaluatorene i Azure AI Evaluation er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#8\",\n    \"claim\": \"De agentiske evaluatorene er: IntentResolutionEvaluator | ToolCallAccuracyEvaluator | TaskAdherenceEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#9\",\n    \"claim\": \"Azure AI Evaluation SDK distribueres som Python-pakken azure-ai-evaluation og installeres med `pip install azure-ai-evaluation`.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#10\",\n    \"claim\": \"MLflow 3 er versjonen som brukes til tracing og online evaluering, installert med `pip install mlflow`.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#11\",\n    \"claim\": \"Prompt Flow pensjoneres 2027-04-20 og erstattes av MAF (Microsoft Agent Framework).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#12\",\n    \"claim\": \"Standard evaluatorer har et token-budsjett på 800 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#13\",\n    \"claim\": \"RetrievalEvaluator har et token-budsjett på 1600 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#14\",\n    \"claim\": \"ToolCallAccuracyEvaluator har et token-budsjett på 3000 tokens.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#15\",\n    \"claim\": \"LLM-judges i evaluering bruker EU-hostede modeller for EU/EØS-arbeidsområder, og US-hostede modeller for andre regioner.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#16\",\n    \"claim\": \"Abuse monitoring kan man reservere seg mot (opt out) etter godkjenning.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-evaluation-frameworks.md#17\",\n    \"claim\": \"Azure AI Evaluation SDK er gratis (open source); kostnaden kommer fra Azure OpenAI-token-bruk for LLM-judge-kall.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/how-to/develop/evaluate-sdk\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-evaluation-frameworks.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-iterative-refinement.md",
    "claim_count": 18,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-iterative-refinement.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-iterative-refinement.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#1\",\n    \"claim\": \"Samtalehistorikk håndteres av `ChatHistory` i Semantic Kernel og `AgentSession` i Agent Framework.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#2\",\n    \"claim\": \"Session state i Microsoft-stakken dekkes av `AgentSession.Serialize()` (Agent Framework) og `WhiteboardProvider` (Semantic Kernel).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-memory\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#3\",\n    \"claim\": \"Semantic Kernel har to reducere for håndtering av context window / token limits: `ChatHistoryTruncationReducer` | `ChatHistorySummarizationReducer`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#4\",\n    \"claim\": \"Stateless (klient-styrt historikk) i Microsoft-stakken eksemplifiseres av Azure OpenAI Chat Completions | Semantic Kernel `ChatCompletionAgent`, der hele historikken sendes til tjenesten ved hver request.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dotnet/ai/ichatclient\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#5\",\n    \"claim\": \"Stateful (tjeneste-styrt historikk) eksemplifiseres av Azure AI Agent Service | Copilot Studio topics, der kun en referanse (conversation ID) sendes ved hver request.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dotnet/ai/ichatclient\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#6\",\n    \"claim\": \"`ChatHistorySummarizationReducer` i Semantic Kernel konstrueres med chatCompletionService, targetCount og thresholdCount, og kjøres via `ReduceAsync(chatHistory)`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#7\",\n    \"claim\": \"`WhiteboardProvider` i Semantic Kernel konfigureres via `WhiteboardProviderOptions` med egenskapen `MaxWhiteboardMessages`, og whiteboardet oppdateres automatisk når meldinger legges til `AgentThread`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-memory\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#8\",\n    \"claim\": \"LangChain-funksjonen `create_history_aware_retriever()` tar parameterne llm, retriever og prompt, og brukes sammen med Azure DocumentDB for history-aware retrieval.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/documentdb/rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#9\",\n    \"claim\": \"`ChatHistoryTruncationReducer` i Semantic Kernel konstrueres med targetCount og thresholdCount.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#10\",\n    \"claim\": \"Begge Semantic Kernel-reducerne (truncation og summarization) bevarer alltid system messages.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#11\",\n    \"claim\": \"Agent Framework tilbyr følgende API for multi-turn-samtaler: `agent.CreateSessionAsync()` | `agent.RunAsync(prompt, session)` | `session.Serialize()` som returnerer `JsonElement` | `agent.DeserializeSessionAsync(serializedSession)`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/agent-framework/tutorials/agents/multi-turn-conversation\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#12\",\n    \"claim\": \"Azure OpenAI Web App konfigurerer chat-historikk i Cosmos DB via miljøvariablene AZURE_COSMOSDB_ACCOUNT | AZURE_COSMOSDB_DATABASE | AZURE_COSMOSDB_CONTAINER.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/use-web-app\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#13\",\n    \"claim\": \"Feedback-innsamling i Azure OpenAI Web App aktiveres med miljøvariabelen AZURE_COSMOSDB_ENABLE_FEEDBACK satt til \\\"True\\\".\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/use-web-app\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#14\",\n    \"claim\": \"Copilot Studio håndterer multi-turn-tilstand via Topics som holder conversation state automatisk | Entities som ekstraheres og persisteres på tvers av turns | Conversation variables som lagrer brukerpreferanser og kontekst.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#15\",\n    \"claim\": \"EU-dataresidens for Azure Cosmos DB og Azure OpenAI oppnås i regionene Norway East | West Europe.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#16\",\n    \"claim\": \"Anbefalt modellvalg for kostnadsoptimalisering: GPT-3.5 for oppsummeringer og GPT-4 for svar-generering.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/semantic-kernel/concepts/ai-services/chat-completion/chat-history\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#17\",\n    \"claim\": \"`IChatClient` (Microsoft.Extensions.AI) er .NET-standarden for interaksjon med AI chat services og støtter stateless og stateful samtaler | tool calling | streaming | caching | OpenTelemetry.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dotnet/ai/ichatclient\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-iterative-refinement.md#18\",\n    \"claim\": \"Sentrale egenskaper i IChatClient-økosystemet: `ConversationId` for stateful tjenester (slipper å sende full historikk) | `FunctionInvokingChatClient` for automatisk tool invocation | `DistributedCachingChatClient` som cacher identiske historikker | `ChatClientBuilder` for pipeline-komposisjon av cache, tool invocation og telemetri.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dotnet/ai/ichatclient\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-iterative-refinement.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-query-understanding.md",
    "claim_count": 18,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-query-understanding.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-query-understanding.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#1\",\n    \"claim\": \"Query understanding-kapabilitetene beskrevet i filen er merket med status GA (generelt tilgjengelig).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#2\",\n    \"claim\": \"Azure AI Search støtter synonym maps der man f.eks. definerer «AI, kunstig intelligens, maskinlæring» for automatisk query expansion ved indexing-tid.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#3\",\n    \"claim\": \"Azure AI Search sitt REST-søkeendepunkt POST /indexes/{index-name}/docs/search kalles med api-version=2026-04-01.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-fuzzy\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#4\",\n    \"claim\": \"I Azure AI Search fuzzy search gir operatoren ~ standard redigeringsavstand 2, maksimalt 2 (Damerau-Levenshtein); ~1 gir én redigering og ~0 gir eksakt treff.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-fuzzy\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#5\",\n    \"claim\": \"En synonym map i Azure AI Search defineres med feltene name | format «solr» | synonyms (liste med synonymregler).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#6\",\n    \"claim\": \"Copilot Studio har en «Create search query»-node som utfører automatisk query rewriting med conversation history som no-code-løsning.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/authoring-create-search-query\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#7\",\n    \"claim\": \"Power Automate med AI Builder har handlingen «Create text with GPT» som kan brukes til å klassifisere intent før conditional branching.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#8\",\n    \"claim\": \"Embedding-modellen text-embedding-3-large støtter over 100 språk, slik at spørring og dokumenter på ulike språk matcher i samme vector space.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#9\",\n    \"claim\": \"Azure OpenAI lisensieres som Pay-as-you-go eller Provisioned Throughput Units (PTU).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#10\",\n    \"claim\": \"Azure AI Search Basic tier støtter synonym maps, mens semantic ranker krever Standard-tier eller høyere.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#11\",\n    \"claim\": \"Semantic Kernel er open source under MIT-lisens og har ingen lisenskostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#12\",\n    \"claim\": \"Copilot Studio lisensieres per bruker/sesjon og er inkludert i M365 Copilot eller tilgjengelig standalone.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#13\",\n    \"claim\": \"Azure AI Search bruker allerede Reciprocal Rank Fusion (RRF) internt for hybrid search, som kombinerer BM25 | vektorsøk.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/databricks/generative-ai/tutorials/ai-cookbook/quality-rag-chain\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#14\",\n    \"claim\": \"Azure Arc Edge RAG tilbyr søketype-parametere for query expansion | sub-query generation | hypothetical answer generation.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-arc/edge-rag/search-types#search-type-parameters\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#15\",\n    \"claim\": \"Simple query syntax er default parser for full-text-søk i Azure AI Search.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-simple-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#16\",\n    \"claim\": \"Simple query syntax i Azure AI Search har de tegn-baserte boolske operatorene + = AND (påkrevd term) | - = NOT (ekskluder term) | «|» = OR (alternativ term), og searchMode=all anbefales for presis NOT-atferd.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-simple-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#17\",\n    \"claim\": \"Simple query syntax i Azure AI Search støtter prefix queries med * (lingui* matcher linguistic, linguini) | phrase search med anførselstegn som krever eksakt ordrekkefølge.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-simple-syntax\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-query-understanding.md#18\",\n    \"claim\": \"Simple query syntax i Azure AI Search støtter ikke fuzzy search og ikke suffix/infix-wildcard; full Lucene-syntaks kreves for dette.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/query-simple-syntax\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-query-understanding.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/rag-security-rbac.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/rag-security-rbac.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/rag-security-rbac.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#1\",\n    \"claim\": \"I Azure AI Search er native ACL/RBAC i Preview, mens security filters er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#2\",\n    \"claim\": \"Azure AI Search støtter fire hovedtilnærminger til dokumentnivå-sikkerhet: security filters (GA) | POSIX-like ACL/RBAC scopes (preview) | Microsoft Purview sensitivity labels (preview) | SharePoint in Microsoft 365 ACLs (preview).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#3\",\n    \"claim\": \"Security filters i Azure AI Search gir subsekund responstid når search.in()-funksjonen brukes.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#4\",\n    \"claim\": \"Security-feltet for security filters i Azure AI Search skal ha felttype Collection(Edm.String) med attributtene filterable: true | retrievable: false.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#5\",\n    \"claim\": \"Native ACL/RBAC scope permissions i Azure AI Search per datakilde: ADLS Gen2 ACL = dokumentnivå (filer + directories) | Azure Blob RBAC = container-nivå | SharePoint ACL = dokumentnivå (filer + list items).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#6\",\n    \"claim\": \"Ved query-tid sendes Microsoft Entra-token til Azure AI Search i headeren x-ms-query-source-authorization.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-access-control-rbac-enforcement\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#7\",\n    \"claim\": \"Microsoft Purview sensitivity labels (preview) i Azure AI Search støtter datakildene Azure Blob | ADLS Gen2 | SharePoint | OneLake.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#8\",\n    \"claim\": \"Azure AI Search krever på index-nivå at applikasjonen har rollen Search Index Data Reader ved query-tid.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-security-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#9\",\n    \"claim\": \"Permission sources i Azure AI Search: userIds hentes fra oid-claim i x-ms-query-source-authorization-token | groupIds fra gruppemedlemskap via Microsoft Graph API | rbacScope fra storage container-permissions (rollen Storage Blob Data Reader).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-access-control-rbac-enforcement\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#10\",\n    \"claim\": \"Native ACL/RBAC-mønsteret i Azure AI Search krever preview REST API-versjon 2026-05-01-preview.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-query-access-control-rbac-enforcement\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#11\",\n    \"claim\": \"Native ACL/RBAC mot ADLS Gen2/Blob krever rolletildelingen Storage Blob Data Reader for Search-tjenestens managed identity.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-indexer-access-control-lists-and-role-based-access\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#12\",\n    \"claim\": \"Støttede gruppetyper for SharePoint ACL (preview): Microsoft Entra security groups (støttet) | Microsoft 365-grupper (støttet) | mail-enabled security groups (støttet) | SharePoint-grupper (ikke støttet i preview).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#13\",\n    \"claim\": \"Data residency for norsk offentlig sektor løses ved å kjøre Azure AI Search i regionene Norway East/Norway West.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#14\",\n    \"claim\": \"Azure OpenAI anbefales kjørt i Sweden Central med data residency commitment for norsk offentlig sektor.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#15\",\n    \"claim\": \"Azure AI Search prises tier-basert med tierne Basic | Standard | Storage Optimized.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#16\",\n    \"claim\": \"SharePoint ACL-tilnærmingen forutsetter SharePoint-lisensiering via Microsoft 365 E3/E5.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#17\",\n    \"claim\": \"Purview Information Protection er inkludert i Microsoft 365 E5 Compliance.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/retrieval-augmented-generation-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#18\",\n    \"claim\": \"ABAC (attributtbasert tilgangskontroll) er ikke støttet i Azure AI Search.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-indexer-access-control-lists-and-role-based-access\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#19\",\n    \"claim\": \"For ADLS Gen2 i Azure AI Search evalueres tilgang ved at RBAC sjekkes først og deretter ACL; tilgang gis om én av dem tillater det.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-indexer-access-control-lists-and-role-based-access\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/rag-security-rbac.md#20\",\n    \"claim\": \"Permissions for ADLS Gen2 ACL/RBAC synkroniseres ved: første fulle indexer-kjøring | nye dokumenter | manuell trigger via /resync (preview).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-indexer-access-control-lists-and-role-based-access\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/rag-security-rbac.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/self-reflective-rag.md",
    "claim_count": 16,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/self-reflective-rag.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/self-reflective-rag.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#1\",\n    \"claim\": \"Microsoft Foundry-evaluatorene er GA, mens agentic retrieval er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#2\",\n    \"claim\": \"Microsoft Foundry tilbyr innebygde evaluatorer for RAG-kvalitetsvurdering: groundedness | relevance | coherence — alle med 1-5-skala.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#3\",\n    \"claim\": \"Azure AI Search agentic retrieval er i preview og forbedrer retrieval-relevans med opptil 40 % gjennom LLM-assistert query planning.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#4\",\n    \"claim\": \"Microsoft Foundry har seks RAG-evaluatorer: Retrieval | Groundedness | Groundedness Pro | Relevance | Response Completeness | Document Retrieval.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#5\",\n    \"claim\": \"Retrieval-evaluatoren er en prosess-evaluator med 1-5 Likert-skala som måler query-context-relevans uten ground truth.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#6\",\n    \"claim\": \"Groundedness Pro er en system-evaluator med binært utfall som gjør streng consistency-sjekk via Azure AI Content Safety.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#7\",\n    \"claim\": \"Document Retrieval-evaluatoren er en prosess-evaluator som bruker NDCG og XDCG, og som krever ground truth-labels.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#8\",\n    \"claim\": \"Python-pakken azure.ai.evaluation eksponerer klassene RetrievalEvaluator og GroundednessEvaluator.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#9\",\n    \"claim\": \"RetrievalEvaluator og GroundednessEvaluator instansieres med parametrene model_config og threshold (threshold=3 i eksempelet).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#10\",\n    \"claim\": \"Microsoft Foundry-evaluatorene støtter reasoning-modeller (o-serien) via parameteren is_reasoning_model=True.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#11\",\n    \"claim\": \"Semantic Kernel TextSearchProviderOptions har feltene SearchTime (satt til RagBehavior.OnDemandFunctionCalling) | Top | PluginFunctionName.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#12\",\n    \"claim\": \"Semantic Kernel ChatCompletionAgent konfigureres med egenskapene Name | Instructions | Kernel | UseImmutableKernel.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-rag\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#13\",\n    \"claim\": \"Microsoft Foundry støtter sweep-metrikkene Max Relevance N (maks relevans-score i top-k chunks) | XDCG (resultatkvalitet innenfor top-k dokumenter) | Fidelity (hvor nøyaktig retrieval matcher ground truth).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/ai-ml/guide/rag/rag-llm-evaluation-phase\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#14\",\n    \"claim\": \"Integrasjonspunkter i Microsoft-stakken: Microsoft Foundry (innebygde evaluatorer Groundedness, Relevance, Retrieval) | Azure AI Search (retrieval backend + agentic retrieval i preview) | Semantic Kernel (OnDemandFunctionCalling) | Azure OpenAI (GPT-4o) | Application Insights | Azure AI Content Safety (Groundedness Pro).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#15\",\n    \"claim\": \"Microsoft Foundry-evaluatorene kjøres via Azure OpenAI i Sweden Central, slik at data holdes i EU/EØS.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/self-reflective-rag.md#16\",\n    \"claim\": \"Groundedness Pro avhenger av Azure AI Content Safety, og den regionale tilgjengeligheten for tjenesten må sjekkes.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/evaluation-evaluators/rag-evaluators\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/self-reflective-rag.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#1\",\n    \"claim\": \"Semantic Ranker i Azure AI Search er GA for kjernefunksjonalitet, mens query rewrite og prerelease-modeller er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#2\",\n    \"claim\": \"Semantic Ranker ekstraherer eksisterende tekst og produserer tre typer output: semantiske captions (relevante passasjer) | semantiske answers (direkte svar) | @search.rerankerScore på skala 0.0 til 4.0.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#3\",\n    \"claim\": \"Query rewrite og opt-in til prerelease-modeller er preview-funksjoner i Semantic Ranker.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#4\",\n    \"claim\": \"Query rewrite utvider et spørsmål til opptil 10 varianter.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#5\",\n    \"claim\": \"Semantic Ranker tar imot de topp 50 resultatene fra initial ranking (BM25 eller RRF) som input.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#6\",\n    \"claim\": \"Semantic Ranker assemblerer opptil 2000 tokens per dokument fra konfigurerte felt.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#7\",\n    \"claim\": \"Token-allokeringen i Semantic Ranker er title 128 tokens | keywords 128 tokens | content resten.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#8\",\n    \"claim\": \"Fra november 2024 er summary strings i Semantic Ranker opptil 2048 tokens (tidligere 256 tokens).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#9\",\n    \"claim\": \"Reranker score-skalaen tolkes slik: 4.0 = svært relevant, komplett svar | 3.0 = relevant men mangler noen detaljer | 2.0 = noe relevant, delvis svar | 1.0 = relatert men minimalt svar | 0.0 = irrelevant.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#10\",\n    \"claim\": \"prioritizedFields i en semantic configuration består av titleField | prioritizedContentFields | prioritizedKeywordsFields.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-configure\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#11\",\n    \"claim\": \"Felt brukt i semantic configuration må være searchable og retrievable, og må være strings av typen Edm.String eller Collection(Edm.String).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-configure\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#12\",\n    \"claim\": \"Title-feltet i en semantic configuration anbefales å være maks 25 ord.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-configure\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#13\",\n    \"claim\": \"Semantic Ranker har en kapasitet på omtrent 10 samtidige queries per replika.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#14\",\n    \"claim\": \"Bruk av semantic ranking i RAG-mønsteret krever S1-tier.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-enable-disable\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#15\",\n    \"claim\": \"Agentic Retrieval med LLM-assistert L3-ranking er en preview-funksjon (2025).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#16\",\n    \"claim\": \"Semantic ranker er multilingual og fungerer med norsk tekst uten ekstra konfigurasjon.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#17\",\n    \"claim\": \"Semantic ranker er tilgjengelig i regionene Norway East og Norway West, og all prosessering skjer i valgt region uten at data sendes ut av regionen.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#18\",\n    \"claim\": \"Prismodellen for semantic ranker har to planer: Gratis (1000 semantic ranker-requests per måned, tilgjengelig på alle tier inkludert Free) | Standard (pay-as-you-go per 1000 requests etter gratis kvote).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-enable-disable\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#19\",\n    \"claim\": \"Semantic ranker belastes når queryType=semantic og søkestrengen ikke er tom; den belastes ikke ved search=* (tom query) selv med queryType=semantic.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/semantic-ranker-reranking.md#20\",\n    \"claim\": \"Semantiske answers returneres kun når modellen er 70 % konfident.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-how-to-query-request\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/semantic-ranker-reranking.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/streaming-rag-responses.md",
    "claim_count": 17,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/streaming-rag-responses.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/streaming-rag-responses.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#1\",\n    \"claim\": \"Azure OpenAI Service støtter streaming via Server-Sent Events (SSE) i både Chat Completions API og den nyere Responses API.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#2\",\n    \"claim\": \"SSE-streaming fra Azure OpenAI kjennetegnes av: enveiskommunikasjon over HTTP (server → klient) | Content-Type `text/event-stream` | ett event per `data: <JSON>` fulgt av blank linje | avslutningssignal `data: [DONE]` | langvarig HTTP-tilkobling med `Connection: keep-alive` | `Transfer-Encoding: chunked` uten Content-Length.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference#chat-completions\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#3\",\n    \"claim\": \"Chat Completions API med `stream: true` returnerer `ChatCompletionChunk`-objekter der hver chunk inneholder delta med ny tekst i `choices[0].delta.content`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference#chat-completions\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#4\",\n    \"claim\": \"Responses API med `stream: true` er et nyere API som kombinerer Chat Completions og Assistants, og støtter: synkron og asynkron (background) streaming | event-basert format `response.output_text.delta` | gjenopptakelse av streaming fra `sequence_number` ved avbrudd | stateful conversations med `previous_response_id`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#5\",\n    \"claim\": \"Semantic Kernel Agent Framework har streaming-content-typene `StreamingChatMessageContent` | `StreamingTextContent` | `StreamingFileReferenceContent` | `StreamingAnnotationContent`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/frameworks/agent/agent-streaming\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#6\",\n    \"claim\": \"Azure OpenAI eksponerer et v1-API som brukes med OpenAI-klientbiblioteket via base_url `https://<ressurs>.openai.azure.com/openai/v1/`.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#7\",\n    \"claim\": \"Avbrutt streaming i Responses API gjenopptas med `GET /responses/{response_id}?stream=true&starting_after={cursor}`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#8\",\n    \"claim\": \"Chat Completions streaming i Azure OpenAI (parameteren `stream=true`) er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference#chat-completions\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#9\",\n    \"claim\": \"Responses API streaming i Azure OpenAI, med event-basert format, er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#10\",\n    \"claim\": \"Background streaming i Azure OpenAI er GA, for modellene o3 og o1-pro.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#11\",\n    \"claim\": \"Resume streaming i Azure OpenAI, via `sequence_number`, er GA.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#12\",\n    \"claim\": \"Function calling streaming i Azure OpenAI er GA, der argumenter streames token-for-token.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/reference#chat-completions\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#13\",\n    \"claim\": \"Azure API Management har en standard route timeout på 30 s, som må økes til 300 s+ for SSE-endepunkter.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/reference\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#14\",\n    \"claim\": \"Power Automate støtter ikke native SSE-streaming.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/reference\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#15\",\n    \"claim\": \"Copilot Studio støtter streaming via SSE for custom connectors (preview).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/reference\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#16\",\n    \"claim\": \"En code interpreter-session (Responses API) har levetid på 1 time, med idle timeout på 20 minutter.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/responses\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/streaming-rag-responses.md#17\",\n    \"claim\": \"Lisenskrav: Azure OpenAI krever Azure subscription + OpenAI resource | Semantic Kernel er open source (MIT) uten lisenskostnad | Application Gateway faktureres som Azure-kostnad per gateway-time | API Management har per call/måned-tier (Consumption for lavt volum).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/reference\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/streaming-rag-responses.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-engineering/references/rag-architecture/vector-indexing-techniques.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-engineering/references/rag-architecture/vector-indexing-techniques.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-engineering/references/rag-architecture/vector-indexing-techniques.md`)\n\n[\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#1\",\n    \"claim\": \"Hybrid search i Azure AI Search er GA, mens scalar quantization er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#2\",\n    \"claim\": \"Azure AI Search tilbyr vektoralgoritmene Hierarchical NSW (HNSW, approximate nearest neighbor) | Exhaustive KNN (exact nearest neighbor) | Flat indexing (linear scan).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#3\",\n    \"claim\": \"Hybrid search i Azure AI Search kombinerer tre lag: vector search med cosine similarity på embeddings | full-text search med BM25 keyword matching | semantic ranker (valgfri) basert på Microsofts flerspråklige deep learning-reranking adaptert fra Bing.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#4\",\n    \"claim\": \"Semantic ranker i Azure AI Search reranker de øverste 50 resultatene.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#5\",\n    \"claim\": \"Vekting av hybrid scores i Azure AI Search styres av parametrene alpha (balanse mellom vector 1.0 og BM25 0.0, standard 0.5, range 0.0–1.0) | k (antall vektorer fra vector search, standard 50, range 1–1000) | top (totale resultater etter merge, standard 10, range 1–1000).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#6\",\n    \"claim\": \"Azure AI Search støtter den norske analysatoren nb.microsoft for søkbare tekstfelt.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#7\",\n    \"claim\": \"HNSW-parametere i Azure AI Search er m (antall koblinger per node, standard 4) | efConstruction (byggekostnad per insert, standard 400) | efSearch (søkebredde ved query, standard 500).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#8\",\n    \"claim\": \"Azure AI Search har en rate limit på 3000 requests per sekund per replika.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#9\",\n    \"claim\": \"Embedding-modellen text-embedding-3-large brukes med 1536 dimensjoner i en Azure AI Search-index for standard RAG.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#10\",\n    \"claim\": \"Scalar quantization i Azure AI Search komprimerer vektorer fra 1536 float32 (6 KB) til 384 int8 (384 bytes).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#11\",\n    \"claim\": \"Scalar quantization er en preview-funksjon i Azure AI Search (2026).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#12\",\n    \"claim\": \"Copilot Studio konfigurerer Azure AI Search som kunnskapskilde for Generative answers via Security & Data → Knowledge sources → Add Azure AI Search, uten kontroll over HNSW-parametere (managed service).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#13\",\n    \"claim\": \"AI Builder i Power Automate støtter semantic search via Azure AI Search-connectoren med handlingen «Search documents (semantic)», som krever en index med contentVector-felt.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#14\",\n    \"claim\": \"Azure AI Search støtter regionen Norge Øst (Oslo) for data residency, der både innhold og vektorindeks lagres i Norge Øst.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#15\",\n    \"claim\": \"Embedding-modeller i Azure OpenAI kan konfigureres til regionen Sweden Central (EU).\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#16\",\n    \"claim\": \"Azure AI Search tilbys i tierne Basic (2 GB storage, 3 queries/sek, 1 replica) | S1 (25 GB, 15 queries/sek, 3 replicas) | S2 (100 GB, 60 queries/sek, 6 replicas) | S3 (200 GB, 60 queries/sek, 12 replicas).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#17\",\n    \"claim\": \"Semantic Ranker er inkludert i S1 og høyere tiers, med et gratis usage cap på 50 000 queries per måned.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/semantic-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#18\",\n    \"claim\": \"Integrasjon med Copilot Studio krever Copilot Studio-lisens med 20 000 meldinger per måned.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/search/hybrid-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#19\",\n    \"claim\": \"I hybrid search-indekser er HNSW standard ANN-algoritme med efSearch og maxConnections som tunbare parametere, mens eKNN (exhaustive K-Nearest Neighbors) gir fullstendig søk og aktiveres med \\\"exhaustive\\\": true i spørringen.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-create-index\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#20\",\n    \"claim\": \"Multi-vector fields i Azure AI Search er i preview og bruker felttypen Collection(Edm.ComplexType).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#21\",\n    \"claim\": \"Multi-vector fields i Azure AI Search støtter opptil 100 vektorer per dokument.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  },\n  {\n    \"id\": \"ms-ai-engineering/rag-architecture/vector-indexing-techniques.md#22\",\n    \"claim\": \"perDocumentVectorLimit i Azure AI Search kan settes til 1 (én vektor per dokument i resultater) | 0 (ubegrenset, alle matchende vektorer).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/vector-search-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-engineering/references/rag-architecture/vector-indexing-techniques.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md",
    "claim_count": 21,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#1\",\n    \"claim\": \"Application Insights Smart Detection består av tre hovedfunksjoner: Failure Anomalies Detection | Performance Anomalies Detection | General Degradations.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#2\",\n    \"claim\": \"Failure Anomalies Detection i Application Insights trenger 24 timer med data før aktivering.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#3\",\n    \"claim\": \"Performance Anomalies Detection analyserer både requests og dependencies, identifiserer mønstre i page load time og sammenligner mot historisk baseline.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#4\",\n    \"claim\": \"General Degradations i smart detection dekker: trace severity degradation | memory leaks | abnormal exception volume | security anti-patterns.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#5\",\n    \"claim\": \"Standard e-postvarsler fra smart detection sendes til rollene Monitoring Reader | Monitoring Contributor.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#6\",\n    \"claim\": \"Smart detection konfigureres via ARM-ressurstypen Microsoft.Insights/components/ProactiveDetectionConfigs med egenskapene enabled | sendEmailsToSubscriptionOwners | customEmails.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#7\",\n    \"claim\": \"Microsoft anbefaler å migrere smart detection til det alerts-baserte systemet, med egne alert rules per deteksjonsmodul og action groups for notifikasjoner.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#8\",\n    \"claim\": \"Migrering av smart detection til alerts kan gjøres på tre måter: Azure Portal (manuell migrering) | Azure CLI med REST API | ARM templates for batch-migrering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#9\",\n    \"claim\": \"KQL-funksjonen series_decompose_anomalies() har parametrene Threshold (default 1.5) | Seasonality (default -1, auto-detect) | Trend (default 'avg', alternativer 'avg' | 'linefit' | 'none') | Test_points | AD_method.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/kql-machine-learning-azure-monitor\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#10\",\n    \"claim\": \"KQL-pluginen diffpatterns() kalles via evaluate-operatoren og brukes til root cause-analyse ved å vise hvilke dimensjoner som varierer mest mellom normal og anomal periode.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/kql-machine-learning-azure-monitor\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#11\",\n    \"claim\": \"Smart detection sin learning period for behavioral baseline er minimum 24 timer (ofte 7-14 dager for robust baseline).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/proactive-diagnostics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#12\",\n    \"claim\": \"Metric alerts med dynamiske terskler settes ved å angi threshold = dynamic sammen med sensitivity og failingPeriods (numberOfEvaluationPeriods | minFailingPeriodsToAlert).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-dynamic-thresholds\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#13\",\n    \"claim\": \"Dynamic thresholds i Azure Monitor har sensitivitetsnivåene High | Medium | Low.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-dynamic-thresholds\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#14\",\n    \"claim\": \"Python-klienten for Azure AI Anomaly Detector (azure.ai.anomalydetector.AnomalyDetectorClient med AzureKeyCredential) tilbyr metodene detect_univariate_entire_series | detect_univariate_change_point, med UnivariateDetectionOptions (granularity TimeGranularity.HOURLY, sensitivity 90).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/anomaly-detector/overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#15\",\n    \"claim\": \"Action groups støtter aksjonstypene: Email/SMS/Push notifications | Webhook til incident management-system | Azure Function for automated remediation | Logic App for workflow orchestration | ITSM connector (f.eks. ServiceNow).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/anomaly-detector/overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#16\",\n    \"claim\": \"Microsoft Defender for AI dekker tre AI-spesifikke deteksjonskategorier: Jailbreak Attempt Detection | Model Inference Anomalies | Data Exfiltration Patterns.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/defender-for-cloud/ai-threat-protection\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#17\",\n    \"claim\": \"Defender for AI aktiveres med az security pricing create, plannavn AIServices og tier Standard.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/defender-for-cloud/ai-threat-protection\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#18\",\n    \"claim\": \"Azure AI Anomaly Detector-tjenesten er retired 1. oktober 2026.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/anomaly-detector/overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#19\",\n    \"claim\": \"Alternativer etter retirement av Azure AI Anomaly Detector: Azure ML model monitoring | Azure Monitor KQL-baserte funksjoner | Azure Stream Analytics | custom modeller i Azure ML.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-services/anomaly-detector/overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#20\",\n    \"claim\": \"Microsoft Fabric Real-Time Intelligence støtter anomaly detection via Python-plugin i Eventhouse med SimpleDetectAnomalies fra synapse.ml.services.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/anomaly-detection\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/anomaly-detection-ai-systems.md#21\",\n    \"claim\": \"Application Insights er inkludert i Basic-tier, gratis opp til 5 GB per måned.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/ai-services/anomaly-detector/overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md",
    "claim_count": 26,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#1\",\n    \"claim\": \"Microsoft Purview Compliance Manager har AI-spesifikke assessment-templates: AI Act | ISO/IEC 23053:2022 | NIST AI RMF.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/compliance-manager-assessments#assessments-for-ai-regulations\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#2\",\n    \"claim\": \"Compliance Manager grupperer controls i Microsoft-managed | customer-managed | shared controls, også for AI.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/compliance-manager\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#3\",\n    \"claim\": \"Compliance Manager har 360+ ferdigbygde regulatory templates, inkludert AI-spesifikke: EU AI Act | GDPR AI-tillegg | CCPA | HIPAA.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/compliance-manager\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#4\",\n    \"claim\": \"Azure Policy har enforcement-modusene Audit (logg non-compliance, men tillat deployment) | Deny (blokkér non-compliant ressurser) | Append/Modify (legg automatisk til manglende konfigurasjon) | DeployIfNotExists (deploy påkrevde ressurser automatisk).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/platform/governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#5\",\n    \"claim\": \"Microsoft Security Dashboard for AI er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#6\",\n    \"claim\": \"Security Dashboard for AI har seksjonene AI Agent Inventory | Threat Detection | Data Security | Access Control | Compliance Status.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#7\",\n    \"claim\": \"AI Agent Inventory i Security Dashboard for AI dekker agenter registrert i Microsoft Agent 365, samt modeller, MCP-servere og tredjeparts AI-apper (Gemini, ChatGPT), og skiller managed vs. unmanaged (shadow AI).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#8\",\n    \"claim\": \"Security Dashboard for AI støtter produktene Microsoft Entra (agent identity platform / Agent Registry, conditional access for AI-apper) | Microsoft Defender (AI-modell-/MCP-/app-discovery, posture management, AI threat protection med sanntidsblokkering for støttede agenter) | Microsoft Purview (dataklassifisering, DLP for AI-prompts, insider risk detection / DSPM for AI) | Security Copilot (prompt-basert exploration av AI-risikoer).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#9\",\n    \"claim\": \"Minimumsrollen for full innsikt i Security Dashboard for AI er Security Reader (Microsoft Entra).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/security-dashboard-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#10\",\n    \"claim\": \"Microsoft Foundry AI Reports genererer rapporter i PDF- og SPDX-format med model cards, evalueringsmetrikker og content safety-konfigurasjon.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/platform/governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#11\",\n    \"claim\": \"Azure API Management AI gateway dekker OpenAI Chat Completions/Responses | Anthropic Messages API | Google Vertex AI | modeller i Microsoft Foundry | Amazon Bedrock | remote MCP-servere | A2A agent-APIer.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#12\",\n    \"claim\": \"Anthropic Messages API støttes i Azure API Management v2-tiers.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#13\",\n    \"claim\": \"Unified model API i Azure API Management, som eksponerer flere backends gjennom ett OpenAI-kompatibelt endepunkt med felles governance-policyer, er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#14\",\n    \"claim\": \"AI gateway i Azure API Management kan integreres direkte i Microsoft Foundry (preview).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#15\",\n    \"claim\": \"Azure API Management AI gateway har governance-policyene llm-emit-token-metric | llm-token-limit | llm-semantic-cache-store | llm-semantic-cache-lookup | llm-content-safety.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#16\",\n    \"claim\": \"llm-emit-token-metric støtter maks 5 custom dimensions inbound.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#17\",\n    \"claim\": \"Semantisk caching i APIM AI gateway (llm-semantic-cache-store / llm-semantic-cache-lookup) kjører mot Azure Managed Redis med RediSearch, og score-threshold er et distanse-mål der lavere verdi er strengere (Microsoft-eksempel 0,15).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/api-management/genai-gateway-capabilities#observability-and-governance\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#18\",\n    \"claim\": \"Microsoft Purview har AI-spesifikke løsninger: DSPM for AI | Audit logs for AI-interaksjoner | Communication Compliance | eDiscovery | Insider Risk Management | Retention policies.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#19\",\n    \"claim\": \"Purview Insider Risk Management har policyen «Risky AI usage» som fanger prompt injection-forsøk og tilgang til beskyttet materiale.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#20\",\n    \"claim\": \"Purview-kategorien «Copilot experiences and agents» omfatter Microsoft 365 Copilot (+ Cowork) | Security Copilot | Copilot in Fabric | Copilot Studio | Microsoft Facilitator | Teams Channel Agent.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#21\",\n    \"claim\": \"Purview-kategorien «Enterprise AI apps» omfatter Microsoft Foundry | Entra-registrerte AI-apper | Anthropic Claude (Enterprise) | ChatGPT Enterprise.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#22\",\n    \"claim\": \"Purview-kategorien «Other AI apps», oppdaget via browser og Defender for Cloud Apps, omfatter ChatGPT | Google Gemini | Microsoft Copilot (consumer) | DeepSeek.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#23\",\n    \"claim\": \"Purview har collection policies: «DSPM for AI - Detect sensitive info shared with AI via network» | «DSPM for AI - Capture interactions for enterprise AI apps» | «DSPM for AI - Capture interactions for Copilot experiences».\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/ai-microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#24\",\n    \"claim\": \"I Microsoft Foundry velges region ved opprettelse av prosjektet, og prosjektet kan ikke flyttes til en annen region senere.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/govern\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#25\",\n    \"claim\": \"Azure OpenAI med EU Data Boundary garanterer at prompts og responses ikke forlater EU.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/security/security-for-ai/govern\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/compliance-monitoring-ai-governance.md#26\",\n    \"claim\": \"Compliance Manager-lisensiering: Microsoft 365 E3 gir kun basic assessments (Microsoft baseline) uten AI-spesifikke templates | Microsoft 365 E5 gir 360+ regulatory templates, custom templates og automatiserte assessments inkl. AI Act, ISO/IEC 23053 og NIST AI RMF | Purview Compliance standalone gir full Compliance Manager + DLP + eDiscovery med DSPM for AI, AI audit logs og retention for AI-apper.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/purview/compliance-manager\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/compliance-monitoring-ai-governance.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/cost-monitoring-cost-attribution.md",
    "claim_count": 20,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/cost-monitoring-cost-attribution.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/cost-monitoring-cost-attribution.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#1\",\n    \"claim\": \"Tredjeparts-modeller fra Azure Marketplace (f.eks. Cohere | Meta) faktureres separat via Azure Marketplace-meters, med én SaaS-meter per modelltilbud for input- og output-tokens.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#2\",\n    \"claim\": \"Azure OpenAI har følgende meter-typer: Tokens (input/output, per 1000 tokens for chat/completion) | Fine-tuning training (per token i treningsfilen) | Fine-tuning hosting (per time per distribuert modell) | Fine-tuning inference (per 1000 tokens ved kall til modell) | Image generation (fast pris per bilde).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#3\",\n    \"claim\": \"Azure OpenAI-deployments som har vært inaktive i 15 dager slettes automatisk, mens den underliggende modellen bevares.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#4\",\n    \"claim\": \"I Azure Cost Analysis ligger Azure OpenAI under tjenestekategorien Cognitive Services når man filtrerer på Service Tier.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#5\",\n    \"claim\": \"I Cost by resource-visningen vises enkelte leverandørers modeller under Global resources, med meternavn på formatet model-name-GUID, der GUID er identifikator for den aktuelle Foundry-ressursen.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#6\",\n    \"claim\": \"Azure Marketplace-modeller bruker meternavnene paygo-inference-input-tokens | paygo-inference-output-tokens | paygo-finetuned-model-inference-hosting | paygo-finetuned-model-inference-input-tokens | paygo-finetuned-model-inference-output-tokens.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#7\",\n    \"claim\": \"Tredjeparts-modeller (Azure Marketplace) kan ikke betales med Azure Prepayment.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#8\",\n    \"claim\": \"Kostnadsestimatene i Microsoft Foundry-portalen reflekterer ikke rabatter eller avtalt prising, og dekker kun standard deployment-kostnader — ikke provisioned throughput (PTU).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#9\",\n    \"claim\": \"Estimated cost-flisen under Operate → Overview i Foundry-portalen viser estimert kostnad for alle agenter i valgte prosjekter og datointervall, men inkluderer ikke prompt-agenter eller ikke-Foundry-agenter.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#10\",\n    \"claim\": \"Assets → Agents-fanen i Foundry-portalen har kolonnen Estimated costs, som viser månedsestimat per agent basert på konfigurasjon og bruksmønster.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#11\",\n    \"claim\": \"Build → Models → Monitor-fanen i Foundry-portalen viser total kostnad og et estimert-kostnad-diagram for valgt periode; token-/request-diagrammene kan midlertidig avvike fra Estimated cost på grunn av ingestion-timing.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#12\",\n    \"claim\": \"Kostnadsfordeling (chargeback) på prosjektnivå i Microsoft Foundry er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#13\",\n    \"claim\": \"Hvert Foundry-prosjekt tagges automatisk med en project-tag på underliggende forbruk, uten at manuell tagging kreves.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#14\",\n    \"claim\": \"Prosjektnivå kostnadsfordeling i Foundry støttes foreløpig kun for Models sold by Azure (Azure Direct, inkludert Azure OpenAI), og ikke for modeller levert via Azure Marketplace.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#15\",\n    \"claim\": \"Power BI Premium kreves for å aktivere automatisk oppdatering (automatic refresh) og deling med stakeholders av eksportert kostnadsdata.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#16\",\n    \"claim\": \"Azure OpenAI-deployments eksponerer metrikken TokensGenerated, som kan hentes med Get-AzMetric.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#17\",\n    \"claim\": \"Cost data kan lagres i de norske Azure-regionene Norway East og Norway West.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#18\",\n    \"claim\": \"Azure OpenAI tilbyr to prismodeller: PTU (Provisioned Throughput Units — fast månedlig kostnad med garantert throughput målt i TPM/RPM) | Pay-as-you-go (betaling kun for faktisk forbruk).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/cloud-adoption-framework/scenarios/ai/manage#manage-ai-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#19\",\n    \"claim\": \"Azure API Management tilbys i tierne Consumption | Basic | Standard | Premium.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/cost-monitoring-cost-attribution.md#20\",\n    \"claim\": \"Maksimal throughput per Azure API Management-tier: Consumption 1000 req/sek | Basic 1000 req/sek | Standard 2500 req/sek | Premium 4000 req/sek.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/manage-costs\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/cost-monitoring-cost-attribution.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md",
    "claim_count": 17,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#1\",\n    \"claim\": \"Microsoft Foundry leverer en ferdig, innebygd «Application Analytics»-workbook for AI-monitorering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#2\",\n    \"claim\": \"Microsoft Foundrys out-of-box workbook sporer: Generative AI metrics (total conversations | latency | exceptions) | Tool usage (hvilke extensions og tools brukes mest) | Topic analytics (hvilke conversation topics dominerer) | Operational health (success rates | error patterns | response times).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#3\",\n    \"claim\": \"Workbook-galleriet under Application Insights → Monitoring → Workbooks inneholder en workbook med navnet «Copilot Studio Dashboard».\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#4\",\n    \"claim\": \"ARM-apiVersion for workbook-instanser (type Microsoft.Insights/workbooks) er 2018-06-17-preview, mens workbook-maler (type microsoft.insights/workbooktemplates) bruker 2019-10-17-preview.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-samples\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#5\",\n    \"claim\": \"Bicep støttes nå offisielt som alternativ til ARM JSON for deploy av Azure Workbooks.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-automate\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#6\",\n    \"claim\": \"Microsoft tilbyr en ferdig Grafana-dashboard med ID 24039 for Microsoft Foundry-ressurser, og dashboard-tittelen er nå «Microsoft Foundry dashboard» i offisiell dokumentasjon.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#7\",\n    \"claim\": \"Grafanas managed identity må tildeles rollen Monitoring Reader for å lese Azure Monitor-data til Foundry-dashboardet.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#8\",\n    \"claim\": \"Azure Portal tilbyr direktelenken Monitor → «Dashboards with Grafana» → AI Foundry, og denne funksjonen er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#9\",\n    \"claim\": \"Metric namespace for Azure OpenAI/Foundry-metrikker er Microsoft.CognitiveServices/accounts.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#10\",\n    \"claim\": \"Sentrale metrikker for AI-dashboards er: AzureOpenAIRequests | TokenTransaction | ProcessedPromptTokens | GeneratedTokens | AzureOpenAITTLTInMS (inference latency, time to last byte).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#11\",\n    \"claim\": \"Alle disse metrikkene kan splittes (grupperes) på dimensjonen ModelDeploymentName.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#12\",\n    \"claim\": \"Power BI Desktop har en Azure Monitor Logs-konnektor tilgjengelig via Get Data → Azure → Azure Monitor Logs.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#13\",\n    \"claim\": \"Scheduled refresh, deling, dataflows og incremental refresh krever Power BI Pro eller Premium; gratis-tieren dekker kun lokal rapport-/dashboardbygging.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#14\",\n    \"claim\": \"Tilgangsstyring for Azure Workbooks: Monitoring Contributor inkluderer workbooks/write og kan redigere og lagre delte workbooks | Monitoring Reader kan se, men ikke endre | egendefinerte roller krever microsoft.insights/workbooks/write for redigering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/visualize/workbooks-automate\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#15\",\n    \"claim\": \"Grafana-rollene er: Grafana Admin (full tilgang) | Grafana Editor (kan redigere dashboards) | Grafana Viewer (read-only).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/managed-grafana/azure-ai-foundry-dashboard\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#16\",\n    \"claim\": \"Power BI workspace-roller er: Admin | Member | Contributor | Viewer.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/custom-dashboards-ai-operations.md#17\",\n    \"claim\": \"Eksport fra Log Analytics til Power BI skjer enten som M-query (.txt-fil til Power BI Desktop) eller som nytt Dataset i Power BI-tjenesten.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/log-powerbi\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/custom-dashboards-ai-operations.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/data-residency-audit-monitoring.md",
    "claim_count": 25,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/data-residency-audit-monitoring.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/data-residency-audit-monitoring.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#1\",\n    \"claim\": \"Microsoft Purview Audit Standard omfatter audit records for service configuration, audited activities og audit log query permissions, mens Premium i tillegg gir high-value crucial events med lengre retention.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#2\",\n    \"claim\": \"Microsoft Purview Audit Standard har 90 dagers retention som default, mens Premium gir konfigurerbare retention policies.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#3\",\n    \"claim\": \"Både Microsoft Purview Audit Standard og Premium lagrer audit-data i Local Region Geography.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#4\",\n    \"claim\": \"Microsoft Purview Audit Standard gir API-tilgang via Office 365 Management Activity API, mens Premium gir higher bandwidth access.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#5\",\n    \"claim\": \"Microsoft Purview Audit Standard dekker ISO 27001 | SOC 1/2/3, mens Premium i tillegg dekker FedRAMP | GDPR-optimalisert.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#6\",\n    \"claim\": \"Regionale Azure-tjenester oppfyller EU Data Boundary ved deploy i EU/EFTA-regioner som Norway East og West Europe.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#7\",\n    \"claim\": \"Azure Resource Manager EU Data Boundary kan kun konfigureres for nye tenants uten eksisterende subscriptions eller ressurser, og eksisterende ressurser kan ikke flyttes inn i eller ut av en tenant med EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#8\",\n    \"claim\": \"Application Change Analysis lagres og prosesseres globalt og er ikke tilgjengelig innenfor EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-configure-azure-nonregional-services\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#9\",\n    \"claim\": \"For EU-miljøer i EU Data Boundary brukes Azure OpenAI-endepunkt innenfor samme boundary, mens Bing Search-data prosesseres i USA selv ved EU-residency.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/power-platform/admin/geographical-availability-copilot\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#10\",\n    \"claim\": \"Checkboxen «Allow flex routing during periods of peak load» er tilgjengelig for EU-miljøer i Power Platform.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/power-platform/admin/geographical-availability-copilot\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#11\",\n    \"claim\": \"Microsoft 365-tenants med sign-up i EU/EFTA får automatisk EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#12\",\n    \"claim\": \"Multi-Geo Capabilities i Microsoft 365 ekskluderer tenanten fra EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-learn\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#13\",\n    \"claim\": \"Microsofts pseudonymiseringsteknikker for personopplysninger i system-generated logs er: Encryption av identifiers | Masking av sensitive fields | Tokenization | Data blurring.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/compliance/assurance/assurance-audit-logging\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#14\",\n    \"claim\": \"Audit-data har 90 dagers retention i Cosmos og 180 dagers retention i Kusto.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/compliance/assurance/assurance-audit-logging\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#15\",\n    \"claim\": \"Microsoft Purview DLP med audit krever E5/G5-lisensiering eller Purview standalone.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#16\",\n    \"claim\": \"Azure OpenAI Service kan deployes i Norway East eller West Europe.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/communication-services/concepts/european-union-data-boundary\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#17\",\n    \"claim\": \"Agent-samtaler i Copilot Studio lagres i regionen til Power Platform-miljøet.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/geo-data-residency-security\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#18\",\n    \"claim\": \"Purview Audit Premium logger alle agent-interaksjoner i Copilot Studio.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/geo-data-residency-security\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#19\",\n    \"claim\": \"Norway East og Norway West er parede Azure-regioner, og begge ligger innenfor EU Data Boundary.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/communication-services/concepts/european-union-data-boundary\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#20\",\n    \"claim\": \"Microsoft Purview Audit Standard er inkludert i lisensene E3/E5 og G3/G5.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#21\",\n    \"claim\": \"Microsoft Purview Audit Premium krever E5/G5 eller standalone add-on, og gir konfigurerbar retention opptil 10 år samt high-value events.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-365/enterprise/m365-dr-commitments?view=o365-worldwide#microsoft-purview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#22\",\n    \"claim\": \"Azure Monitor Log Analytics har retention-nivåene Interactive 0-31 dager (default) | Basic 31 dager til 2 år | Archive 2+ år.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/communication-services/concepts/european-union-data-boundary\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#23\",\n    \"claim\": \"Azure non-regionale tjenester som må konfigureres for EU Data Boundary omfatter: Bot Service | Communication Services | Azure Stack Edge/Hub | Azure Local (tidl. Azure Stack HCI) | Azure Migrate | Azure VM Image Builder | Cloud Shell | Microsoft Fabric | Power BI Embedded | Translator (EU-endepunkt) | Microsoft Entra ID.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/privacy/eudb/eu-data-boundary-configure-azure-nonregional-services\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#24\",\n    \"claim\": \"Compliance-tilbudene for Azure, Dynamics 365, Microsoft 365 og Power Platform omfatter ISO 27001 | ISO 27017 | ISO 27018 | SOC 1/2/3 | FedRAMP.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/compliance/offerings/\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/data-residency-audit-monitoring.md#25\",\n    \"claim\": \"EU Data Boundary-støtte i Azure Communication Services dekker voice | video | chat | SMS | email.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/communication-services/concepts/european-union-data-boundary\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/data-residency-audit-monitoring.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md",
    "claim_count": 18,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#1\",\n    \"claim\": \"Application Insights SDK støtter både W3C Trace Context (traceparent | tracestate) og eldre Request-Id-protokoller for cross-service context-propagering.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/distributed-trace-data\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#2\",\n    \"claim\": \"OpenTelemetrys semantiske konvensjoner for generativ AI definerer standard AI-span: gen_ai.model.completion | gen_ai.tool.execution | gen_ai.agent.invoke | gen_ai.agent_planning | gen_ai.agent_to_agent_interaction.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#3\",\n    \"claim\": \"Standard gen_ai-attributter er: gen_ai.system | gen_ai.request.model | gen_ai.usage.prompt_tokens | gen_ai.usage.completion_tokens | gen_ai.response.finish_reason.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#4\",\n    \"claim\": \"Microsoft har utviklet nye semantiske konvensjoner for observability i multi-agent-systemer, i samarbeid med Cisco Outshift.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#5\",\n    \"claim\": \"Span-typene i de nye multi-agent-konvensjonene er: execute_task | agent_to_agent_interaction | agent.state.management | agent_planning | agent_orchestration.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#6\",\n    \"claim\": \"Azure Functions host.json refererer extension bundle Microsoft.Azure.Functions.ExtensionBundle med versjonsområdet [4.*, 5.0.0).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#7\",\n    \"claim\": \"OpenTelemetry i Azure Functions aktiveres på app-nivå både i host.json (telemetryMode satt til OpenTelemetry) og i koden; oppgis både APPLICATIONINSIGHTS_CONNECTION_STRING og en OTLP-eksportør (OTEL_EXPORTER_OTLP_ENDPOINT / OTEL_EXPORTER_OTLP_HEADERS), sendes telemetri til begge endepunktene.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#8\",\n    \"claim\": \"I OpenTelemetry-modus for Azure Functions støtter Azure-portalen ikke log streaming; Recent function invocation-traces vises kun hvis telemetri sendes til Azure Monitor, og logging.applicationInsights-konfigurasjon i host.json gjelder ikke.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#9\",\n    \"claim\": \"I OTel-modus for Azure Functions er parent-based sampling standard; triggere som HTTP, Service Bus og Event Hubs avhenger av context propagation, og OperationId hentes direkte fra traceparent slik at gjenbruk av samme traceparent gir samme OperationId.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#10\",\n    \"claim\": \"Filtre i host.json gjelder kun host-prosessens logger, mens worker-prosessens logger filtreres via språkspesifikke OTel-innstillinger; Go-workeren har egen opt-in OTel-middleware (middleware/otelfunc).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-functions/opentelemetry-howto\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#11\",\n    \"claim\": \"AzureChatOpenAI konfigureres med api_version 2024-08-01-preview i LangChain-integrasjonen mot Azure OpenAI.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#12\",\n    \"claim\": \"Semantic Kernel emitterer automatisk metrikkene semantic_kernel.function.invocation.duration (Histogram) | semantic_kernel.function.streaming.duration (Histogram) | semantic_kernel.function.invocation.token_usage.prompt | semantic_kernel.function.invocation.token_usage.completion.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#13\",\n    \"claim\": \"Semantic Kernel genererer en Activity per kernel function-utførelse og en egen Activity per AI-modellkall, med activity source Microsoft.SemanticKernel.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/semantic-kernel/concepts/enterprise-readiness/observability/\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#14\",\n    \"claim\": \"Application Insights tilbyr funksjonene for trace-visualisering og -analyse: Application Map | Transaction Search | End-to-End Transaction Details | Performance View | Failures Blade.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/distributed-trace-data\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#15\",\n    \"claim\": \"Aspire Dashboard kjøres lokalt som container-image mcr.microsoft.com/dotnet/aspire-dashboard:latest, med dashboard eksponert på port 18888 og OTLP-mottak mappet fra port 4317 til containerens port 18889.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#16\",\n    \"claim\": \"Verdien i OTEL_SERVICE_NAME mappes til cloud_RoleName i Application Insights.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#17\",\n    \"claim\": \"Azure Monitor OpenTelemetry-distroen sampler ikke som standard; den støtter fixed-rate og rate-limited samplere som må konfigureres eksplisitt, og trace-basert sampling for logger er default-på først når sampling er aktivert — adaptive sampling gjelder kun den klassiske Application Insights SDK-en.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/distributed-tracing-ai-pipelines.md#18\",\n    \"claim\": \"Thread Logs i Agents Playground viser: Thread details | Run information | Ordered run steps | Tool calls | Linked evaluations.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/ai-studio/how-to/develop/trace-local-sdk\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/distributed-tracing-ai-pipelines.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/endpoint-health-and-capacity-planning.md",
    "claim_count": 25,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/endpoint-health-and-capacity-planning.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/endpoint-health-and-capacity-planning.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#1\",\n    \"claim\": \"Azure OpenAI har ingen latens-SLA for Standard-tilbudet, mens Provisioned Throughput Units (PTU) gir forutsigbar ytelse for produksjonskritiske workloads.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#2\",\n    \"claim\": \"Azure Monitor-plattformmetrikker for Azure OpenAI omfatter AzureOpenAIRequests | AzureOpenAIAvailabilityRate | GeneratedTokens | ActiveTokens | AzureOpenAIProvisionedManagedUtilizationV2 | AzureOpenAITimeToResponse, alle med tidsromdetaljering PT1M (1 minutt).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-cognitiveservices-accounts-metrics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#3\",\n    \"claim\": \"Av Azure OpenAI-plattformmetrikkene støtter AzureOpenAIRequests | GeneratedTokens | ActiveTokens | AzureOpenAITimeToResponse eksport via diagnostic settings (DS Export), mens AzureOpenAIAvailabilityRate | AzureOpenAIProvisionedManagedUtilizationV2 ikke støtter det.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/reference/supported-metrics/microsoft-cognitiveservices-accounts-metrics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#4\",\n    \"claim\": \"Requests Per Minute (RPM) for en Azure OpenAI-deployment beregnes automatisk fra TPM, og forholdet varierer per modell.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/quota\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#5\",\n    \"claim\": \"Kvote for Azure OpenAI tildeles via Quota Tiers — Free Tier (Tier 0) | Tier 1 | Tier 2 | Tier 3 | Tier 4 | Tier 5 | Tier 6 — ikke via «1 Unit Capacity».\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#6\",\n    \"claim\": \"For Tier 1 og GlobalStandard er RPM/TPM-forholdet modellspesifikt: eldre chat-modeller 6 RPM / 1,000 TPM | o1 og o1-preview 1 RPM / 6,000 TPM | o3 1 RPM / 1,000 TPM | o3-mini og o1-mini 1 RPM / 10,000 TPM.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#7\",\n    \"claim\": \"Deployment-TPM kan ikke overskride subscription-kvoten for den aktuelle modellen i den aktuelle regionen.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/quota\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#8\",\n    \"claim\": \"Azure OpenAI tilbyr to innebygde dashboards: Azure Portal Dashboard (Overview-pane) | AI Foundry Metrics Dashboard (nås via «Go to AI Foundry portal» → Tools → Metrics dashboard).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#9\",\n    \"claim\": \"Azure Portal Dashboard for Azure OpenAI viser HTTP Requests (total, statuskoder, feilrate) | Tokens-Based Usage (prompt, completion, total tokens) | PTU Utilization (kun for PTU-deployments) | Fine-tuning metrics.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#10\",\n    \"claim\": \"Dynamic Quota for Azure OpenAI er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/dynamic-quota\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#11\",\n    \"claim\": \"Dynamic Quota tillater opportunistisk burst utover base-TPM når regional kapasitet er tilgjengelig, uten ekstra kostnad, men er ikke garantert og kan ikke redusere TPM under base-grensen.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/dynamic-quota\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#12\",\n    \"claim\": \"Deployments under Microsoft.CognitiveServices/accounts opprettes/oppdateres via Azure Resource Manager REST API med api-version=2023-05-01.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#13\",\n    \"claim\": \"Modellen gpt-4o refereres med modellversjon 2024-11-20 i deployment-definisjonen.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#14\",\n    \"claim\": \"PTU (Provisioned Throughput) krever commitment på 1 måned eller 1 år.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#15\",\n    \"claim\": \"Microsoft anbefaler nå (2026) sub-kommandoen `az monitor metrics alert condition create` for å opprette betingelser og dimensjoner til metric alerts i Azure CLI.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/azure-cli-metrics-alert-sample\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#16\",\n    \"claim\": \"Python-klienten AzureOpenAI konfigureres med api_version 2024-10-21.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#17\",\n    \"claim\": \"Både Azure OpenAI-ressursen og Log Analytics workspace kan plasseres i regionen Norway East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#18\",\n    \"claim\": \"Databehandling for Azure OpenAI skjer i EU, selv om kontrollplanet er globalt.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#19\",\n    \"claim\": \"Azure Monitor er inkludert i Azure-abonnementet uten separat lisens, og oppsett av Log Analytics workspace krever Owner- eller Contributor-rolle.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/openai/quotas-limits\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#20\",\n    \"claim\": \"Roller som gir innsyn i kvote: Cognitive Services Usages Reader (minimal rolle for å se kvote på tvers av subscription, anbefalt) | Reader (gir også kvote-innsyn, men bredere tilgang enn nødvendig); rollen må settes på subscription-nivå, ikke resource-nivå.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/quota\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#21\",\n    \"claim\": \"Azure AI Search har to prismodeller: Dedicated (search units, SU) | Serverless (capacity units per time + per GB, uten SLA).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#22\",\n    \"claim\": \"Serverless-prismodellen i Azure AI Search er i Preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#23\",\n    \"claim\": \"Azure AI Search Serverless er kun tilgjengelig i West Central US | Switzerland North | Japan East.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#24\",\n    \"claim\": \"Basic-tier i Azure AI Search (tjenester opprettet etter 2024-04-03) støtter inntil 3 partisjoner × 3 replikaer (9 SU).\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-limits-quotas-capacity\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/endpoint-health-and-capacity-planning.md#25\",\n    \"claim\": \"Verktøy for å overvåke modellkvalitet og endepunkthelse i Azure Databricks Model Serving: ephemeral service logs | OpenTelemetry for custom endpoints (Unity Catalog Delta tables, langtidsretensjon) | build logs (30 dagers retensjon) | endpoint health metrics (siste 14 dager) | AI Gateway-enabled inference tables (automatisk logging av requests/responses til Delta tables).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/databricks/machine-learning/model-serving/monitor-diagnose-endpoints\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/endpoint-health-and-capacity-planning.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md",
    "claim_count": 16,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#1\",\n    \"claim\": \"Viktige Azure Monitor-tabeller for AI-monitoring er AzureDiagnostics (resource logs fra Azure-tjenester) | AzureMetrics (platform metrics) | CDBCassandraRequests (Cosmos DB) | ABSBotRequests (Azure Bot Service) | AmlComputeJobEvent (Azure Machine Learning job events) | AmlComputeClusterEvent (Azure ML cluster events) | AmlOnlineEndpointTrafficLog (Azure ML online endpoint traffic).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#2\",\n    \"claim\": \"Azure OpenAI-ressurslogger i AzureDiagnostics har loggkategorien RequestResponse (Category == \\\"RequestResponse\\\") for request/response-logging.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#3\",\n    \"claim\": \"Azure AI Search-diagnostikk i AzureDiagnostics logger operasjonen Query.Search med feltene Query_s | IndexName_s | Documents_d | DurationMs.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-monitor-enable-logging\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#4\",\n    \"claim\": \"Tabellen AmlComputeJobEvent inneholder feltene ClusterId | EventType | ExecutionState | ToolType, og EventType-verdien JobFailed markerer feilede Azure ML-jobber.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#5\",\n    \"claim\": \"Tabellen AmlComputeClusterEvent inneholder feltene TargetNodeCount | ClusterName | VmSize for node-allokering i Azure ML compute-cluster.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#6\",\n    \"claim\": \"Tabellen AmlOnlineEndpointTrafficLog inneholder feltene EndpointName | DeploymentName | ResponseCode | ResponseCodeReason for Azure ML online endpoints.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#7\",\n    \"claim\": \"Microsoft dokumenterer tre standard alert rules for Azure Machine Learning: Model Deploy Failed (Total > 0) | Quota Utilization Percentage (Average > 90 %) | Unusable Nodes (Total > 0).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#8\",\n    \"claim\": \"Azure OpenAI signaliserer throttling (rate limiting) med HTTP-statuskode 429 i feltet ResultSignature i AzureDiagnostics.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#9\",\n    \"claim\": \"Azure OpenAI-metrikker i AzureMetrics omfatter TokenTransaction | TotalTokens | PromptTokens | CompletionTokens.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry-classic/openai/how-to/monitor-openai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#10\",\n    \"claim\": \"Log Analytics har et Query Details-panel med tre faner: Overview (KPI-er som CPU, tidsomfang, alder på data, antall workspaces, antall regioner, parallellisme og Memory peak) | Raw statistics | Errors.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#11\",\n    \"claim\": \"Execution time i Log Analytics er delt i tre komponenter: Engine Execution Time (tid i underliggende data-engine, Azure Data Explorer) | Service Execution Time (intern Azure Monitor-prosessering og orkestrering) | Service Queue Time (ventetid i kø pga. concurrency-grenser).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#12\",\n    \"claim\": \"Memory peak er maksimal RAM observert under kjøring i Log Analytics, og høy memory peak kan trigge feilene E_RUNAWAY_QUERY eller E_LOW_MEMORY_CONDITION.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#13\",\n    \"claim\": \"Grensen er maks 5 kolonne-ekstraksjoner per parse-setning i KQL; over 5 øker prosesseringstiden markant.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#14\",\n    \"claim\": \"I transformasjoner er grensen 10 ekstraksjoner per parse-setning.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/logs/query-optimization\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#15\",\n    \"claim\": \"Azure AI Search bruker api-version 2025-09-01 i query-strenger (?api-version=2025-09-01&search=*).\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/search/search-monitor-enable-logging\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/log-analytics-kql-ai-queries.md#16\",\n    \"claim\": \"AmlOnlineEndpointTrafficLog er en ny tabell for overvåkning av inference-endepunkter i Azure Machine Learning.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/monitor-azure-machine-learning\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/log-analytics-kql-ai-queries.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/model-performance-drift-detection.md",
    "claim_count": 23,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/model-performance-drift-detection.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/model-performance-drift-detection.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#1\",\n    \"claim\": \"Model performance monitoring og drift detection i Azure Machine Learning er GA (generelt tilgjengelig).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#2\",\n    \"claim\": \"Azure Machine Learning støtter seks built-in monitoring signals: Data Drift | Prediction Drift | Data Quality | Feature Attribution Drift | Model Performance (Classification) | Model Performance (Regression).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#3\",\n    \"claim\": \"Data Drift-signalet i Azure ML sporer endringer i distribusjonen av model inputs med metrikkene Jensen-Shannon Distance | Population Stability Index | Normalized Wasserstein Distance | Two-Sample Kolmogorov-Smirnov Test | Pearson's Chi-Squared Test, og bruker training data eller recent production data som referansedata.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#4\",\n    \"claim\": \"Prediction Drift-signalet i Azure ML sporer endringer i distribusjonen av model outputs med metrikkene Jensen-Shannon Distance | Population Stability Index | Normalized Wasserstein Distance | Chebyshev Distance | Two-Sample Kolmogorov-Smirnov Test | Pearson's Chi-Squared Test, og bruker validation data eller recent production data som referansedata.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#5\",\n    \"claim\": \"Data Quality-signalet i Azure ML sporer dataintegritet i model inputs med metrikkene Null value rate | Data type error rate | Out-of-bounds rate, og bruker training data eller recent production data som referansedata.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#6\",\n    \"claim\": \"Feature Attribution Drift-signalet i Azure ML bruker metrikken Normalized Discounted Cumulative Gain, dekker model inputs + outputs, og krever training data som referansedata.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#7\",\n    \"claim\": \"Model Performance-signalet for classification i Azure ML måler Accuracy | Precision | Recall på model outputs, og krever ground truth data.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#8\",\n    \"claim\": \"Model Performance-signalet for regression i Azure ML måler MAE | MSE | RMSE på model outputs, og krever ground truth data.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#9\",\n    \"claim\": \"Azure Machine Learning støtter tre data quality metrics med opptil 0.00001 presisjon.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#10\",\n    \"claim\": \"Out-of-box monitoring for Azure ML online endpoints gir built-in signals for data drift | prediction drift | data quality.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#11\",\n    \"claim\": \"Model performance monitoring i Azure ML krever produksjons-output med unik ID (correlation ID fra data collector eller custom ID) | ground truth data med samme unike ID | join-kolonne som kobler produksjonsdata og ground truth.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#12\",\n    \"claim\": \"Monitoring-frekvens i Azure ML konfigureres med `frequency` (day | week | month) kombinert med `interval`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#13\",\n    \"claim\": \"Azure ML monitoring jobs kjøres på Serverless Spark Compute med instansstørrelser fra Standard_E4s_v3 til E64s_v3.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#14\",\n    \"claim\": \"Event Grid System Topic kan abonnere på `Run status changed`-events fra Azure ML og filtrere på `data.RunTags.azureml_modelmonitor_threshold_breached`.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#15\",\n    \"claim\": \"Event handlers for Azure ML monitoring-events er Azure Event Hubs | Azure Functions | Logic Apps.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/concept-model-monitoring?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#16\",\n    \"claim\": \"Microsoft Foundry observability omfatter continuous evaluation for generative AI-applikasjoner | AI Red Teaming (planlagt adversarial testing for safety/security) | Application Insights for sanntids operasjonelle metrikker.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/concepts/observability?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#17\",\n    \"claim\": \"Microsoft Foundry tilbyr evaluatorene Groundedness | Relevance | Fluency | Coherence for generativ AI.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#18\",\n    \"claim\": \"Python SDK v2 (`azure.ai.ml.entities`) eksponerer klassene DataDriftSignal | DataQualitySignal | ModelPerformanceSignal | FeatureAttributionDriftSignal | MonitorSchedule | MonitorDefinition for model monitoring.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#19\",\n    \"claim\": \"Med Azure CLI v2 opprettes en monitoring schedule med kommandoen `az ml schedule create -f <yaml>`.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-model-performance?view=azureml-api-2\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#20\",\n    \"claim\": \"Azure-regioner innenfor EU/EØS som kan velges for produksjonsdata og monitoring jobs inkluderer Norway East | Sweden Central | France Central.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#21\",\n    \"claim\": \"Azure Machine Learning lisensieres via Enterprise Agreement eller Pay-As-You-Go.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#22\",\n    \"claim\": \"Microsoft Foundry safety evaluations er hostet i East US 2 | France Central | UK South | Sweden Central.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/foundry/how-to/evaluate-generative-ai-app?view=foundry-classic\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/model-performance-drift-detection.md#23\",\n    \"claim\": \"Den eldre dataset-baserte data drift monitoring i Azure ML (azureml-api-1) er legacy og under utfasing, med migrering til Model Monitor.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/machine-learning/how-to-monitor-datasets?view=azureml-api-1\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/model-performance-drift-detection.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md",
    "claim_count": 22,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#1\",\n    \"claim\": \"Referansen for observability for Copilot-extensions er merket med status GA (per 2026-05).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#2\",\n    \"claim\": \"For Copilot Studio-agenter er Application Insights-integrasjon en out-of-the-box-funksjon som logger innkommende/utgående meldinger, topic-triggere og egendefinerte telemetri-hendelser.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#3\",\n    \"claim\": \"Telemetri for Copilot-extensions deles i fem lag med tilhørende verktøy: Copilot Studio Agent → Application Insights | Plugin/Connector Runtime → Application Insights SDK | LLM Interaction → Azure OpenAI metrics | User Engagement → Custom events | Security/Compliance → Microsoft Sentinel, Purview.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#4\",\n    \"claim\": \"Application Insights konfigureres i Copilot Studio under Settings → Advanced med en Connection string, og har tre valgfrie innstillinger: Log activities | Log sensitive Activity properties | Log custom telemetry events.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#5\",\n    \"claim\": \"Innstillingen «Log sensitive Activity properties» i Copilot Studio logger egenskapene userid | name | text | speak.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#6\",\n    \"claim\": \"customDimensions-feltet i Application Insights-telemetri fra Copilot Studio inneholder feltene type | channelId | designMode | locale | text.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#7\",\n    \"claim\": \"Aktivitetstypen `type` i customDimensions kan ha verdiene message | conversationUpdate | event | invoke.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#8\",\n    \"claim\": \"Feltet `channelId` i customDimensions kan ha verdiene emulator | directline | msteams | webchat.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/advanced-bot-framework-composer-capture-telemetry\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#9\",\n    \"claim\": \"Copilot Studio Workbook er i Preview og er tilgjengelig i Application Insights (Application Insights → Monitoring → Workbooks → «Copilot Studio Dashboard»).\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dynamics365/guidance/resources/copilot-studio-appinsights\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#10\",\n    \"claim\": \"Copilot Studio Workbook viser metrikkene Total conversations | latency | exceptions | tool usage | topic analytics.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/dynamics365/guidance/resources/copilot-studio-appinsights\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#11\",\n    \"claim\": \"Azure Copilot observability agent er i preview.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#12\",\n    \"claim\": \"Azure OpenAI-metrikker (token usage | model latency | throttling) emitteres automatisk til Azure Monitor uten ekstra konfigurasjon.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#13\",\n    \"claim\": \"Purview-revisjonslogger dekker Copilot Studio-hendelsene BotCreate | BotPublish | BotShare, og revisjonslogging må aktiveres for Microsoft 365-lisensinnehavere.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/microsoft-copilot-studio/guidance/sec-gov-phase5\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#14\",\n    \"claim\": \"Application Insights lagrer data i en Azure-region, for eksempel West Europe.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#15\",\n    \"claim\": \"EU Data Boundary er en Microsoft-forpliktelse per november 2024 og kan brukes for å holde data innenfor EU.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#16\",\n    \"claim\": \"Data Residency aktiveres i Application Insights under Settings → Data retention.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#17\",\n    \"claim\": \"Application Insights gir de første 5 GB datainntak gratis per måned.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#18\",\n    \"claim\": \"Application Insights har gratis dataoppbevaring i 90 dager; oppbevaring utover dette faktureres per GB per måned.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#19\",\n    \"claim\": \"Innebygd Copilot Studio-analyse krever Power Virtual Agents-lisens eller Copilot Studio-kapasitet.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#20\",\n    \"claim\": \"Application Insights-integrasjon krever et Azure-abonnement (gratisnivå tilgjengelig).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#21\",\n    \"claim\": \"Microsoft Sentinel for revisjonslogger krever Microsoft 365 E5 eller Sentinel standalone.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/observability-for-copilot-extensions.md#22\",\n    \"claim\": \"Power BI-dashboards krever Power BI Pro per bruker eller Premium-kapasitet.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/microsoft-cloud/dev/copilot/isv/observability-for-ai\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/observability-for-copilot-extensions.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  },
  {
    "file": "skills/ms-ai-governance/references/monitoring-observability/real-time-streaming-monitoring.md",
    "claim_count": 24,
    "prompt": "# Per-claim groundedness judge — bake-off **v3.1** (recall-hardened over v3's 3 confirmed FNs)\n\nv3.1 of `judge-claim-prompt-v3.md`. Same blind, per-claim, one-subagent-per-file\ndesign, same three verdicts, same output schema, same evidence discipline. v3.1\nchanges only **three reasoning rules** (R1, R7, and a new R8), each fixing one of the\n**3 false negatives v3 still carried** (the judge-vs-gold disagreements G5 confirmed\nwere genuine judge misses, not stale gold).\n\n**Why v3.1 exists — and what it is NOT (transparent, not p-hacking).** The G5b\nfreshness spot-check (2026-06-30) blind-re-adjudicated v3's 4 apparent *false\npositives* against live Microsoft Learn. **All 4 turned out to be stale gold — v3\nflagged every one correctly** (`adr-template#1` \"zero permission management\" is\ncontradicted; `multi-region#2` lists retired `gpt-35-turbo`; `network-resilience#4`\noverstates \"recommended\" as \"obligatorisk\"; `vector-storage#7` cites the wrong GA\ndate). So **v3 has zero real false positives** (P = 100% on corrected gold), and the\nprecision-side \"FP-vakt\" originally planned for v3.1 is **dropped — there is nothing to\ndefend.** v3.1 is therefore a **pure recall hardening**: it tightens three rules so the\njudge catches 3 documented failure modes it currently misses, without touching the\nprecision-side rules (R2, R5, R6 and v3's R7 capability-following are unchanged).\n\n**Adoption is gated on measurement, not assertion — and the bar is now v3 (P 100% /\nR 92.9% on corrected gold), not v2.** v3 sits at the precision ceiling, so v3.1 can only\nbe adopted if it **holds P = 100% AND lifts R above 92.9%** (catches FNs without\nintroducing a single new false positive). Any new FP drops P below 100% and fails the\ngate — keep v3. Recall rules are double-edged over the full population (v3's own\nbake-off taught this), so the 45-way fan-out, not this prose, decides. v3 results stay\nfrozen; v3.1 writes to `judge-bakeoff-results-v3.1.json` and is graded against the\ncorrected `gold-correctness-set.json`.\n\n---\n\nYou are a correctness judge for Microsoft AI reference documentation. You verify\nfactual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).\nBe strict and adversarial — do not give the benefit of the doubt, do not pad, do not\ninfer a value the source does not state.\n\nYou are judging claims extracted from `skills/ms-ai-governance/references/monitoring-observability/real-time-streaming-monitoring.md`. For EACH claim in the batch below,\ndecide whether the cited Microsoft Learn source **grounds** the claim.\n\n## The three verdicts (exhaustive, mutually exclusive)\n\n- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed\n  value(s). The page supports the claim. (Maps to gold `correct`.)\n- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a\n  **different / contradicting / superseded** value for what the claim asserts. The\n  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)\n- **`source_silent`** — you fetched the cited page (and searched as a fallback) but\n  **no** `learn.microsoft.com` page states the claimed value at all. You cannot\n  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure\n  pages typically lands here — that is expected, not a failure. **Exception: existence\n  claims — see Rule R2.**\n\n## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force)\n\nA claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s).\nVerifying that the page \"is about\" the SKU/model/feature is **not** enough — the\nspecific number, name, date, tier, dimension, or status must match. If the claim\nasserts value **X** and the page states a **different** value **Y** (even if adjacent\nor plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for\n`not_grounded`: you still need a fetched quote stating the **differing** value.\n\nApplies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`.\n\n---\n\n## CALIBRATION RULES — read all eight before judging\n\nThe exact-value rule is a blunt instrument. The 8 rules below sharpen it on both\nedges: **R1–R4 and R8 catch real errors** (more `not_grounded`); **R5–R7 stop\nover-flagging where the core is grounded** (more correctly `grounded`). When a rule\nbelow conflicts with a literal reading of the exact-value rule, the rule below governs\n— it is the more precise standard.\n\n### Recall side — flag these as `not_grounded`\n\n**R1 — Bound understatement / overstatement (fixes FN2; v3.1 splits lower vs upper).**\nA claim may assert a **bound**. Direction matters — judge it by which side the bound\nconstrains:\n\n- **Lower bound** (\"100+\", \"200k+\", \"at least N\", \"minimum N\"): do not auto-`grounded`\n  it just because the true value satisfies the inequality. Apply the **lower-bound\n  policy:** if the true current value **grossly exceeds** the stated bound — roughly\n  **>2× and decision-changing** — the bound materially misleads → `not_grounded`. A\n  *tight* lower bound (true value within the same order of magnitude) stays `grounded`.\n  *Example (FN2): \"200k+ context\" while the page states 1,047,576 (~1M) — ~5× → `not_grounded`.*\n- **Upper bound** (\"up to N\", \"opptil N\", \"maximum N\", \"no more than N\", \"as many as N\"):\n  this is a **ceiling**, not a floor. The lower-bound leniency does **NOT** apply. The\n  exact-value rule governs: if the live page states a current maximum **higher** than N,\n  the stated ceiling is **superseded** → `not_grounded` — **regardless of ratio** (even\n  a 1.1× exceedance breaks a ceiling). The claim tells the reader the limit is N when it\n  is really higher. *Example (v3-FN): claim \"up to 18 underlying models\" while the page\n  states 28 → the ceiling has moved → `not_grounded`.* (Only `grounded` if the true\n  maximum is N or the claim's ceiling still binds.)\n\n**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the\nclaim asserts that a named entity **exists / is offered / is in a list** (\"X is a\nbuilt-in judge\", \"feature Y is available\", \"tier Z exists\"), and you fetch the\nauthoritative page that *would* enumerate it and the entity is **absent**, that absence\nis **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve\n`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered\nprices). State in `reason` that you checked the canonical enumerating page and the\nentity was not present. *Example (FN5): claim \"99.99% SLA tier\" while the reliability\npage lists only 99.9% → absence of any 99.99% tier = `not_grounded`.*\n\n**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can\nbe superseded even when derived ratios survive. If the page shows the claim's framing\nhas been **replaced** (e.g. \"1 Unit Capacity\" → \"Quota Tiers\"; a renamed/retired\nmetric), the claim is `not_grounded` even if some embedded numbers still appear\nsomewhere — the claim describes a world that no longer exists. Check that the *unit and\nstructure* the claim assumes still match the current page, not just the digits.\n\n**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or\neffective-dated rows (\"Before April 3, 2024\", \"Legacy\", \"Retiring\"). A claim is\n`grounded` only if it matches the **current/effective** row. Matching a clearly\ntime-stamped *past* row is `not_grounded` (the value has since changed). Always locate\nthe row that applies *today*. *Example (FN6): storage limits matching only the\n\"Before April 3, 2024\" row while current limits differ → `not_grounded`.*\n\n**R8 — Multi-part claims: every load-bearing part must hold (v3.1 — fixes the\n`ai-foundry-dr#9` FN).** A single claim often bundles **several load-bearing\nsub-assertions** (a status AND a region; a capability AND a named target; a date AND a\nGA level). Verify **each load-bearing part separately**. If **any one** load-bearing\npart is contradicted by the source, the whole claim is `not_grounded` — even when the\nother parts check out. Do not let a correct first half earn a `grounded` for a wrong\nsecond half. *Example (v3-FN): \"Global training (Public Preview), cheaper, no data\nresidency; use regional in Norway East\" — the GA-vs-Preview part and the \"no residency\"\npart hold, but **Norway East is a Global (non-residency) training region, not a regional\none** → one load-bearing part is wrong → `not_grounded`.* (R8 is the mirror of R6:\nR6 forgives an **omitted, non-load-bearing** detail; R8 condemns a **stated,\nload-bearing** part that is wrong. Decide first whether the part is load-bearing — if\nthe claim *asserts* it and a reader would act on it, it is.)\n\n### Precision side — keep these `grounded` (do not over-flag)\n\n**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a\nnumeric claim merely because the exact string is not verbatim, when the asserted value\nis the **documented theoretical or benchmark equivalent** of what the page states and\nboth trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the\nsame technique, same order of magnitude, same direction). The exact-value rule targets\n*drifted/contradicting* values — not two Microsoft-sourced expressions of the same\nfact. If the page substantiates the magnitude and the technique, keep `grounded` and\nnote the equivalence in `reason`.\n\n**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish \"the\nclaim's **core** assertion is grounded but it omits a sub-category\" from \"the core is\nungrounded.\" If the page confirms the claim's **central** behavior/categorization and\nthe only gap is an *unstated additional* case the claim did not deny, that is\n`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the\npage **maps the core differently** or the claim **asserts** something the page\ncontradicts. Omission ≠ contradiction. (Contrast R8: an omitted case is forgiven here;\na *stated* but wrong load-bearing part is not — that is R8's domain.)\n\n**R7 — Follow the capability to its canonical page; don't punish illustrative numbers\n(fixes FP5; v3.1 sharpens the load-bearing carve-out).** If a claim asserts a **real\ncapability** and the cited `evidence_url` does not foreground it, search for the\n**canonical** page that documents the capability before judging — do not return\n`not_grounded` merely because the *cited* page was a weak choice. And when a capability\nis solidly grounded, do **not** flag it over an *illustrative* attached number (e.g.\n\"~0 RTO/RPO\", \"≈15 min\") that the claim offers as an order-of-magnitude illustration\nrather than a cited spec. Judge the **capability**; treat an illustrative figure as\ngrounded if the capability is.\n\n> **⚠️ Load-bearing carve-out (v3.1, fixes the `token-usage#3` FN).** R7's leniency\n> covers only *illustrative* values. It does **NOT** cover a value or **exact string**\n> that **IS the assertion** — a metric name, an API field, an SDK identifier, an enum\n> value, a specific date/version. When the claim's load-bearing content is the literal\n> name/string itself (e.g. \"the metrics are `PromptTokens` and `CompletionTokens`\"),\n> the exact-value rule applies in full: if the live page names them differently\n> (`ProcessedPromptTokens` / `InputTokens` / `GeneratedTokens` / `OutputTokens`), the\n> claim is `not_grounded`. \"Follow to the canonical page\" means find the **right\n> names**, not rescue wrong ones. A reader would copy that string into code; an\n> illustrative magnitude they would not.\n\n---\n\n## Procedure (per claim)\n\n1. **Identify the volatile assertion(s)** in the claim text — and when the claim\n   bundles several (R8), enumerate **each load-bearing part**. The `claim_type` tells\n   you what to check:\n   - `version` → model/API version, GA date, context window, max output, training cutoff\n   - `tpm` → tokens-per-minute / throughput / quota numbers\n   - `sku` → SKU name, tier, PTU minimums, deployment type\n   - `region` → regional availability\n   - `status` → GA / preview / retirement / deprecation status\n   - `taxonomy` → categorization, capability mapping, which-feature-does-what\n2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's\n   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not\n   address the assertion, run `microsoft_docs_search` to find the authoritative page.\n   **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak\n   cited URL is not the last word.\n3. **Exact-value entailment check** each checkable value (and each load-bearing part\n   under R8), then apply the calibration rules R1–R8. Classify which rule(s), if any,\n   govern the claim. For R1, first decide whether the bound is a **lower** bound (floor)\n   or an **upper** bound (ceiling) — they invert.\n4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim\n   quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence\n   absence), the quote is the canonical enumeration in which the entity does **not**\n   appear — quote the enumeration and state the entity is absent. No quote → `source_silent`.\n\n## Hard rules\n\n- Verify against the fetched page only. Do not rely on prior knowledge of model\n  specs / prices — those are exactly what may have drifted.\n- Stable identifiers are not volatile and are not your job to refute: regulation year\n  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10\n  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on\n  whatever volatile value it carries, else `source_silent`.\n- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.\n- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the\n  verdict (empty string for `source_silent`). `evidence_url` = the page you actually\n  used (may differ from the cited one if you fell back to search).\n- `rule` = which calibration rule governed, if any (`R1`–`R8`), else empty.\n\n## Batch to judge (from `skills/ms-ai-governance/references/monitoring-observability/real-time-streaming-monitoring.md`)\n\n[\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#1\",\n    \"claim\": \"Application Insights Live Metrics gir sanntidsovervåking av applikasjonsytelse med under 1 sekunds latency.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#2\",\n    \"claim\": \"Application Insights Live Metrics består av komponentene Live Metrics Stream (push-basert streaming av telemetri, < 1 sekund) | Custom Filters (real-time filtrering på URL, duration, telemetry type) | Server Instance Filtering | Performance Counters (Windows performance counters: CPU, memory, requests) | Exception Stack Traces | Control Channel (krever Entra ID-autentisering).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#3\",\n    \"claim\": \"Live Metrics-plattformstøtte dokumenteres langs fire kapabiliteter — Basic metrics | Performance metrics | Custom filtering | Sample telemetry — for plattformene .NET (ASP.NET Core) | ASP.NET (classic) | Java | Node.js | Python.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#4\",\n    \"claim\": \"ASP.NET (classic) støtter ikke Live Metrics med OpenTelemetry.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#5\",\n    \"claim\": \"For Java støtter Live Metrics basic metrics og performance metrics, custom filtering er ikke støttet, og sample telemetry krever V3.2.0 eller nyere.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#6\",\n    \"claim\": \"For Node.js krever Live Metrics basic metrics V1.3.0 eller nyere; performance metrics, custom filtering og sample telemetry er støttet.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#7\",\n    \"claim\": \"For Python krever Live Metrics distro 1.6.0+ for basic metrics, distro 1.8.2+ for performance metrics, distro 1.0.0+ for custom filtering og distro 1.5.0+ for sample telemetry.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#8\",\n    \"claim\": \"Live Metrics er aktivert som standard i Azure Monitor OpenTelemetry — options.EnableLiveMetrics har default-verdi true.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/opentelemetry-configuration#live-metrics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#9\",\n    \"claim\": \"Live Metrics-data medfører ingen ekstra kostnad; kun standard ingestion av underliggende telemetri faktureres.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#10\",\n    \"claim\": \"Fabric Real-Time Intelligence omfatter Real-Time Dashboard (no-code dashboard med KQL-queries) | Eventhouse (time-series-optimalisert database) | Eventstream (no-code streaming-pipelines) | Data Activator | Copilot for Dashboards | Git integration | Permission separation | Explorer Data.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#11\",\n    \"claim\": \"Data Activator utfører event detection med subsecond latency (< 1 sekund) og kan trigge handlinger.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/architecture/example-scenario/monitoring/monitoring-observable-systems-media\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#12\",\n    \"claim\": \"Real-Time Dashboards kan versjonskontrolleres via Git-integrasjon mot GitHub eller Azure DevOps.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#13\",\n    \"claim\": \"Real-Time Dashboard støtter datakildene Eventhouse (subsecond) | Azure Data Explorer (< 5 sekunder) | Application Insights (1-5 minutter) | Log Analytics (1-5 minutter).\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#14\",\n    \"claim\": \"Application Insights og Log Analytics har en iboende ingestion-latency på 1-5 minutter.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#15\",\n    \"claim\": \"Minimum auto-refresh-intervall for Fabric Real-Time Dashboard er 10 sekunder.\",\n    \"claim_type\": \"tpm\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#16\",\n    \"claim\": \"API-nøkler for Live Metrics' control channel ble retired 30. september 2025; control channel må sikres med Entra ID-autentisering i produksjon.\",\n    \"claim_type\": \"status\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#17\",\n    \"claim\": \"Live Metrics bruker et separat endepunkt, live.applicationinsights.azure.com, i stedet for endepunktet for vanlig telemetri.\",\n    \"claim_type\": \"taxonomy\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/troubleshoot/azure/azure-monitor/app-insights/troubleshoot-live-metrics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#18\",\n    \"claim\": \"Live Metrics krever TLS 1.2 eller nyere.\",\n    \"claim_type\": \"version\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/troubleshoot/azure/azure-monitor/app-insights/troubleshoot-live-metrics\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#19\",\n    \"claim\": \"Fabric-kapasitet med eventhouse kan velges i Norway East/West for datasuverenitet.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#20\",\n    \"claim\": \"Live Metrics-endepunktet live.applicationinsights.azure.com hostes i Azure public cloud, og Live Metrics er ikke tilgjengelig i Azure Government.\",\n    \"claim_type\": \"region\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#21\",\n    \"claim\": \"Entra ID-autentisering for Live Metrics er inkludert i Entra ID P1/P2 og er påkrevd fra september 2025.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/azure/azure-monitor/app/live-stream\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#22\",\n    \"claim\": \"Real-Time Dashboard er inkludert i Fabric-kapasiteten uten ekstra kostnad.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#23\",\n    \"claim\": \"Data Activator faktureres som egen SKU per reflex/trigger, med preview-prising ikke annonsert (TBA).\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/azure/foundry/concepts/observability\"\n  },\n  {\n    \"id\": \"ms-ai-governance/monitoring-observability/real-time-streaming-monitoring.md#24\",\n    \"claim\": \"Real-Time Dashboard krever minimum Fabric-kapasitet F2 SKU, som kan skaleres opp eller ned.\",\n    \"claim_type\": \"sku\",\n    \"evidence_url\": \"https://learn.microsoft.com/en-us/fabric/real-time-intelligence/real-time-dashboards-overview\"\n  }\n]\n\n## Output (strict JSON, no fence)\n\n```\n{\"file\":\"skills/ms-ai-governance/references/monitoring-observability/real-time-streaming-monitoring.md\",\"results\":[\n  {\"id\":\"<claim id>\",\"judge_verdict\":\"grounded|not_grounded|source_silent\",\"rule\":\"<R1-R8 or empty>\",\"evidence_url\":\"<url actually used>\",\"evidence_quote\":\"<verbatim quote or empty>\",\"reason\":\"<one sentence: what the source said vs the claim>\"}\n]}\n```\n"
  }
]