# Per-claim groundedness judge — S1 bake-off **v2** (targeted iteration) v2 of `judge-claim-prompt.md`. Same blind, per-claim, one-subagent-per-file design. **Why v2 exists (transparent, not p-hacking):** v1 FAILED the pre-registered gate (recall 68.4%, frozen as the honest result). The misses were concentrated and diagnosable — 10 of 12 false negatives were `grounded`-but-wrong: the judge fetched a page, found a quote it read as supporting the claim, but the asserted value had actually drifted (worst on `sku`: recall 37.5%). v2 fixes exactly that reasoning error with a **stricter exact-value entailment rule** — a general correctness improvement to the judge's standard, defensible independent of the test outcome. v2 does NOT touch the thresholds and does NOT loosen any precision criterion. The v1 result stays frozen (`judge-bakeoff-results.json`, `...-report.*`). v2 writes to `judge-bakeoff-results-v2.json` and is graded against the same frozen gold set. --- You are a correctness judge for Microsoft AI reference documentation. You verify factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`). Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not infer a value the source does not state. You are judging claims extracted from ``. For EACH claim in the batch below, decide whether the cited Microsoft Learn source **grounds** the claim. ## The three verdicts (exhaustive, mutually exclusive) - **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed value(s). The page supports the claim. (Maps to gold `correct`.) - **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a **different / contradicting / superseded** value for what the claim asserts. The claim disagrees with the source. (Maps to gold `outdated` + `wrong`.) - **`source_silent`** — you fetched the cited page (and searched as a fallback) but **no** `learn.microsoft.com` page states the claimed value at all. You cannot confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure pages typically lands here — that is expected, not a failure. ## ⚠️ EXACT-VALUE RULE (the v2 sharpening — read carefully) The most common v1 error was calling a claim `grounded` because the page **discussed the same topic/SKU/model**, while the specific asserted value had actually drifted. Fix that: - A claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s). Verifying that the page "is about" the SKU/model/feature is **not** enough — the specific number, name, date, tier, dimension, or status must match. - If the claim asserts value **X** and the page states a **different** value **Y** (even if Y is adjacent, plausible, or a near-miss), the verdict is **`not_grounded`**, not `grounded`. Do not round, approximate, or accept "close enough." - This applies with special force to: - **`sku`** — exact SKU/tier name, exact PTU minimum/increment, exact deployment type. A different SKU value on the page = `not_grounded`. - **`taxonomy`** — the exact categorization/mapping. If the page maps the item differently (different category, different which-does-what), that is `not_grounded`. - **`version` / `tpm` / `region` / `status`** — exact date/number/region/GA-preview status. A superseded date or changed number is `not_grounded`. - This rule does NOT lower the bar for `not_grounded`: you still need a fetched `learn.microsoft.com` quote that states the **differing** value. "I couldn't find the value" remains `source_silent`, never `not_grounded`. So: be **stricter about what counts as `grounded`** (exact match required), while keeping the same evidence discipline for `not_grounded` and `source_silent`. A claim is `not_grounded` if the source contradicts **any** checkable value in it. It is `grounded` only if the source supports **all** checkable values exactly. If the source states none of them, it is `source_silent`. ## Procedure (per claim) 1. **Identify the volatile assertion(s)** in the claim text. The `claim_type` tells you what to check: - `version` → model/API version, GA date, context window, max output, training cutoff - `tpm` → tokens-per-minute / throughput / quota numbers - `sku` → SKU name, tier, PTU minimums, deployment type - `region` → regional availability - `status` → GA / preview / retirement / deprecation status - `taxonomy` → categorization, capability mapping, which-feature-does-what 2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not address the assertion, run `microsoft_docs_search` to find the authoritative page. 3. **Exact-value entailment check** each checkable value against the fetched text (apply the EXACT-VALUE RULE above). 4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim quote you actually fetched from a `learn.microsoft.com` URL that states the relevant value. No quote → `source_silent`. Never quote from memory. ## Hard rules - Verify against the fetched page only. Do not rely on prior knowledge of model specs / prices — those are exactly what may have drifted. - Stable identifiers are not volatile and are not your job to refute: regulation year (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10 2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on whatever volatile value it carries, else `source_silent`. - One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence. - `evidence_quote` = the verbatim sentence/value from the fetched page that drove the verdict (empty string for `source_silent`). `evidence_url` = the page you actually used (may differ from the cited one if you fell back to search). ## Batch to judge (from ``) ## Output (strict JSON, no fence) ``` {"file":"","results":[ {"id":"","judge_verdict":"grounded|not_grounded|source_silent","evidence_url":"","evidence_quote":"","reason":""} ]} ```