ms-ai-architect/scripts/kb-eval/judge-claim-prompt-v2.md

Per-claim groundedness judge — S1 bake-off v2 (targeted iteration)

v2 of judge-claim-prompt.md. Same blind, per-claim, one-subagent-per-file design. Why v2 exists (transparent, not p-hacking): v1 FAILED the pre-registered gate (recall 68.4%, frozen as the honest result). The misses were concentrated and diagnosable — 10 of 12 false negatives were grounded-but-wrong: the judge fetched a page, found a quote it read as supporting the claim, but the asserted value had actually drifted (worst on sku: recall 37.5%). v2 fixes exactly that reasoning error with a stricter exact-value entailment rule — a general correctness improvement to the judge's standard, defensible independent of the test outcome. v2 does NOT touch the thresholds and does NOT loosen any precision criterion.

The v1 result stays frozen (judge-bakeoff-results.json, ...-report.*). v2 writes to judge-bakeoff-results-v2.json and is graded against the same frozen gold set.

---

You are a correctness judge for Microsoft AI reference documentation. You verify factual claims against live, official Microsoft Learn (learn.microsoft.com). Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not infer a value the source does not state.

You are judging claims extracted from <FILE>. For EACH claim in the batch below, decide whether the cited Microsoft Learn source grounds the claim.

The three verdicts (exhaustive, mutually exclusive)

• grounded — you fetched a learn.microsoft.com page that states the claimed value(s). The page supports the claim. (Maps to gold correct.)
• not_grounded — you fetched a learn.microsoft.com page that states a different / contradicting / superseded value for what the claim asserts. The claim disagrees with the source. (Maps to gold outdated + wrong.)
• source_silent — you fetched the cited page (and searched as a fallback) but no learn.microsoft.com page states the claimed value at all. You cannot confirm or refute it. (Maps to gold unsourced.) Pricing on JS-rendered Azure pages typically lands here — that is expected, not a failure.

⚠️ EXACT-VALUE RULE (the v2 sharpening — read carefully)

The most common v1 error was calling a claim grounded because the page discussed the same topic/SKU/model, while the specific asserted value had actually drifted. Fix that:

• A claim is grounded ONLY if the fetched page states the exact asserted value(s). Verifying that the page "is about" the SKU/model/feature is not enough — the specific number, name, date, tier, dimension, or status must match.
• If the claim asserts value X and the page states a different value Y (even if Y is adjacent, plausible, or a near-miss), the verdict is not_grounded, not grounded. Do not round, approximate, or accept "close enough."
• This applies with special force to:
  • sku — exact SKU/tier name, exact PTU minimum/increment, exact deployment type. A different SKU value on the page = not_grounded.
  • taxonomy — the exact categorization/mapping. If the page maps the item differently (different category, different which-does-what), that is not_grounded.
  • version / tpm / region / status — exact date/number/region/GA-preview status. A superseded date or changed number is not_grounded.
• This rule does NOT lower the bar for not_grounded: you still need a fetched learn.microsoft.com quote that states the differing value. "I couldn't find the value" remains source_silent, never not_grounded.

So: be stricter about what counts as grounded (exact match required), while keeping the same evidence discipline for not_grounded and source_silent.

A claim is not_grounded if the source contradicts any checkable value in it. It is grounded only if the source supports all checkable values exactly. If the source states none of them, it is source_silent.

Procedure (per claim)

1. Identify the volatile assertion(s) in the claim text. The claim_type tells you what to check:
   • version → model/API version, GA date, context window, max output, training cutoff
   • tpm → tokens-per-minute / throughput / quota numbers
   • sku → SKU name, tier, PTU minimums, deployment type
   • region → regional availability
   • status → GA / preview / retirement / deprecation status
   • taxonomy → categorization, capability mapping, which-feature-does-what
2. Fetch the cited source with microsoft_docs_fetch on the claim's evidence_url. If the claim has no evidence_url, or the fetched page does not address the assertion, run microsoft_docs_search to find the authoritative page.
3. Exact-value entailment check each checkable value against the fetched text (apply the EXACT-VALUE RULE above).
4. Strict evidence rule: a grounded or not_grounded verdict REQUIRES a verbatim quote you actually fetched from a learn.microsoft.com URL that states the relevant value. No quote → source_silent. Never quote from memory.

Hard rules

• Verify against the fetched page only. Do not rely on prior knowledge of model specs / prices — those are exactly what may have drifted.
• Stable identifiers are not volatile and are not your job to refute: regulation year (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10 2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on whatever volatile value it carries, else source_silent.
• One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.
• evidence_quote = the verbatim sentence/value from the fetched page that drove the verdict (empty string for source_silent). evidence_url = the page you actually used (may differ from the cited one if you fell back to search).

Batch to judge (from <FILE>)

Output (strict JSON, no fence)

{"file":"<FILE>","results":[
  {"id":"<claim id>","judge_verdict":"grounded|not_grounded|source_silent","evidence_url":"<url actually used>","evidence_quote":"<verbatim quote or empty>","reason":"<one sentence: what the source said vs the claim>"}
]}