From fde6ac1c1fb8289f84947ec96eb137fc011fdedf Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Tue, 30 Jun 2026 10:14:31 +0200 Subject: [PATCH] =?UTF-8?q?feat(ms-ai-architect):=20judge-claim-prompt-v3?= =?UTF-8?q?=20(R1-R7=20mot=208=20feilmoduser)=20=E2=80=94=20G4=20lukket,?= =?UTF-8?q?=20G1=20v3=20forfattet=20[skip-docs]?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- docs/ref-kb-correctness-program-2026-06.md | 5 +- scripts/kb-eval/judge-claim-prompt-v3.md | 181 +++++++++++++++++++++ 2 files changed, 184 insertions(+), 2 deletions(-) create mode 100644 scripts/kb-eval/judge-claim-prompt-v3.md diff --git a/docs/ref-kb-correctness-program-2026-06.md b/docs/ref-kb-correctness-program-2026-06.md index e8e2091..f2e471f 100644 --- a/docs/ref-kb-correctness-program-2026-06.md +++ b/docs/ref-kb-correctness-program-2026-06.md @@ -99,11 +99,12 @@ Status-nøkkel: 🔴 ikke startet · 🟡 pågår · 🟢 lukket. | **G1** | Judgen er ikke herdet mot de 8 dokumenterte feilmodusene (`source_silent`-maskerer-fravær, legacy-rad-match, ramme-skifte-tall-overlever, nedre-grense-understatement, eksakt-streng-pedanteri, taksonomi-nyanse, kapabilitet-bom) | Judge-FN/FP påvist i Spor 2b (8 mål, se `ref-kb-gold-reconciliation-2026-06.md`) | **Spor 2a** — judge-prompt-v3, MÅLT single vs v3/ensemble på herdet gull; adopter kun ved målt forbedring (ad-hoc-patch overfitter + bytter P/R) | 🔴 | Spor 1 korpus-pass (judgen brukes i ~2700 fetches) | | **G2** | Herdet judge er ikke wired inn i Port 2 (born-verified create-guard) + Port 3 (kadens) — uten innplugging binder ikke hardningen mekanisk | Re-introdusert drift ved nye/regenererte filer + kadens-bom | Del av Spor 2a→3: bytt ut v2 med v3 i `transform.mjs`-judge-passet + kadens-runneren | 🔴 (avh. G1) | Spor 1 | | **G3** | Ingen deterministisk gull-intern-konsistens-vakt (`verdict=correct` mens egen `notes` sier «uverifisert/illustrativ») | Gull-labeling-feil av FP1-klassen (selvmotsigende annotasjon) | Liten TDD-lint over `gold-correctness-set.json` (+ kjøres på fremtidige gull-bygg) | 🟢 **lukket 2026-06-30** | Spor 1 (nytt gull bygges) / §7 friskt utvalg | -| **G4** | Nedre-grense-policyen lever kun i prosa (denne dok + reconciliation-logg) — ikke kodet i judge-prompt ELLER `build-gold-set`-instruks | Re-introdusert nedre-grense-ambivalens i fremtidige gull-bygg + judge-kjøringer | Kod policyen inn i judge-prompt-v3 (G1) + build-gold-set-instruks | 🟡 **build-instruks ✅ (2026-06-30); judge-prompt-v3-del i G1** | Spor 1 / §7 friskt utvalg | +| **G4** | Nedre-grense-policyen lever kun i prosa (denne dok + reconciliation-logg) — ikke kodet i judge-prompt ELLER `build-gold-set`-instruks | Re-introdusert nedre-grense-ambivalens i fremtidige gull-bygg + judge-kjøringer | Kod policyen inn i judge-prompt-v3 (G1) + build-gold-set-instruks | 🟢 **kodet 2026-06-30** (build-instruks + v3 R1); håndheving rir på G1/G2-adopsjon | Spor 1 / §7 friskt utvalg | **Ikke mekanisme-gap, men sporet backlog (innhold, ikke loop):** reference-`.md`-fil-fiksene fra Spor 2b (FP1 11000+/40+, FP2 «kun», FP6 Preview/Norway-East, FN2–FN6 utdaterte tall) er **Spor 0/1**-innholdsarbeid — pekt per-claim i `notes`, ikke gjentakelses-mekanisme. Føres i Spor 0-manifest / Spor 1-korpus-pass, ikke her. ### Lukke-logg - **G3 🟢 lukket (2026-06-30).** Deterministisk lint `lib/gold-consistency.mjs` + CLI `lint-gold-consistency.mjs` + TDD `tests/kb-eval/test-gold-consistency.test.mjs` (4 tester); wiret som hard gate på `build-gold-set.mjs --write` (dry-run forhåndsviser ikke-fatalt). De 3 nåværende shippet-gull-treffene resolvert med `consistency_waiver` (én — `reserved-capacity-planning#5` — live kilde-bekreftet FØR waiver, verifiseringsplikt). **Bevist mot FP1:** rebuild fra batch-returns flagger `azure-ai-foundry.md#2` (verdict=correct, note «11000+/40+ uverifisert») — nøyaktig claimen Spor 2b fant via dyr live re-fetch. Linten fanger klassen gratis ved bygg. -- **G4 🟡 build-instruks ✅ (2026-06-30).** Nedre-grense-policyen + konsistens-policyen kodet inn i `build-gold-set.mjs` `_meta.lower_bound_policy` / `_meta.consistency_policy` (instruks subagenter følger ved fremtidig gull-bygg). Resterende halvdel — policyen inn i `judge-claim-prompt-v3.md` — lukkes i G1 når v3 forfattes. +- **G4 🟢 kodet (2026-06-30).** Nedre-grense-policyen + konsistens-policyen kodet inn i `build-gold-set.mjs` `_meta.lower_bound_policy` / `_meta.consistency_policy` (instruks subagenter følger ved fremtidig gull-bygg) OG inn i `judge-claim-prompt-v3.md` som **R1** (nedre-grense-understatement >~2× = `not_grounded`). Policyen er nå kodet begge steder; **håndheving** rir på at G1/G2 adopterer en prompt som bærer R1 (v3 hvis bakeoff-målt forbedring; ellers carry R1 inn i adoptert prompt). +- **G1 (Spor 2a) 🟡 v3 forfattet (2026-06-30), måling gjenstår.** `judge-claim-prompt-v3.md` bygget fra v2 med R1–R7 mot de 8 feilmodusene (R1–R4 strammere recall; R5–R7 gjenvunnet presisjon). Table-top: alle 8 mål mapper korrekt ved design. **Fan-out-målingen (255 claims/45 filer, live) ikke kjørt** — venter på operatør-valg av mekanisme (inline Agent / Workflow / Voyage). Adopter v3 KUN ved målt P/R-forbedring vs `-v2-reconciled` (P 92,1 / R 87,5). diff --git a/scripts/kb-eval/judge-claim-prompt-v3.md b/scripts/kb-eval/judge-claim-prompt-v3.md new file mode 100644 index 0000000..5a15439 --- /dev/null +++ b/scripts/kb-eval/judge-claim-prompt-v3.md @@ -0,0 +1,181 @@ +# Per-claim groundedness judge — bake-off **v3** (calibrated against 8 documented failure modes) + +v3 of `judge-claim-prompt-v2.md`. Same blind, per-claim, one-subagent-per-file design, +same three verdicts, same output schema, same evidence discipline. v3 changes only the +**reasoning rules**, calibrated against the 8 judge-vs-gold disagreements adjudicated in +Spor 2b (`docs/ref-kb-gold-reconciliation-2026-06.md`). + +**Why v3 exists (transparent, not p-hacking):** v2's exact-value rule correctly fixed +v1's *grounded-but-wrong* false negatives, but it **overshot** into false positives — +it flagged claims whose core was grounded over a pedantic exact-string miss (FP3), an +omitted sub-category (FP4), or an illustrative number on a solid capability (FP5). v2 +also still **missed** a distinct class of false negatives: lower bounds that grossly +understate (FN2), `source_silent` masking real absence (FN3/FN5), a superseded +organizing frame whose derived numbers survive (FN4), and a match against a legacy +historical table row (FN6). v3 adds calibration on BOTH sides: tighter recall on the +4 missed FN modes, recovered precision on the 3 over-flagged FP modes. Each rule below +is a general correctness improvement, defensible independent of the test outcome. + +**Adoption is gated on measurement, not assertion.** v3 must be graded against the +**reconciled** gold set and adopted ONLY if it beats v2 on P AND R (or holds one while +lifting the other) — adding nuance can swap precision for recall, so the bake-off, not +this prose, decides. v2 results stay frozen; v3 writes to +`judge-bakeoff-results-v3.json` and is graded against `gold-correctness-set.json`. + +--- + +You are a correctness judge for Microsoft AI reference documentation. You verify +factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`). +Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not +infer a value the source does not state. + +You are judging claims extracted from ``. For EACH claim in the batch below, +decide whether the cited Microsoft Learn source **grounds** the claim. + +## The three verdicts (exhaustive, mutually exclusive) + +- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed + value(s). The page supports the claim. (Maps to gold `correct`.) +- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a + **different / contradicting / superseded** value for what the claim asserts. The + claim disagrees with the source. (Maps to gold `outdated` + `wrong`.) +- **`source_silent`** — you fetched the cited page (and searched as a fallback) but + **no** `learn.microsoft.com` page states the claimed value at all. You cannot + confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure + pages typically lands here — that is expected, not a failure. **Exception: existence + claims — see v3 Rule R2.** + +## ⚠️ EXACT-VALUE RULE (inherited from v2 — still in force) + +A claim is `grounded` ONLY if the fetched page states the **exact** asserted value(s). +Verifying that the page "is about" the SKU/model/feature is **not** enough — the +specific number, name, date, tier, dimension, or status must match. If the claim +asserts value **X** and the page states a **different** value **Y** (even if adjacent +or plausible), the verdict is **`not_grounded`**. This rule does NOT lower the bar for +`not_grounded`: you still need a fetched quote stating the **differing** value. + +Applies with special force to `sku`, `taxonomy`, `version`, `tpm`, `region`, `status`. + +--- + +## v3 CALIBRATION RULES — read all seven before judging + +The exact-value rule is a blunt instrument. The 7 rules below sharpen it on both +edges: **R1–R4 catch real errors v2 missed** (more `not_grounded`); **R5–R7 stop +over-flagging where the core is grounded** (more correctly `grounded`). When a rule +below conflicts with a literal reading of the exact-value rule, the rule below governs +— it is the more precise standard. + +### Recall side — flag these as `not_grounded` (v2 missed them) + +**R1 — Lower-bound understatement (fixes FN2).** A claim may assert a *lower bound* +("100+", "200k+", "at least N", "up to N"). Do not auto-`grounded` it just because the +true value satisfies the inequality. Apply the **lower-bound policy:** if the true +current value **grossly exceeds** the stated bound — roughly **>2× and +decision-changing** — the bound materially misleads and the verdict is `not_grounded`. +A *tight* bound (true value within the same order of magnitude) stays `grounded`. +*Example (FN2): claim "200k+ context" while the model page states 1,047,576 (~1M) — +~5× understatement → `not_grounded`.* + +**R2 — `source_silent` does NOT excuse an existence claim (fixes FN3, FN5).** When the +claim asserts that a named entity **exists / is offered / is in a list** ("X is a +built-in judge", "feature Y is available", "tier Z exists"), and you fetch the +authoritative page that *would* enumerate it and the entity is **absent**, that absence +is **evidence the claim is wrong** — return `not_grounded`, not `source_silent`. Reserve +`source_silent` for values a page would not be expected to enumerate (e.g. JS-rendered +prices). State in `reason` that you checked the canonical enumerating page and the +entity was not present. *Example (FN5): claim "99.99% SLA tier" while the reliability +page lists only 99.9% → absence of any 99.99% tier = `not_grounded`.* + +**R3 — Frame/unit replacement (fixes FN4).** A claim's **organizing frame or unit** can +be superseded even when derived ratios survive. If the page shows the claim's framing +has been **replaced** (e.g. "1 Unit Capacity" → "Quota Tiers"; a renamed/retired +metric), the claim is `not_grounded` even if some embedded numbers still appear +somewhere — the claim describes a world that no longer exists. Check that the *unit and +structure* the claim assumes still match the current page, not just the digits. + +**R4 — Current row, never a legacy row (fixes FN6).** Pages often carry historical or +effective-dated rows ("Before April 3, 2024", "Legacy", "Retiring"). A claim is +`grounded` only if it matches the **current/effective** row. Matching a clearly +time-stamped *past* row is `not_grounded` (the value has since changed). Always locate +the row that applies *today*. *Example (FN6): storage limits matching only the +"Before April 3, 2024" row while current limits differ → `not_grounded`.* + +### Precision side — keep these `grounded` (v2 over-flagged them) + +**R5 — Documented theoretical↔benchmark equivalence (fixes FP3).** Do not flag a +numeric claim merely because the exact string is not verbatim, when the asserted value +is the **documented theoretical or benchmark equivalent** of what the page states and +both trace to Microsoft sources (e.g. a theoretical max vs a measured benchmark of the +same technique, same order of magnitude, same direction). The exact-value rule targets +*drifted/contradicting* values — not two Microsoft-sourced expressions of the same +fact. If the page substantiates the magnitude and the technique, keep `grounded` and +note the equivalence in `reason`. + +**R6 — Core grounded, detail omitted ≠ ungrounded (fixes FP4).** Distinguish "the +claim's **core** assertion is grounded but it omits a sub-category" from "the core is +ungrounded." If the page confirms the claim's **central** behavior/categorization and +the only gap is an *unstated additional* case the claim did not deny, that is +`grounded` (the claim is incomplete, not wrong). Reserve `not_grounded` for when the +page **maps the core differently** or the claim **asserts** something the page +contradicts. Omission ≠ contradiction. + +**R7 — Follow the capability to its canonical page; don't punish illustrative numbers +(fixes FP5).** If a claim asserts a **real capability** and the cited `evidence_url` +does not foreground it, search for the **canonical** page that documents the capability +before judging — do not return `not_grounded` merely because the *cited* page was a +weak choice. And when a capability is solidly grounded, do **not** flag it over an +*illustrative* attached number (e.g. "~0 RTO/RPO", "≈15 min") that the claim offers as +an order-of-magnitude illustration rather than a cited spec. Judge the **capability**; +treat an illustrative figure as grounded if the capability is. (If the number is itself +the load-bearing assertion, the exact-value rule still applies.) + +--- + +## Procedure (per claim) + +1. **Identify the volatile assertion(s)** in the claim text. The `claim_type` tells + you what to check: + - `version` → model/API version, GA date, context window, max output, training cutoff + - `tpm` → tokens-per-minute / throughput / quota numbers + - `sku` → SKU name, tier, PTU minimums, deployment type + - `region` → regional availability + - `status` → GA / preview / retirement / deprecation status + - `taxonomy` → categorization, capability mapping, which-feature-does-what +2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's + `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not + address the assertion, run `microsoft_docs_search` to find the authoritative page. + **Under R2/R7, actively seek the canonical enumerating/capability page** — a weak + cited URL is not the last word. +3. **Exact-value entailment check** each checkable value, then apply the v3 calibration + rules R1–R7. Classify which rule(s), if any, govern the claim. +4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a verbatim + quote you actually fetched from a `learn.microsoft.com` URL. For R2 (existence + absence), the quote is the canonical enumeration in which the entity does **not** + appear — quote the enumeration and state the entity is absent. No quote → `source_silent`. + +## Hard rules + +- Verify against the fetched page only. Do not rely on prior knowledge of model + specs / prices — those are exactly what may have drifted. +- Stable identifiers are not volatile and are not your job to refute: regulation year + (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10 + 2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on + whatever volatile value it carries, else `source_silent`. +- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence. +- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the + verdict (empty string for `source_silent`). `evidence_url` = the page you actually + used (may differ from the cited one if you fell back to search). +- `rule` = which v3 calibration rule governed, if any (`R1`–`R7`), else empty. + +## Batch to judge (from ``) + + + +## Output (strict JSON, no fence) + +``` +{"file":"","results":[ + {"id":"","judge_verdict":"grounded|not_grounded|source_silent","rule":"","evidence_url":"","evidence_quote":"","reason":""} +]} +```