feat(ms-ai-architect): S1 judge bake-off harness + forhåndsregistrert gate (TDD) [skip-docs]

Deterministisk de-risk-harness for Fase 3-judgen (kjøres på frosset 373-påstands gull-sett):
- lib/judge-bakeoff.mjs: P-filter (volatil+fetchbar, price ekskl.), confusion-matrix
  for 3 armer (staleness/judge/hybrid), Wilson-bånd, forhåndsregistrert gate. 14 tester.
- extract-judge-claims.mjs: blind manifest (255 påstander, 0 label-lekkasje — testet invariant).
- judge-claim-prompt.md: blind per-påstands groundedness-judge (Opus xhigh, microsoft_docs_fetch).
- run-judge-bakeoff.mjs: join gull+results på id, gate-rapport (.json/.md).

Gate FORHÅNDSREGISTRERT (operatørvalg, før fan-out): recall ≥0.80, presisjon ≥0.70,
OG slår staleness (0/38). Evalueringspop P = 240 verifiserbare, 38 positive.
Suite 551/551 (538 + 13 nye).

docs/ref-kb-workflow-plan-2026-06.md

@@ -148,6 +148,8 @@ Fase 0 ✅ lukket; gaten sa **BYGG Fase 3 (scoped)**. Gjenstående arbeid har **
 
 **S1 — Judge-prototype + bake-off på frosset gull-sett (DE-RISK, retnings-avgjørende).** Bygg en minimal per-påstands groundedness-judge (Opus 4.8 xhigh) og kjør den mot de 373 gull-påstandene (hver har allerede `evidence_url` + ordrett påstand). Mål **staleness vs judge vs hybrid** på volatil populasjon, scoped til fetchbare claim_types (`taxonomy/status/version/tpm/sku/region`; `price` ekskludert). Trenger **ingen** backfill/frontmatter — gull-settet er selvbærende. **Gate:** judge presisjon/recall mot gull ≥ avtalt terskel OG slår staleness (recall 0/40); kjent felle adressert (invertert leverage — judgen «vinner» ikke ved å auto-score stabile lav-risiko-påstander). **HVIS FAIL:** stopp — judge ikke berettiget; fall tilbake på staleness + operatør-gating; **bygg IKKE S3**. ~1 sesjon de-risker hele kritisk sti.
 
+> **FORHÅNDSREGISTRERT GATE (låst 2026-06-26, operatørvalg, FØR fan-out):** judge **recall ≥ 0,80 OG presisjon ≥ 0,70** (punktestimat), målt på den verifiserbare evaluerings-populasjonen P = volatil + fetchbar claim_type (price ekskl.) = **240 påstander, 38 positive**. PLUSS nødvendig betingelse: judge-recall > staleness-recall (staleness = 0/38). Wilson 95 %-bånd rapporteres som kontekst (n=38 → bredt bånd); grensetilfeller flagges for operatør, ikke mekanisk avvist. Strengt nivå valgt: bygg S3 KUN hvis judgen er svært sterk. **Harness (testet, 14 tester):** `lib/judge-bakeoff.mjs` + `extract-judge-claims.mjs` (blind manifest, 0 label-lekkasje) + `judge-claim-prompt.md` (blind per-påstands-judge) + `run-judge-bakeoff.mjs --min-recall 0.80 --min-precision 0.70`. Blindhet: judgen ser aldri gull-verdict; join på `id` i koden etterpå.
+
 **S2 — Fase 2: minimal type-tag (judge-uavhengig, nyttig uansett).** Klassifiser ~389 filer `reference|template|methodology|regulatory` (sidecar-manifest el. mappekonvensjon — IKKE full YAML ennå). Skiller de ~83 kildeløse legitimt (mal/metodikk: `decision-trees`, `cost-models`) fra MS-fakta-uten-kilde. NY `scripts/kb-eval/classify-ref-type.mjs` (TDD-først). Kan kjøres før/parallelt med S1 (billig). **Scope:** klassifiser advisor-filer, men MUTÉR dem ikke (Cosmo-kollisjon). **Gate:** hver fil har én type; kildeløse delt i to bøtter; reproduserbar; suite grønn.
 
 **S3 — Fase 3-forutsetning (a)+(b): backfill + full frontmatter (KUN hvis S1 passerte).** Produksjons-scaffolding for judgen: gi hver verifiserbar fil (~306 som siterer ≥1 MS Learn-URL, **EKSKL. advisor**) en `authority_source` + full frontmatter (`type/source/verified`, OKF-form). Advisor folder inn i **S-Cosmo**. Gated bak S1 fordi dette er judge-spesifikt og wasted hvis judgen feilet. **Gate:** hver ikke-advisor verifiserbar fil har `authority_source`; frontmatter validerer; suite grønn.

scripts/kb-eval/extract-judge-claims.mjs

@@ -0,0 +1,51 @@
+#!/usr/bin/env node
+// extract-judge-claims.mjs — S1 step: emit the BLIND claim manifest the judge
+// subagents are graded on. Reads the frozen gold set, filters to the eval
+// population P (volatile + fetchable, price excluded) via the tested lib, strips
+// every claim to the fields the judge may see (no gold verdict / notes /
+// lastmod_changed / stratum — no label leakage), and writes a flat manifest.
+//
+// The fan-out groups these by file at dispatch time (one judge subagent per file,
+// mirroring how the gold set was built). The harness later joins the judge results
+// back to gold by id and runs run-judge-bakeoff.mjs.
+//
+// Usage:  node scripts/kb-eval/extract-judge-claims.mjs [--write]
+//   (default: print population summary; --write persists data/judge-bakeoff-claims.json)
+
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { blindClaims, evalPopulation } from './lib/judge-bakeoff.mjs';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const DATA = path.join(__dirname, 'data');
+
+const gold = JSON.parse(fs.readFileSync(path.join(DATA, 'gold-correctness-set.json'), 'utf8'));
+const blind = blindClaims(gold.claims);
+
+// File -> claim count, for fan-out sizing (read-only summary).
+const byFile = {};
+for (const c of blind) (byFile[c.file] ||= 0), (byFile[c.file] += 1);
+const files = Object.keys(byFile).length;
+const withUrl = blind.filter((c) => c.evidence_url).length;
+
+if (process.argv.includes('--write')) {
+  const out = path.join(DATA, 'judge-bakeoff-claims.json');
+  const payload = {
+    _meta: {
+      source: 'gold-correctness-set.json',
+      population: 'volatile + fetchable claim_type (price excluded)',
+      blind: 'gold verdict/notes/lastmod_changed/stratum withheld — no label leakage',
+      claim_count: blind.length,
+      files,
+    },
+    claims: blind,
+  };
+  fs.writeFileSync(out, JSON.stringify(payload, null, 2) + '\n');
+  console.log(`wrote ${out} (${blind.length} blind claims across ${files} files)`);
+} else {
+  console.log(`eval population P: ${evalPopulation(gold.claims).length} claims`);
+  console.log(`blind manifest: ${blind.length} claims across ${files} files`);
+  console.log(`with evidence_url: ${withUrl} | without (judge falls back to docs_search): ${blind.length - withUrl}`);
+  console.log('(dry run — pass --write to persist judge-bakeoff-claims.json)');
+}

scripts/kb-eval/judge-claim-prompt.md

@@ -0,0 +1,80 @@
+# Per-claim groundedness judge — S1 bake-off (Fase 3 de-risk)
+
+Canonical instruction for the per-claim correctness judge. Runs as an Opus 4.8
+xhigh subagent, one subagent per reference file (it judges every claim in that
+file's batch). The dispatcher fills `<FILE>` and the `<CLAIMS>` batch from
+`data/judge-bakeoff-claims.json` (the BLIND manifest — it carries no gold verdict).
+
+The judge is **blind**: it never sees the gold label. Its verdict is joined back to
+the gold set by `id` in `run-judge-bakeoff.mjs` and scored as a detection task. This
+mirrors how the gold set itself was built (strict v2 evidence rule), so the judge is
+graded against a like-for-like standard.
+
+---
+
+You are a correctness judge for Microsoft AI reference documentation. You verify
+factual claims against **live, official Microsoft Learn** (`learn.microsoft.com`).
+Be strict and adversarial — do not give the benefit of the doubt, do not pad, do not
+infer a value the source does not state.
+
+You are judging claims extracted from `<FILE>`. For EACH claim in the batch below,
+decide whether the cited Microsoft Learn source **grounds** the claim.
+
+## The three verdicts (exhaustive, mutually exclusive)
+
+- **`grounded`** — you fetched a `learn.microsoft.com` page that states the claimed
+  value(s). The page supports the claim. (Maps to gold `correct`.)
+- **`not_grounded`** — you fetched a `learn.microsoft.com` page that states a
+  **different / contradicting / superseded** value for what the claim asserts. The
+  claim disagrees with the source. (Maps to gold `outdated` + `wrong`.)
+- **`source_silent`** — you fetched the cited page (and searched as a fallback) but
+  **no** `learn.microsoft.com` page states the claimed value at all. You cannot
+  confirm or refute it. (Maps to gold `unsourced`.) Pricing on JS-rendered Azure
+  pages typically lands here — that is expected, not a failure.
+
+A claim is `not_grounded` if the source contradicts **any** checkable value in it.
+It is `grounded` only if the source supports **all** checkable values. If the source
+states none of them, it is `source_silent`.
+
+## Procedure (per claim)
+
+1. **Identify the volatile assertion(s)** in the claim text. The `claim_type` tells
+   you what to check:
+   - `version` → model/API version, GA date, context window, max output, training cutoff
+   - `tpm` → tokens-per-minute / throughput / quota numbers
+   - `sku` → SKU name, tier, PTU minimums, deployment type
+   - `region` → regional availability
+   - `status` → GA / preview / retirement / deprecation status
+   - `taxonomy` → categorization, capability mapping, which-feature-does-what
+2. **Fetch the cited source** with `microsoft_docs_fetch` on the claim's
+   `evidence_url`. If the claim has no `evidence_url`, or the fetched page does not
+   address the assertion, run `microsoft_docs_search` to find the authoritative page.
+3. **Entailment check** each checkable value against the fetched text.
+4. **Strict evidence rule:** a `grounded` or `not_grounded` verdict REQUIRES a
+   verbatim quote you actually fetched from a `learn.microsoft.com` URL that states
+   the relevant value. No quote → `source_silent`. Never quote from memory.
+
+## Hard rules
+
+- Verify against the fetched page only. Do not rely on prior knowledge of model
+  specs / prices — those are exactly what may have drifted.
+- Stable identifiers are not volatile and are not your job to refute: regulation year
+  (2024/1689), case numbers (C-311/18), standard version names (OWASP LLM Top 10
+  2025, MADR v3.0), file names. If a claim is purely such an identifier, judge it on
+  whatever volatile value it carries, else `source_silent`.
+- One verdict per claim. Return EXACTLY the JSON below — no prose, no markdown fence.
+- `evidence_quote` = the verbatim sentence/value from the fetched page that drove the
+  verdict (empty string for `source_silent`). `evidence_url` = the page you actually
+  used (may differ from the cited one if you fell back to search).
+
+## Batch to judge (from `<FILE>`)
+
+<CLAIMS>
+
+## Output (strict JSON, no fence)
+
+```
+{"file":"<FILE>","results":[
+  {"id":"<claim id>","judge_verdict":"grounded|not_grounded|source_silent","evidence_url":"<url actually used>","evidence_quote":"<verbatim quote or empty>","reason":"<one sentence: what the source said vs the claim>"}
+]}
+```

scripts/kb-eval/lib/judge-bakeoff.mjs

@@ -0,0 +1,206 @@
+// scripts/kb-eval/lib/judge-bakeoff.mjs — S1 judge bake-off aggregation (deterministic).
+//
+// De-risks the Fase 3 correctness judge BEFORE production scaffolding (S3) is built:
+// grade a per-claim groundedness judge against the frozen gold set and decide whether
+// it beats the cheap staleness baseline (recall 0/40) by enough to justify ~2700
+// non-batchable microsoft_docs_fetch calls per full pass. Pure functions only — no
+// I/O, no Date.now/Math.random.
+//
+// Eval population P = volatile stratum + fetchable claim_type (price excluded). This
+// is the only population that matters: it is where the real errors concentrate, and
+// scoping to it structurally avoids the "inverted leverage" trap (a judge that wins
+// only by auto-scoring cheap stable claims it was never at risk on).
+//
+// Detection confusion matrix vs the gold verdicts:
+//   gold positive  = verdict ∈ {outdated, wrong}   (a real error the judge should catch)
+//   gold negative  = verdict === 'correct'
+//   excluded       = verdict === 'unsourced'        (no ground-truth value to grade against)
+//   predicted positive (flag) = judge_verdict === 'not_grounded'
+// judge_verdict vocabulary: 'grounded' | 'not_grounded' | 'source_silent'.
+
+import { wilson } from './base-rate.mjs';
+
+export const FETCHABLE_TYPES = new Set([
+  'taxonomy', 'status', 'version', 'tpm', 'sku', 'region',
+]);
+const ERROR_VERDICTS = new Set(['outdated', 'wrong']);
+
+// P = the claims the bake-off is measured on. A claim is in P iff it is volatile and
+// of a fetchable claim_type (price is excluded — 74% unsourced, JS-rendered Azure
+// pages defeat microsoft_docs_fetch, so a fetch-based judge cannot reach it either).
+export function evalPopulation(claims) {
+  return claims.filter(
+    (c) => c.stratum === 'volatile' && FETCHABLE_TYPES.has(c.claim_type),
+  );
+}
+
+// The BLIND manifest handed to the judge subagents: every claim in P, stripped to
+// the fields the judge is allowed to see. The gold verdict, notes, lastmod_changed
+// and stratum are deliberately withheld so the eval is blind (no label leakage).
+// All of P is included (unsourced too) — the judge must not know which claims the
+// human could not source; reproducing that boundary as source_silent is itself a
+// measured signal.
+const BLIND_FIELDS = ['id', 'file', 'skill', 'claim', 'claim_type', 'evidence_url'];
+export function blindClaims(claims) {
+  return evalPopulation(claims).map((c) => {
+    const out = {};
+    for (const k of BLIND_FIELDS) out[k] = c[k];
+    return out;
+  });
+}
+
+export function stalenessFlag(claim) {
+  return claim.lastmod_changed === true;
+}
+export function judgeFlag(claim) {
+  return claim.judge_verdict === 'not_grounded';
+}
+export function hybridFlag(claim) {
+  return stalenessFlag(claim) || judgeFlag(claim);
+}
+
+// Grade one detection arm over a list of *verifiable* claims (gold correct/outdated/
+// wrong only — unsourced must be filtered out before calling). flagFn(claim) -> bool
+// is the arm's predicted-error rule. Returns the confusion matrix plus precision /
+// recall / F1. precision is null when nothing is flagged (0/0, not NaN); recall is
+// null when there are no positives at all.
+export function gradeArm(verifiableClaims, flagFn) {
+  let tp = 0, fp = 0, fn = 0, tn = 0;
+  for (const c of verifiableClaims) {
+    const isError = ERROR_VERDICTS.has(c.verdict);
+    const flagged = flagFn(c);
+    if (isError && flagged) tp += 1;
+    else if (!isError && flagged) fp += 1;
+    else if (isError && !flagged) fn += 1;
+    else tn += 1;
+  }
+  const positives = tp + fn;
+  const negatives = fp + tn;
+  const flagged = tp + fp;
+  const precision = flagged ? tp / flagged : null;
+  const recall = positives ? tp / positives : null;
+  const f1 =
+    precision != null && recall != null && precision + recall > 0
+      ? (2 * precision * recall) / (precision + recall)
+      : null;
+  // Wilson 95% bands surface sampling noise — the positive sample is small (≈38),
+  // so a point estimate near the threshold should not be read as crisp. Bands are
+  // reported context; the gate itself uses the point estimate.
+  const recallWilson = positives ? wilson(tp, positives) : null;
+  const precisionWilson = flagged ? wilson(tp, flagged) : null;
+  return {
+    tp, fp, fn, tn, positives, negatives, flagged,
+    precision, recall, f1, recallWilson, precisionWilson,
+  };
+}
+
+// The PRE-REGISTERED gate. thresholds = { minRecall, minPrecision }. Pass requires
+// the judge to clear both thresholds AND strictly beat the staleness baseline's
+// recall (the directional point of S1: staleness recall is 0/40, so any positive
+// recall beats it — but the threshold guards against a judge too weak to justify
+// the fetch cost). A null judge precision (judge flagged nothing) cannot clear a
+// positive minPrecision.
+export function gateDecision(judgeArm, stalenessArm, thresholds) {
+  const { minRecall, minPrecision } = thresholds;
+  const recall = judgeArm.recall ?? 0;
+  const precision = judgeArm.precision; // may be null
+  const staleRecall = stalenessArm.recall ?? 0;
+
+  const recallOk = recall >= minRecall;
+  const precisionOk = precision != null && precision >= minPrecision;
+  const beatsStaleness = recall > staleRecall;
+  const pass = recallOk && precisionOk && beatsStaleness;
+
+  const reasons = [];
+  if (!recallOk) reasons.push(`recall ${fmt(recall)} < minRecall ${minRecall}`);
+  if (!precisionOk) {
+    reasons.push(
+      precision == null
+        ? 'judge flagged nothing (precision undefined)'
+        : `precision ${fmt(precision)} < minPrecision ${minPrecision}`,
+    );
+  }
+  if (!beatsStaleness) {
+    reasons.push(`recall ${fmt(recall)} does not beat staleness ${fmt(staleRecall)}`);
+  }
+  if (pass) reasons.push('all criteria met');
+  return { pass, recallOk, precisionOk, beatsStaleness, thresholds, reasons };
+}
+
+function fmt(x) {
+  return x == null ? 'n/a' : x.toFixed(3);
+}
+
+// source_silent is the judge's "I fetched but the page does not state this value"
+// verdict. It is diagnostic, not a flag. Two populations matter:
+//   - on verifiable claims (the human DID find a value): every source_silent is a
+//     judge verification miss. Split by whether the gold value was correct vs an error
+//     (a source_silent on a real error is a missed catch via "can't verify").
+//   - on unsourced-in-P claims (the human ALSO found no value): source_silent here is
+//     agreement — a good sign the judge reproduces the unverifiable boundary.
+function sourceSilentDiagnostics(P) {
+  let onVerifiableNegative = 0;
+  let onVerifiableError = 0;
+  let agreesWithUnsourced = 0;
+  let disagreesWithUnsourced = 0;
+  for (const c of P) {
+    const silent = c.judge_verdict === 'source_silent';
+    if (c.verdict === 'unsourced') {
+      if (silent) agreesWithUnsourced += 1;
+      else disagreesWithUnsourced += 1;
+    } else if (silent) {
+      if (ERROR_VERDICTS.has(c.verdict)) onVerifiableError += 1;
+      else onVerifiableNegative += 1;
+    }
+  }
+  return {
+    onVerifiableNegative,
+    onVerifiableError,
+    agreesWithUnsourced,
+    disagreesWithUnsourced,
+  };
+}
+
+// Per claim_type judge confusion matrix over the verifiable subset (so the report can
+// show where the judge earns its keep — sku/version/tpm carry the highest error rates).
+function judgeByClaimType(verifiable) {
+  const byType = {};
+  for (const c of verifiable) {
+    (byType[c.claim_type] ||= []).push(c);
+  }
+  const out = {};
+  for (const [t, arr] of Object.entries(byType)) {
+    out[t] = gradeArm(arr, judgeFlag);
+  }
+  return out;
+}
+
+// Top-level: join already done by the caller (each claim carries judge_verdict).
+// Filters to P, splits verifiable vs unsourced, grades all three arms over the same
+// verifiable set, computes diagnostics + the gate, returns one report object.
+export function computeBakeoff(joinedClaims, thresholds) {
+  const P = evalPopulation(joinedClaims);
+  const verifiable = P.filter((c) => c.verdict !== 'unsourced');
+  const unsourcedInP = P.length - verifiable.length;
+
+  const arms = {
+    staleness: gradeArm(verifiable, stalenessFlag),
+    judge: gradeArm(verifiable, judgeFlag),
+    hybrid: gradeArm(verifiable, hybridFlag),
+  };
+  const gate = gateDecision(arms.judge, arms.staleness, thresholds);
+
+  return {
+    population: {
+      total: P.length,
+      verifiable: verifiable.length,
+      positives: verifiable.filter((c) => ERROR_VERDICTS.has(c.verdict)).length,
+      negatives: verifiable.filter((c) => c.verdict === 'correct').length,
+      unsourcedInP,
+    },
+    arms,
+    sourceSilent: sourceSilentDiagnostics(P),
+    byClaimType: judgeByClaimType(verifiable),
+    gate,
+  };
+}

scripts/kb-eval/run-judge-bakeoff.mjs

@@ -0,0 +1,155 @@
+#!/usr/bin/env node
+// run-judge-bakeoff.mjs — S1 final glue: join the blind judge results back to the
+// frozen gold set, grade the judge / staleness / hybrid arms, apply the
+// PRE-REGISTERED gate, and write the bake-off report. All math lives in tested
+// lib/judge-bakeoff.mjs; this CLI is thin wiring (same shape as compute-base-rate.mjs).
+//
+// The thresholds are required flags, not defaults: the gate must be locked BEFORE the
+// judge fan-out runs (pre-registration), and the locked values are recorded in the
+// report _meta so the decision is auditable.
+//
+// Usage:
+//   node scripts/kb-eval/run-judge-bakeoff.mjs --min-recall 0.70 --min-precision 0.60 [--write]
+//
+// Inputs:  data/gold-correctness-set.json  (answer key)
+//          data/judge-bakeoff-results.json  ({ results: [{id, judge_verdict, ...}] })
+// Outputs: data/judge-bakeoff-report.{json,md}  (with --write)
+
+import fs from 'node:fs';
+import path from 'node:path';
+import { fileURLToPath } from 'node:url';
+import { computeBakeoff, evalPopulation } from './lib/judge-bakeoff.mjs';
+
+const __dirname = path.dirname(fileURLToPath(import.meta.url));
+const DATA = path.join(__dirname, 'data');
+
+function flag(name) {
+  const i = process.argv.indexOf(name);
+  return i >= 0 ? process.argv[i + 1] : undefined;
+}
+const minRecall = Number(flag('--min-recall'));
+const minPrecision = Number(flag('--min-precision'));
+if (!Number.isFinite(minRecall) || !Number.isFinite(minPrecision)) {
+  console.error('error: --min-recall and --min-precision are required (pre-registered gate)');
+  process.exit(2);
+}
+const thresholds = { minRecall, minPrecision };
+
+const gold = JSON.parse(fs.readFileSync(path.join(DATA, 'gold-correctness-set.json'), 'utf8'));
+const resultsPath = path.join(DATA, 'judge-bakeoff-results.json');
+if (!fs.existsSync(resultsPath)) {
+  console.error(`error: ${resultsPath} not found — run the judge fan-out first`);
+  process.exit(2);
+}
+const judge = JSON.parse(fs.readFileSync(resultsPath, 'utf8'));
+
+// Join judge verdicts back to gold by id.
+const verdictById = new Map((judge.results || []).map((r) => [r.id, r.judge_verdict]));
+const joined = gold.claims.map((c) => ({ ...c, judge_verdict: verdictById.get(c.id) }));
+
+// Completeness check: every claim in P must have a judge verdict.
+const missing = evalPopulation(joined).filter((c) => c.judge_verdict == null);
+if (missing.length && !process.argv.includes('--allow-incomplete')) {
+  console.error(`error: ${missing.length} P-claims have no judge verdict (incomplete fan-out).`);
+  console.error('first few:', missing.slice(0, 5).map((c) => c.id).join(', '));
+  console.error('pass --allow-incomplete to grade anyway (missing claims count as not-flagged).');
+  process.exit(2);
+}
+
+const r = computeBakeoff(joined, thresholds);
+
+const pct = (x) => (x == null ? 'n/a' : `${(x * 100).toFixed(1)}%`);
+const num = (x) => (x == null ? 'n/a' : x.toFixed(3));
+const band = (w) => (w == null ? 'n/a' : `[${pct(w.low)}, ${pct(w.high)}]`);
+
+function armRow(name, a) {
+  return `| ${name} | ${a.tp} | ${a.fp} | ${a.fn} | ${a.tn} | ${pct(a.precision)} | ${pct(a.recall)} | ${band(a.recallWilson)} | ${num(a.f1)} |`;
+}
+function typeRow(t, a) {
+  return `| ${t} | ${a.positives} | ${a.tp} | ${a.fp} | ${a.fn} | ${pct(a.precision)} | ${pct(a.recall)} |`;
+}
+
+const p = r.population;
+const g = r.gate;
+const ss = r.sourceSilent;
+const verdict = g.pass ? '✅ PASS — bygg S3' : '❌ FAIL — stopp, ikke bygg S3';
+
+const md = `# Judge bake-off-rapport — S1 (Fase 3 de-risk)
+
+_Generert deterministisk av \`run-judge-bakeoff.mjs\` over \`gold-correctness-set.json\` + \`judge-bakeoff-results.json\`. Tall fra testet \`lib/judge-bakeoff.mjs\`. Ikke rediger for hånd — regenerer._
+
+**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ ${minRecall}, presisjon ≥ ${minPrecision}, OG judge-recall > staleness-recall.
+
+## Evaluerings-populasjon (P)
+
+Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
+
+| metrikk | verdi |
+|---|---|
+| P totalt | ${p.total} |
+| Verifiserbare (correct/outdated/wrong) | ${p.verifiable} |
+| Positive (reelle feil å fange) | ${p.positives} |
+| Negative (correct) | ${p.negatives} |
+| Unsourced i P (kjørt, men utenfor P/R) | ${p.unsourcedInP} |
+
+## Arm-sammenligning (detektering over de ${p.verifiable} verifiserbare)
+
+| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
+|---|---|---|---|---|---|---|---|---|
+${armRow('staleness (billig baseline)', r.arms.staleness)}
+${armRow('judge (per-påstand groundedness)', r.arms.judge)}
+${armRow('hybrid (union)', r.arms.hybrid)}
+
+## Judge per claim_type (verifiserbar delmengde)
+
+| claim_type | positive | TP | FP | FN | presisjon | recall |
+|---|---|---|---|---|---|---|
+${Object.entries(r.byClaimType).sort((a, b) => b[1].positives - a[1].positives).map(([t, a]) => typeRow(t, a)).join('\n')}
+
+## source_silent-diagnostikk
+
+Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
+
+| signal | antall | tolkning |
+|---|---|---|
+| På verifiserbar feil | ${ss.onVerifiableError} | judge-bom: reell feil oversett via «kan ikke verifisere» |
+| På verifiserbar correct | ${ss.onVerifiableNegative} | judge reproduserte ikke et korrekt faktum mennesket fant |
+| Enig med unsourced | ${ss.agreesWithUnsourced} | judge reproduserer den uverifiserbare grensen (godt) |
+| Uenig med unsourced | ${ss.disagreesWithUnsourced} | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
+
+## GATE: ${verdict}
+
+- recall ${num(r.arms.judge.recall)} ≥ ${minRecall}? **${g.recallOk ? 'ja' : 'nei'}**
+- presisjon ${num(r.arms.judge.precision)} ≥ ${minPrecision}? **${g.precisionOk ? 'ja' : 'nei'}**
+- slår staleness (recall ${num(r.arms.staleness.recall)})? **${g.beatsStaleness ? 'ja' : 'nei'}**
+- begrunnelse: ${g.reasons.join('; ')}
+`;
+
+if (process.argv.includes('--write')) {
+  const jsonOut = path.join(DATA, 'judge-bakeoff-report.json');
+  const mdOut = path.join(DATA, 'judge-bakeoff-report.md');
+  fs.writeFileSync(
+    jsonOut,
+    JSON.stringify(
+      {
+        _meta: {
+          source: 'gold-correctness-set.json + judge-bakeoff-results.json',
+          thresholds,
+          judged: (judge.results || []).length,
+        },
+        ...r,
+      },
+      null,
+      2,
+    ) + '\n',
+  );
+  fs.writeFileSync(mdOut, md);
+  console.log(`wrote ${jsonOut}`);
+  console.log(`wrote ${mdOut}`);
+} else {
+  console.log(`P=${p.total} verifiable=${p.verifiable} positives=${p.positives}`);
+  console.log(`judge:     precision=${num(r.arms.judge.precision)} recall=${num(r.arms.judge.recall)} f1=${num(r.arms.judge.f1)}`);
+  console.log(`staleness: recall=${num(r.arms.staleness.recall)}`);
+  console.log(`GATE: ${g.pass ? 'PASS' : 'FAIL'} — ${g.reasons.join('; ')}`);
+  console.log('(dry run — pass --write to persist judge-bakeoff-report.json + .md)');
+}

tests/kb-eval/test-judge-bakeoff.test.mjs

@@ -0,0 +1,199 @@
+// tests/kb-eval/test-judge-bakeoff.test.mjs
+// Unit tests for the S1 judge bake-off aggregation lib.
+//
+// S1 de-risks the Fase 3 correctness judge BEFORE any production scaffolding is
+// built: run a per-claim groundedness judge against the frozen 373-claim gold set
+// and ask whether it beats the cheap staleness baseline (recall 0/40) by enough to
+// justify ~2700 non-batchable fetches per full pass.
+//
+// The eval population P is volatile + fetchable claim_types (price excluded) — the
+// judge is measured exactly where the real errors live, which structurally avoids
+// the "inverted leverage" trap (a judge that wins only by auto-scoring cheap stable
+// claims). Grading is a pure detection confusion matrix vs the gold verdicts:
+//   gold positive  = verdict ∈ {outdated, wrong}   (a real error to catch)
+//   gold negative  = verdict === 'correct'
+//   excluded       = verdict === 'unsourced'        (no ground-truth value)
+//   predicted positive (flag) = judge_verdict === 'not_grounded'
+// All computation is deterministic — no I/O, no Date.now/Math.random.
+
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import {
+  evalPopulation,
+  blindClaims,
+  gradeArm,
+  stalenessFlag,
+  judgeFlag,
+  hybridFlag,
+  gateDecision,
+  computeBakeoff,
+} from '../../scripts/kb-eval/lib/judge-bakeoff.mjs';
+
+// --- synthetic joined gold+judge set ----------------------------------------
+// Each claim carries its gold fields plus the (blind) judge_verdict the harness
+// joined back by id. Comments give the intended confusion-matrix cell.
+const J = [
+  // 1: volatile/version, correct, judge grounded            -> P, verifiable, TN
+  { id: 'a#1', skill: 'x', stratum: 'volatile', claim_type: 'version', verdict: 'correct', lastmod_changed: false, judge_verdict: 'grounded' },
+  // 2: volatile/sku, outdated, judge not_grounded           -> P, TP (judge catches)
+  { id: 'a#2', skill: 'x', stratum: 'volatile', claim_type: 'sku', verdict: 'outdated', lastmod_changed: false, judge_verdict: 'not_grounded' },
+  // 3: volatile/tpm, wrong, judge grounded                  -> P, FN (judge misses)
+  { id: 'a#3', skill: 'x', stratum: 'volatile', claim_type: 'tpm', verdict: 'wrong', lastmod_changed: false, judge_verdict: 'grounded' },
+  // 4: volatile/status, correct, judge not_grounded         -> P, FP (false alarm)
+  { id: 'a#4', skill: 'x', stratum: 'volatile', claim_type: 'status', verdict: 'correct', lastmod_changed: false, judge_verdict: 'not_grounded' },
+  // 5: volatile/region, outdated, lastmod TRUE, judge ng    -> P, TP; staleness also catches
+  { id: 'a#5', skill: 'x', stratum: 'volatile', claim_type: 'region', verdict: 'outdated', lastmod_changed: true, judge_verdict: 'not_grounded' },
+  // 6: volatile/PRICE, wrong                                -> excluded from P (price)
+  { id: 'a#6', skill: 'x', stratum: 'volatile', claim_type: 'price', verdict: 'wrong', lastmod_changed: false, judge_verdict: 'not_grounded' },
+  // 7: CONTROL/taxonomy, wrong                              -> excluded from P (control stratum)
+  { id: 'a#7', skill: 'x', stratum: 'control', claim_type: 'taxonomy', verdict: 'wrong', lastmod_changed: false, judge_verdict: 'not_grounded' },
+  // 8: volatile/status, unsourced, judge source_silent      -> P but not verifiable (excluded from P/R)
+  { id: 'a#8', skill: 'x', stratum: 'volatile', claim_type: 'status', verdict: 'unsourced', lastmod_changed: false, judge_verdict: 'source_silent' },
+  // 9: volatile/taxonomy, correct, judge source_silent      -> P, verifiable negative, not flagged -> TN
+  { id: 'a#9', skill: 'x', stratum: 'volatile', claim_type: 'taxonomy', verdict: 'correct', lastmod_changed: false, judge_verdict: 'source_silent' },
+];
+
+// --- evalPopulation ----------------------------------------------------------
+test('evalPopulation — keeps volatile + fetchable, drops price and control', () => {
+  const P = evalPopulation(J);
+  const ids = P.map((c) => c.id).sort();
+  assert.deepEqual(ids, ['a#1', 'a#2', 'a#3', 'a#4', 'a#5', 'a#8', 'a#9']);
+});
+
+// --- blindClaims (no label leakage) -----------------------------------------
+test('blindClaims — emits all of P with only the allowed blind fields', () => {
+  const blind = blindClaims(J);
+  assert.equal(blind.length, 7); // all of P, unsourced included
+  const allowed = new Set(['id', 'file', 'skill', 'claim', 'claim_type', 'evidence_url']);
+  for (const c of blind) {
+    for (const k of Object.keys(c)) {
+      assert.ok(allowed.has(k), `leaked field: ${k}`);
+    }
+    // the answer key must never appear
+    assert.equal(c.verdict, undefined);
+    assert.equal(c.notes, undefined);
+    assert.equal(c.lastmod_changed, undefined);
+    assert.equal(c.stratum, undefined);
+  }
+});
+
+// --- flag predicates ---------------------------------------------------------
+test('stalenessFlag — true only when lastmod_changed === true', () => {
+  assert.equal(stalenessFlag({ lastmod_changed: true }), true);
+  assert.equal(stalenessFlag({ lastmod_changed: false }), false);
+  assert.equal(stalenessFlag({ lastmod_changed: null }), false);
+});
+test('judgeFlag — true only when judge_verdict === not_grounded', () => {
+  assert.equal(judgeFlag({ judge_verdict: 'not_grounded' }), true);
+  assert.equal(judgeFlag({ judge_verdict: 'grounded' }), false);
+  assert.equal(judgeFlag({ judge_verdict: 'source_silent' }), false);
+});
+test('hybridFlag — union of staleness and judge', () => {
+  assert.equal(hybridFlag({ lastmod_changed: true, judge_verdict: 'grounded' }), true);
+  assert.equal(hybridFlag({ lastmod_changed: false, judge_verdict: 'not_grounded' }), true);
+  assert.equal(hybridFlag({ lastmod_changed: false, judge_verdict: 'source_silent' }), false);
+});
+
+// --- gradeArm ----------------------------------------------------------------
+// Verifiable subset of P = ids 1,2,3,4,5,9 (8 excluded: unsourced).
+function verifiableP() {
+  return evalPopulation(J).filter((c) => c.verdict !== 'unsourced');
+}
+
+test('gradeArm — judge arm confusion matrix', () => {
+  const a = gradeArm(verifiableP(), judgeFlag);
+  // positives = outdated/wrong = {2,3,5}=3 ; negatives = correct = {1,4,9}=3
+  assert.equal(a.positives, 3);
+  assert.equal(a.negatives, 3);
+  // flags: 2(TP),4(FP),5(TP) ; misses: 3(FN); correct-rejects: 1,9 (TN)
+  assert.equal(a.tp, 2);
+  assert.equal(a.fp, 1);
+  assert.equal(a.fn, 1);
+  assert.equal(a.tn, 2);
+  assert.ok(Math.abs(a.precision - 2 / 3) < 1e-9);
+  assert.ok(Math.abs(a.recall - 2 / 3) < 1e-9);
+  assert.ok(Math.abs(a.f1 - 2 / 3) < 1e-9);
+});
+
+test('gradeArm — staleness arm catches only the lastmod-changed error', () => {
+  const a = gradeArm(verifiableP(), stalenessFlag);
+  // only id 5 has lastmod true (a positive) -> tp=1; positives 2,3 missed -> fn=2
+  assert.equal(a.tp, 1);
+  assert.equal(a.fp, 0);
+  assert.equal(a.fn, 2);
+  assert.equal(a.tn, 3);
+  assert.ok(Math.abs(a.recall - 1 / 3) < 1e-9);
+  assert.equal(a.precision, 1); // 1 flag, 1 right
+});
+
+test('gradeArm — precision is null when nothing is flagged', () => {
+  const noFlag = gradeArm(verifiableP(), () => false);
+  assert.equal(noFlag.tp, 0);
+  assert.equal(noFlag.fp, 0);
+  assert.equal(noFlag.precision, null); // 0/0 -> null, not NaN
+  assert.equal(noFlag.recall, 0);
+  assert.equal(noFlag.f1, null);
+  assert.equal(noFlag.precisionWilson, null); // no flags -> no band
+});
+
+test('gradeArm — Wilson bands bracket the point estimates', () => {
+  const a = gradeArm(verifiableP(), judgeFlag); // recall .667 (2/3), precision .667 (2/3)
+  assert.ok(a.recallWilson.low <= a.recall && a.recall <= a.recallWilson.high);
+  assert.ok(a.precisionWilson.low <= a.precision && a.precision <= a.precisionWilson.high);
+});
+
+// --- gateDecision ------------------------------------------------------------
+test('gateDecision — passes when judge clears both thresholds and beats staleness', () => {
+  const judge = gradeArm(verifiableP(), judgeFlag); // recall .667 prec .667
+  const stale = gradeArm(verifiableP(), stalenessFlag); // recall .333
+  const g = gateDecision(judge, stale, { minRecall: 0.6, minPrecision: 0.6 });
+  assert.equal(g.pass, true);
+  assert.equal(g.beatsStaleness, true);
+  assert.equal(g.recallOk, true);
+  assert.equal(g.precisionOk, true);
+});
+
+test('gateDecision — fails when recall below threshold', () => {
+  const judge = gradeArm(verifiableP(), judgeFlag); // recall .667
+  const stale = gradeArm(verifiableP(), stalenessFlag);
+  const g = gateDecision(judge, stale, { minRecall: 0.7, minPrecision: 0.6 });
+  assert.equal(g.pass, false);
+  assert.equal(g.recallOk, false);
+  assert.equal(g.precisionOk, true);
+  assert.ok(Array.isArray(g.reasons) && g.reasons.length >= 1);
+});
+
+test('gateDecision — beatsStaleness requires strictly greater recall', () => {
+  const judge = gradeArm(verifiableP(), judgeFlag); // recall .667
+  // staleness with identical recall -> not strictly greater
+  const fakeStale = { recall: 0.667, precision: 1, tp: 2, fp: 0, fn: 1, tn: 2, positives: 3, negatives: 3 };
+  const g = gateDecision(judge, fakeStale, { minRecall: 0.6, minPrecision: 0.6 });
+  assert.equal(g.beatsStaleness, false);
+  assert.equal(g.pass, false);
+});
+
+// --- computeBakeoff (top-level) ---------------------------------------------
+test('computeBakeoff — wires population, three arms, diagnostics and gate', () => {
+  const r = computeBakeoff(J, { minRecall: 0.6, minPrecision: 0.6 });
+  assert.equal(r.population.total, 7); // P
+  assert.equal(r.population.verifiable, 6);
+  assert.equal(r.population.positives, 3);
+  assert.equal(r.population.unsourcedInP, 1); // id 8
+  // three arms present
+  assert.ok(r.arms.judge && r.arms.staleness && r.arms.hybrid);
+  assert.equal(r.arms.judge.tp, 2);
+  // hybrid = union; here same recall as judge (staleness adds the already-caught #5)
+  assert.equal(r.arms.hybrid.tp, 2);
+  // source_silent diagnostics
+  assert.equal(r.sourceSilent.onVerifiableNegative, 1); // id 9 correct but judge couldn't verify
+  assert.equal(r.sourceSilent.agreesWithUnsourced, 1);  // id 8 unsourced, judge source_silent
+  // gate
+  assert.equal(r.gate.pass, true);
+});
+
+test('computeBakeoff — per claim_type judge breakdown present', () => {
+  const r = computeBakeoff(J, { minRecall: 0.6, minPrecision: 0.6 });
+  // version/sku/tpm/status/region/taxonomy each appear in verifiable P
+  assert.ok(r.byClaimType.sku);
+  assert.equal(r.byClaimType.sku.tp, 1); // id 2 outdated sku, judge ng
+});