feat(ms-ai-architect): Sesjon 3 - decision-ledger (lag 2) + discovery-dedup

Lukker discovery-løkken gjennom operatør-gaten. data/decisions.json er eneste
skrive-autoriserte bro mellom deteksjon og KB/registry; discovery LESER den og
utelater alt operatøren har tatt stilling til.

Dedup-policy = A (operatør-valg): isDecided = enhver ledger-entry (approved/
rejected/pending). Kun helt fraværende URLer re-foreslås.

- lib/decisions-io.mjs: createLedger/load/save(atomisk)/isDecided/recordDecision
  (ren)/filterUndecided. TDD: 10 tester før kode.
- discover-new-urls.mjs leser ledger, filtrerer, rapporterer deduped_by_ledger.
  Importerer ALDRI write-utils (invariant verifisert, 6 guard-tester).
- Gate dokumentert i kb-update.md §3b (eneste skrivevei).
- decisions.json tracket via gitignore-negasjon (som domain-taxonomy.json).

Kriterium møtt: dedup-diff (rejected re-foreslås ikke runde 2).
Tester: validate 239 · kb-update 82 (+16) · kb-eval 13 · kb-integrity 115/115.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01REiKFhP4w6xGXXqWKpPCJJ

.gitignore

@@ -23,9 +23,10 @@ node_modules/
 .work/
 org/
 # Generated KB-update artifacts (registry, reports) are ignored, but the
-# hand-authored taxonomy (single source of truth, lag 0) is tracked.
+# hand-authored taxonomy (lag 0) and the decision ledger (lag 2) are tracked.
 scripts/kb-update/data/*
 !scripts/kb-update/data/domain-taxonomy.json
+!scripts/kb-update/data/decisions.json
 .kb-backup/
 .rollback-in-progress
 

commands/kb-update.md

@@ -13,7 +13,7 @@ Holder Microsoft AI-kunnskapsbasen i `skills/*/references/` ferskt ved å sammen
 ## Hva kommandoen gjør
 
 1. **Polle sitemaps:** kjører `node scripts/kb-update/run-weekly-update.mjs --force` for å hente fersk `<lastmod>` for hver Microsoft Learn-URL i registeret
-2. **Optional discovery:** med default `--discover` finner nye relevante URLer i sitemap som ikke er i registeret (`scripts/kb-update/discover-new-urls.mjs --limit 500`)
+2. **Optional discovery:** med default `--discover` finner nye relevante URLer i sitemap som ikke er i registeret (`scripts/kb-update/discover-new-urls.mjs --limit 500`). Discovery **leser beslutnings-ledgeren** (`data/decisions.json`, lag 2) og utelater alt operatøren allerede har tatt stilling til (policy A: approved/rejected/pending) — så de samme kandidatene drukner ikke gaten kjøring etter kjøring
 3. **Generere endringsrapport:** `report-changes.mjs` produserer `data/change-report.json` med per-fil prioritering (critical/high/medium/low) basert på antall endrede kilder + alder på lokal fil
 4. **Vise rapporten:** lese rapport, presentere oppsummering til bruker, vente på `go`
 5. **Oppdatere filer:** for hver fil i valgt prioritetsbøtte (default: critical + high):
@@ -63,6 +63,30 @@ Presenter til bruker:
 
 Hvis `--dry-run`: stopp her, ikke oppdater filer.
 
+### 3b. Discovery-gate — nye URLer via decision-ledger (lag 2)
+
+Kjørte discovery (default), lukk løkken her. Dette er **operatør-gaten** — det eneste stedet beslutninger om nye URLer skrives.
+
+a. **Les kandidatene:** `data/discovery-report.json` → `candidates[]`. Listen er allerede dedupet av scriptet mot både registry og `data/decisions.json`, så alt her er genuint ubesluttet. Feltet `deduped_by_ledger` viser hvor mange tidligere beslutninger som ble utelatt.
+
+b. **Presenter for operatør**, gruppert per `suggested_skill` (vis `url`, `suggested_category`, `lastmod`). For hver kandidat (eller batch): **approve** (ta inn i KB), **reject** (irrelevant — ikke vis igjen), eller **utsett** (`pending` — ikke avgjort, vises i pending-bøtta men re-foreslås ikke).
+
+c. **Skriv beslutningene til ledgeren — ENESTE skrivevei.** Bruk `lib/decisions-io.mjs` (ikke håndskriv JSON):
+
+```
+import { loadDecisions, recordDecision, saveDecisions } from './scripts/kb-update/lib/decisions-io.mjs';
+let led = loadDecisions();
+led = recordDecision(led, url, { status: 'rejected', decided_at: '<i dag>', suggested_skill, suggested_category, note });
+// ...én recordDecision per beslutning...
+saveDecisions(led);
+```
+
+`decided_at` settes til dagens dato (caller-injisert — `recordDecision` er ren). `decisions.json` er tracket i git (overlever, til forskjell fra de genererte rapportene).
+
+d. **For `approved`:** registrér URLen i `url-registry` (gated) så den fanges av polling heretter — via `lib/registry-io.mjs` `saveRegistry`. Bruk `suggested_skill`/`suggested_category` fra kandidaten. Deretter hentes innholdet og en KB-fil skrives i transformasjonslaget (lag 4, Sesjon 5) — for nå holder det å registrere + flagge for henting.
+
+e. **Invariant:** `discover-new-urls.mjs` (deteksjon) skriver **aldri** ledgeren — den kun leser. Bare denne gaten skriver. Verifisert av `tests/kb-update/test-discover-invariant.test.mjs`.
+
 ### 4. Per-fil oppdatering (etter brukerens `y`)
 
 For hver fil i valgte prioriteter:

scripts/kb-update/data/decisions.json

@@ -0,0 +1,5 @@
+{
+  "version": 1,
+  "updated_at": null,
+  "decisions": {}
+}

scripts/kb-update/discover-new-urls.mjs

@@ -9,6 +9,9 @@ import { normalizeUrl } from './lib/url-normalize.mjs';
 import { loadRegistry, saveReport } from './lib/registry-io.mjs';
 import { streamSitemap, fetchSitemapIndex } from './lib/sitemap-stream.mjs';
 import { loadTaxonomy, getSitemapPrefixes, makeClassifier } from './lib/taxonomy.mjs';
+// Read-only: detection filters against the ledger but NEVER writes it (arkitektur-
+// invariant — no saveDecisions / registry / atomic-write / backup imports here).
+import { loadDecisions, isDecided } from './lib/decisions-io.mjs';
 
 const __dirname = dirname(fileURLToPath(import.meta.url));
 const DATA_DIR = join(__dirname, 'data');
@@ -33,6 +36,13 @@ async function main() {
   const knownUrls = new Set(Object.keys(registry.urls));
   console.log(`Registry: ${knownUrls.size} known URLs`);
 
+  // Decision ledger (lag 2): skip anything the operator already decided on
+  // (policy A — approved | rejected | pending all dedup). Read-only here.
+  const decisions = loadDecisions(DATA_DIR);
+  const decidedCount = Object.keys(decisions.decisions || {}).length;
+  console.log(`Decision ledger: ${decidedCount} decided URLs (excluded from re-proposal)`);
+  let dedupedByLedger = 0;
+
   console.log('Fetching sitemap index...');
   const indexEntries = await fetchSitemapIndex();
 
@@ -55,6 +65,7 @@ async function main() {
       for await (const entry of streamSitemap(child.loc)) {
         const normalized = normalizeUrl(entry.loc);
         if (!normalized || knownUrls.has(normalized)) continue;
+        if (isDecided(decisions, normalized)) { dedupedByLedger++; continue; }
 
         const classification = classifyUrl(normalized);
         if (!classification) continue;
@@ -89,6 +100,7 @@ async function main() {
   const report = {
     generated_at: new Date().toISOString().split('T')[0],
     new_candidates: candidates.length,
+    deduped_by_ledger: dedupedByLedger,
     by_suggested_skill: bySkill,
     candidates,
   };
@@ -97,6 +109,7 @@ async function main() {
 
   console.log(`\n=== Discovery Report ===`);
   console.log(`New relevant URLs found: ${candidates.length}`);
+  console.log(`Excluded by decision ledger: ${dedupedByLedger}`);
   console.log('By skill:', JSON.stringify(bySkill, null, 2));
   if (candidates.length > 0) {
     console.log('\nNewest 10:');

scripts/kb-update/lib/decisions-io.mjs

@@ -0,0 +1,86 @@
+// decisions-io.mjs — Read/write + pure helpers for data/decisions.json (lag 2).
+// The decision ledger is the ONLY write-authorized bridge between detection and
+// the KB/registry: discovery READS it to filter, the operator gate WRITES it.
+// Zero dependencies. Atomic writes via .tmp + rename (mirrors registry-io).
+//
+// Dedup policy = A (operatør 2026-06-19): isDecided(url) is true for ANY ledger
+// entry (pending | approved | rejected). Discovery re-proposes only URLs that
+// are entirely absent from the ledger; everything the operator has touched —
+// approved, rejected, or explicitly deferred (pending) — stays off the list.
+
+import { readFileSync, writeFileSync, renameSync, existsSync, mkdirSync } from 'node:fs';
+import { join, dirname } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const DEFAULT_DATA_DIR = join(__dirname, '..', 'data');
+
+/**
+ * Empty ledger scaffold.
+ * @returns {{version: number, updated_at: string|null, decisions: object}}
+ */
+export function createLedger() {
+  return { version: 1, updated_at: null, decisions: {} };
+}
+
+/**
+ * Load the decision ledger from disk.
+ * @param {string} [dataDir] — defaults to ../data/ relative to lib/
+ * @returns {object} parsed ledger or empty scaffold
+ */
+export function loadDecisions(dataDir = DEFAULT_DATA_DIR) {
+  const path = join(dataDir, 'decisions.json');
+  if (!existsSync(path)) return createLedger();
+  return JSON.parse(readFileSync(path, 'utf8'));
+}
+
+/**
+ * Save the decision ledger atomically (write to .tmp, then rename).
+ * This is the gate's write path — detection scripts must NOT import it.
+ * @param {object} ledger
+ * @param {string} [dataDir]
+ */
+export function saveDecisions(ledger, dataDir = DEFAULT_DATA_DIR) {
+  if (!existsSync(dataDir)) mkdirSync(dataDir, { recursive: true });
+  const path = join(dataDir, 'decisions.json');
+  const tmp = path + '.tmp';
+  writeFileSync(tmp, JSON.stringify(ledger, null, 2) + '\n', 'utf8');
+  renameSync(tmp, path);
+}
+
+/**
+ * Has the operator already decided on this URL? Policy A: any entry counts.
+ * @param {object} ledger
+ * @param {string} url — normalized URL
+ * @returns {boolean}
+ */
+export function isDecided(ledger, url) {
+  return Boolean(ledger.decisions && ledger.decisions[url]);
+}
+
+/**
+ * Record a decision for a URL. Pure — returns a new ledger, does not mutate.
+ * @param {object} ledger
+ * @param {string} url — normalized URL
+ * @param {{status: string, decided_at?: string, suggested_skill?: string|null,
+ *          suggested_category?: string, note?: string}} decision
+ * @returns {object} new ledger
+ */
+export function recordDecision(ledger, url, decision) {
+  return {
+    ...ledger,
+    updated_at: decision.decided_at ?? ledger.updated_at,
+    decisions: { ...ledger.decisions, [url]: { ...decision } },
+  };
+}
+
+/**
+ * Drop candidates the operator has already decided on (policy A).
+ * Discovery applies this before emitting its report.
+ * @param {object} ledger
+ * @param {Array<{url: string}>} candidates
+ * @returns {Array<{url: string}>} candidates with no ledger entry
+ */
+export function filterUndecided(ledger, candidates) {
+  return candidates.filter((c) => !isDecided(ledger, c.url));
+}

tests/kb-update/test-decisions-io.test.mjs

@@ -0,0 +1,148 @@
+// tests/kb-update/test-decisions-io.test.mjs
+// Unit tests for scripts/kb-update/lib/decisions-io.mjs — the lag-2 decision
+// ledger (data/decisions.json), the ONLY write-authorized bridge between
+// detection and the KB/registry. Dedup policy = A (operatør 2026-06-19):
+// isDecided(url) is true for ANY ledger entry (pending | approved | rejected),
+// so discovery re-proposes only URLs entirely absent from the ledger.
+
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { mkdtempSync, rmSync, readFileSync, existsSync, readdirSync } from 'node:fs';
+import { tmpdir } from 'node:os';
+import { join } from 'node:path';
+import {
+  createLedger,
+  loadDecisions,
+  saveDecisions,
+  isDecided,
+  recordDecision,
+  filterUndecided,
+} from '../../scripts/kb-update/lib/decisions-io.mjs';
+
+function withTmp(fn) {
+  const dir = mkdtempSync(join(tmpdir(), 'dec-test-'));
+  try {
+    return fn(dir);
+  } finally {
+    rmSync(dir, { recursive: true, force: true });
+  }
+}
+
+const U1 = 'https://learn.microsoft.com/azure/foundry/concepts/built-in-evaluators';
+const U2 = 'https://learn.microsoft.com/azure/databricks/sql/language-manual/sql-ref-sharing';
+const U3 = 'https://learn.microsoft.com/azure/ai-services/openai/whats-new';
+
+// --- scaffold ---
+
+test('createLedger — empty scaffold shape', () => {
+  const led = createLedger();
+  assert.equal(led.version, 1);
+  assert.equal(led.updated_at, null);
+  assert.deepEqual(led.decisions, {});
+});
+
+// --- load / save (atomic) ---
+
+test('loadDecisions — missing file returns empty scaffold', () => {
+  withTmp((dir) => {
+    const led = loadDecisions(dir);
+    assert.equal(led.version, 1);
+    assert.deepEqual(led.decisions, {});
+  });
+});
+
+test('saveDecisions + loadDecisions — round-trips', () => {
+  withTmp((dir) => {
+    let led = createLedger();
+    led = recordDecision(led, U1, {
+      status: 'rejected',
+      decided_at: '2026-06-19',
+      suggested_skill: 'ms-ai-engineering',
+      suggested_category: 'data-engineering',
+    });
+    saveDecisions(led, dir);
+    const back = loadDecisions(dir);
+    assert.deepEqual(back, led);
+  });
+});
+
+test('saveDecisions — leaves no .tmp orphan and writes decisions.json', () => {
+  withTmp((dir) => {
+    saveDecisions(createLedger(), dir);
+    const files = readdirSync(dir);
+    assert.ok(existsSync(join(dir, 'decisions.json')));
+    assert.deepEqual(files.filter((f) => f.includes('.tmp')), []);
+  });
+});
+
+// --- isDecided — DEDUP POLICY A ---
+
+test('isDecided — true for ANY entry status (pending | approved | rejected)', () => {
+  let led = createLedger();
+  led = recordDecision(led, U1, { status: 'rejected', decided_at: '2026-06-19' });
+  led = recordDecision(led, U2, { status: 'approved', decided_at: '2026-06-19' });
+  led = recordDecision(led, U3, { status: 'pending', decided_at: '2026-06-19' });
+  assert.equal(isDecided(led, U1), true, 'rejected must dedup');
+  assert.equal(isDecided(led, U2), true, 'approved must dedup (policy A)');
+  assert.equal(isDecided(led, U3), true, 'pending must dedup (policy A)');
+});
+
+test('isDecided — false for URL absent from ledger', () => {
+  const led = createLedger();
+  assert.equal(isDecided(led, U1), false);
+});
+
+// --- recordDecision — pure ---
+
+test('recordDecision — sets entry, returns new ledger, does not mutate input', () => {
+  const led = createLedger();
+  const next = recordDecision(led, U1, {
+    status: 'rejected',
+    decided_at: '2026-06-19',
+    suggested_skill: 'ms-ai-engineering',
+    suggested_category: 'data-engineering',
+    note: 'off-domain databricks page',
+  });
+  assert.equal(next.decisions[U1].status, 'rejected');
+  assert.equal(next.decisions[U1].note, 'off-domain databricks page');
+  // input untouched
+  assert.deepEqual(led.decisions, {});
+  assert.notEqual(next, led);
+});
+
+test('recordDecision — advances updated_at to the decision date', () => {
+  const led = createLedger();
+  const next = recordDecision(led, U1, { status: 'approved', decided_at: '2026-06-19' });
+  assert.equal(next.updated_at, '2026-06-19');
+});
+
+// --- filterUndecided — the dedup gate discovery applies ---
+
+test('filterUndecided — drops decided candidates (any status), keeps fresh ones', () => {
+  let led = createLedger();
+  led = recordDecision(led, U1, { status: 'rejected', decided_at: '2026-06-19' });
+  led = recordDecision(led, U2, { status: 'approved', decided_at: '2026-06-19' });
+  const candidates = [
+    { url: U1, suggested_skill: 'ms-ai-engineering' }, // rejected → drop
+    { url: U2, suggested_skill: 'ms-ai-engineering' }, // approved → drop
+    { url: U3, suggested_skill: 'ms-ai-engineering' }, // fresh → keep
+  ];
+  const out = filterUndecided(led, candidates);
+  assert.deepEqual(out.map((c) => c.url), [U3]);
+});
+
+// --- ACCEPTANCE CRITERION (roadmap §165): discovery 2x does not re-propose a rejected URL ---
+
+test('dedup-diff — a rejected URL is re-proposed in round 1 but NOT in round 2', () => {
+  // Round 1: empty ledger — the URL is a fresh candidate, passes the filter.
+  let led = createLedger();
+  const round1 = filterUndecided(led, [{ url: U1, suggested_skill: 'ms-ai-engineering' }]);
+  assert.deepEqual(round1.map((c) => c.url), [U1], 'round 1 should surface the new URL');
+
+  // Operatør rejects it through the gate — the ONLY write path.
+  led = recordDecision(led, U1, { status: 'rejected', decided_at: '2026-06-19' });
+
+  // Round 2: same discovery output, now the rejected URL must be filtered out.
+  const round2 = filterUndecided(led, [{ url: U1, suggested_skill: 'ms-ai-engineering' }]);
+  assert.deepEqual(round2, [], 'round 2 must NOT re-propose the rejected URL');
+});

tests/kb-update/test-discover-invariant.test.mjs

@@ -0,0 +1,44 @@
+// tests/kb-update/test-discover-invariant.test.mjs
+// Architecture-invariant guard (STATE.md §83 / roadmap §159): the detection
+// script discover-new-urls.mjs must NEVER import write-utils. It only READS the
+// decision ledger to filter; all writes go through the operator gate. saveReport
+// (its own discovery-report.json) is allowed — that is detection's own output,
+// not a KB/registry mutation.
+
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { readFileSync } from 'node:fs';
+import { dirname, join } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const __dirname = dirname(fileURLToPath(import.meta.url));
+const DISCOVER = join(__dirname, '..', '..', 'scripts', 'kb-update', 'discover-new-urls.mjs');
+const src = readFileSync(DISCOVER, 'utf8');
+
+// Sjekkene er IMPORT-spesifikke med vilje: en invariant-guard som brytes av en
+// kommentar som *beskriver* invarianten er verdiløs. Vi gransker import-linjer.
+const importLines = src.split('\n').filter((l) => /^\s*import\b/.test(l)).join('\n');
+
+test('discover does NOT import the ledger write path (saveDecisions)', () => {
+  assert.doesNotMatch(importLines, /\bsaveDecisions\b/);
+});
+
+test('discover does NOT import the registry write path (saveRegistry)', () => {
+  assert.doesNotMatch(importLines, /\bsaveRegistry\b/);
+});
+
+test('discover does NOT import atomic-write', () => {
+  assert.doesNotMatch(importLines, /from\s+'[^']*atomic-write[^']*'/);
+});
+
+test('discover does NOT import backup', () => {
+  assert.doesNotMatch(importLines, /from\s+'[^']*\/backup\.mjs'/);
+});
+
+test('discover DOES read the ledger (loadDecisions from decisions-io)', () => {
+  assert.match(src, /import \{[^}]*\bloadDecisions\b[^}]*\} from '\.\/lib\/decisions-io\.mjs'/);
+});
+
+test('discover actually applies the dedup filter (isDecided)', () => {
+  assert.match(src, /isDecided\(/);
+});