diff --git a/docs/ref-kb-audit-2026-06.md b/docs/ref-kb-audit-2026-06.md new file mode 100644 index 0000000..bb594a7 --- /dev/null +++ b/docs/ref-kb-audit-2026-06.md @@ -0,0 +1,37 @@ +# Reference-KB audit — verifisert ground truth (2026-06-26) + +_Read-only audit av de 389 ref-filene under `skills//references/**/*.md`. Reproduserbar via `python3 scripts/kb-eval/ref-file-audit.py`. Utløst av spørsmålet «måler vi kvaliteten på ref-filene, og brukes de best mulig?». Beslutningsnotatet som velger retning lever separat (se nederst)._ + +## Bakgrunn +Skill-kvalitetsscoringen (Spor D, `scripts/kb-eval/`) scorer SKILL.md-forfatterkvalitet + ref-filenes STRUKTUR/hygiene — ikke substansiell innholdskorrekthet mot MS Learn. KB-refresh (`scripts/kb-update/`) flagger staleness, men gir ingen score. Gapet — «stemmer ref-innholdet mot MS Learn i dag» — måles ikke. Denne auditen kartla ref-filenes faktiske tilstand før vi velger hvordan gapet skal lukkes. + +## Selvkorreksjoner (premiss-verifisering fanget to artefakter i en tidligere kjøring) +- **«220 orphans» → ekte: 0.** Den første heuristikken testet kun om filnavnet var navngitt i en hub. Men 220 filer nås via **mappe-referanse** (progressive disclosure — K5 named-ratio-mål er bare 0,2). Folder-bevisst telling gir 0 ekte orphans. KB-en bærer ingen død vekt. +- **«136 datoløse» → ekte: 6.** Heuristikken krevde full `YYYY-MM-DD`. 130 filer har bevisst **måned-presisjon** (`YYYY-MM`), som er konvensjonen `report-changes.mjs` forutsetter. Kun 6 er ekte datoløse. + +## Verifiserte funn + +| Dimensjon | Tall | Vurdering | +|-----------|------|-----------| +| **Inventar** | 389 filer: advisor 62, engineering 153, governance 78, infrastructure 34, security 62 | — | +| **Størrelse** | median 481 linjer, snitt 507, **183/389 >500 linjer**, største 1265 (`adr-template.md`); kun 2 filer ≤100 | KB-en er nesten utelukkende store filer. Granularitets-spørsmål (se notat). | +| **Dato** | 253 dag-presise · 130 måned-presise · **6 ekte datoløse** | De 6 (4 er AI Act-filer) bør stemples — kort, høyverdi fiks. | +| **Reachability** | 169 navngitt · 220 via mappe · **0 ekte orphans** | Ingen død vekt. Mappe-referanse er den dominerende lastemekanismen. | +| **Kilde-URI** | **83 filer (21 %) har ingen MS Learn/docs-URL** | Noen legitimt kildeløse (maler/metodikk); andre gjør MS-påstander uten sporbar kilde → ikke auto-verifiserbare. | +| **Metadata** | **34 distinkte prosa-header-nøkler**, 0 YAML-frontmatter (`Category` ×322 vs `Kategori` ×42; `Last updated` vs `Sist oppdatert` vs `Dato` vs `Oppdatert`) | Fragmentert → skjør detektor, ingen maskinlesbar kilde-URI for en korrekthets-judge. | +| **Topologi** | flatt tre; N3 forbyr ref→ref-lenker; kun 2/389 har .md-kryss-lenke | Bevisst — progressive disclosure, ikke en graf. | +| **TOC** | 384/389 store filer uten TOC (N4) | Reell, men lavvekt — skills står på 91–96 likevel. Polish. | + +## Eksisterende Spor D-scorer (kontekst) +advisor 91 · engineering 96 · governance 96 · infrastructure 96 · security 96. **Null under mål (90).** Struktur/forfatterkvalitet er altså ikke problemet — innholdskorrekthet er den umålte aksen. + +## De 6 ekte datoløse (quick-fix-kandidater) +- `ms-ai-advisor/references/architecture/decision-trees.md` +- `ms-ai-governance/references/monitoring-observability/anomaly-detection-ai-systems.md` +- `ms-ai-governance/references/responsible-ai/ai-act-classification-methodology.md` +- `ms-ai-governance/references/responsible-ai/ai-act-deployer-obligations.md` +- `ms-ai-governance/references/responsible-ai/ai-act-fria-template.md` +- `ms-ai-governance/references/responsible-ai/ai-act-provider-obligations.md` + +## Neste steg +Et faktabasert, best-practice-forankret beslutningsnotat (5 akser: struktur/størrelse · innhold/korrekthet · kvalitetsmåling · MS Learn-fetch-dekning · metadata-substrat) avgjør retning før noen av de 389 filene endres. Se `docs/ref-kb-direction-note-2026-06.md` (genereres). Bakgrunn for gapet: `docs/kb-refresh-backlog-2026-06.md` («Separate spor»). diff --git a/scripts/kb-eval/ref-file-audit.py b/scripts/kb-eval/ref-file-audit.py new file mode 100644 index 0000000..8f36f44 --- /dev/null +++ b/scripts/kb-eval/ref-file-audit.py @@ -0,0 +1,109 @@ +#!/usr/bin/env python3 +"""ref-file-audit.py — read-only structural/usage audit of the reference KB. + +Reports, with verified ground truth, on the ~389 files under +skills//references/**/*.md across six dimensions: + + 1. Inventory — count per skill. + 2. Size — line distribution (median/mean/max + buckets + biggest). + 3. Date hygiene — day-precise (YYYY-MM-DD) vs month-only (YYYY-MM) vs truly + dateless. Month-only is a deliberate convention + (report-changes.mjs parses YYYY-MM as -01), NOT a defect. + 4. Reachability — named by basename in a SKILL.md/agent, reachable via FOLDER + reference (progressive disclosure), or a true orphan. + Folder-awareness matters: K5 named-ratio target is only 0.2, + so most files are reached by folder, not by name. + 5. Source URI — does the file cite any learn/docs.microsoft.com URL. + 6. Header keys — variant count (prose-header fragmentation; 0 use YAML). + +Pairs with the Spor D scorer (score-skill.mjs), which covers SKILL.md authoring +quality + structural ref hygiene but NOT substantive content correctness. + +Usage: python3 scripts/kb-eval/ref-file-audit.py +Zero dependencies, no network, makes no changes. +""" +import os, re, glob, collections, statistics + +ROOT = os.path.abspath(os.path.join(os.path.dirname(__file__), "..", "..")) +ref_files = sorted(glob.glob(os.path.join(ROOT, "skills/*/references/**/*.md"), recursive=True)) + +# Concatenate every routing hub (SKILL.md + agents) to test reachability. +hub_text = "" +for p in (glob.glob(os.path.join(ROOT, "skills/*/SKILL.md")) + + glob.glob(os.path.join(ROOT, "agents/*.md"))): + with open(p, encoding="utf-8") as f: + hub_text += f.read() + "\n" + +DAY_RE = re.compile(r"20\d\d-\d\d-\d\d") +MONTH_RE = re.compile(r"\*\*(?:Last updated|Sist oppdatert|Dato|Oppdatert):\*\*\s*20\d\d-\d\d(?!-)") +DATE_KEY = re.compile(r"\*\*(?:Last updated|Sist oppdatert|Dato|Oppdatert):\*\*") +URL_RE = re.compile(r"https?://(?:learn|docs)\.microsoft\.com[^\s)\"']*") +HDR_KEY_RE = re.compile(r"^\*\*([^:*]+):\*\*", re.M) + +per_skill = collections.Counter() +day_prec, month_only, truly_dateless = [], [], [] +named, folder_only, true_orphan = [], [], [] +no_source, sizes = [], [] +header_keys = collections.Counter() + +for p in ref_files: + rel = os.path.relpath(p, ROOT) + per_skill[rel.split("/")[1]] += 1 + with open(p, encoding="utf-8") as f: + txt = f.read() + head = "\n".join(txt.splitlines()[:10]) + sizes.append((txt.rstrip().count("\n") + 1, rel)) + + if DAY_RE.search(head) and DATE_KEY.search(head): + day_prec.append(rel) + elif MONTH_RE.search(head): + month_only.append(rel) + else: + truly_dateless.append(rel) + + if not URL_RE.search(txt): + no_source.append(rel) + + for k in HDR_KEY_RE.findall(head): + header_keys[k.strip()] += 1 + + base = os.path.basename(p) + folder = os.path.basename(os.path.dirname(p)) + if base in hub_text: + named.append(rel) + elif ("references/" + folder) in hub_text or ("/" + folder + "/") in hub_text: + folder_only.append(rel) + else: + true_orphan.append(rel) + +ln = [s for s, _ in sizes] +print(f"TOTAL: {len(ref_files)} reference files") +print(" per skill:", dict(per_skill)) +print() +print("[SIZE] lines") +print(f" min={min(ln)} median={int(statistics.median(ln))} mean={int(statistics.mean(ln))} max={max(ln)}") +b = collections.Counter() +for n in ln: + b["0-100" if n <= 100 else "101-300" if n <= 300 else "301-500" if n <= 500 else "500+"] += 1 +for k in ["0-100", "101-300", "301-500", "500+"]: + print(f" {k:8s}: {b[k]}") +print(" biggest 10:") +for n, r in sorted(sizes, reverse=True)[:10]: + print(f" {n:5d} {r}") +print() +print("[DATE] header precision") +print(f" day-precise (YYYY-MM-DD): {len(day_prec)}") +print(f" month-only (YYYY-MM): {len(month_only)} (deliberate convention)") +print(f" TRULY DATELESS: {len(truly_dateless)}") +for r in truly_dateless: + print(" -", r) +print() +print(f"[REACHABILITY] named={len(named)} via-folder={len(folder_only)} true-orphans={len(true_orphan)}") +for r in true_orphan: + print(" orphan:", r) +print() +print(f"[SOURCE] no learn/docs.microsoft.com URL: {len(no_source)} ({100*len(no_source)//len(ref_files)}%)") +print() +print(f"[HEADER KEYS] {len(header_keys)} distinct variants (0 files use YAML frontmatter)") +for k, c in header_keys.most_common(12): + print(f" {c:4d} **{k}:**")