Kjell Tore Guttormsen
215772df87
Read-only eval-verktoy scripts/kb-eval/eval.mjs (determ. K2/K3/K5/K6 + ref-tall) + operator-gated LLM-judge (judge-prompt.md -> data/judge-results.json) flettet til data/eval-baseline.json. 13 enhetstester (tests/kb-eval/), 42 kb-update-tester gronne. Baseline: K5 3/5 fail, K9 4/5 fail, K4 2/5 fail, ref-tall 2/5 fail; 2 konkrete bugs flagget (security 6x5-vekting-motstrid, governance broken ref-path). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01REiKFhP4w6xGXXqWKpPCJJ
43 lines
6.4 KiB
JSON
43 lines
6.4 KiB
JSON
{
|
|
"_meta": {
|
|
"rubric": "scripts/kb-eval/judge-prompt.md",
|
|
"judge_model": "opus",
|
|
"method": "5 parallel adversarial LLM-judges, one per skill",
|
|
"note": "K1 precision is PROVISIONAL — operator must curate the final 20 trigger-prompts per skill before K1 is authoritative."
|
|
},
|
|
"ms-ai-advisor": {
|
|
"K1_triggerPrecision": { "provisional": true, "precision": 1.0, "notes": "20/20 from description. BORDERLINE: cost/diagram/DPIA/AI-Act prompts are MS-AI-adjacent; broad 'Microsoft AI architecture' phrase risks over-triggering vs sibling commands. Operator must curate + stress-test sibling overlap." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "Body = persona + 7-phase workflow + ref-index; no ref detail reproduced. Only internal MCP-table redundancy (SKILL-internal, not SKILL<->ref)." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative." },
|
|
"K8_sourceCitation": { "ratio": 0.8, "pass": true, "notes": "AT THRESHOLD: architecture/decision-trees.md lacks dated header (footer source only). Add dated header to harden margin." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": ["Only meta-instructions (preview/GA as dynamic-to-verify) + stable identifiers (M365 SKUs, MADR v3.0). No stale-able product claim in body."] }
|
|
},
|
|
"ms-ai-engineering": {
|
|
"K1_triggerPrecision": { "provisional": true, "precision": 0.95, "notes": "19/20. Miss: 'Semantic Kernel agent governance' overlaps security/governance because description lists 'governance' under agent-orchestration. Operator must curate." },
|
|
"K4_noDuplication": { "score": 4, "pass": true, "evidence": "Mostly router. Two inline tables (RAG-vs-finetuning; MLOps test-types w/ Ragas/red-teaming) restate comparison content also in ref files — summary-level, not verbatim." },
|
|
"K7_imperativeStyle": { "ratio": 0.9, "pass": true, "notes": "9/10; section intros are descriptive prose by design." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 dated headers; schema drift ('Dato:' vs 'Last updated'); no source-URL in headers." },
|
|
"K9_noTimeSensitive": { "pass": false, "findings": ["Line 119: 'Foundry Agent Service GA' — explicit GA-status claim in body", "GPT-4o/Whisper/text-embedding-3/Florence — version-pinned product names in body", "Line 48: 'Agent Framework (erstatter Semantic Kernel Agents)' — lifecycle/transition claim", "A2A/CUA/Foundry Workflows — era-bound feature names"] }
|
|
},
|
|
"ms-ai-governance": {
|
|
"K1_triggerPrecision": { "provisional": true, "precision": 0.95, "notes": "19/20. Miss: Schrems II / data-transfer — body section 2.3 covers it but description has NO Schrems/dataoverføring keyword. Add trigger phrase to description." },
|
|
"K4_noDuplication": { "score": 3, "pass": false, "evidence": "Moderate duplication: §6.1 DPIA risk-factor tree (incl. >=2-faktorer threshold) duplicates dpia-norwegian-methodology-ai.md; §6.2 + §2.1 AI Act taxonomy duplicates ai-act-classification-methodology.md; §1.2 Digdir 7-principle table restates per-principle files. Regulatory update must be applied twice." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 imperative." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 dated headers + Status + inline source citations (Lovdata/NSM/europalov). BONUS DEFECT: SKILL.md line 191 references 'drift-detection-automated-retraining.md' which does NOT exist (actual: model-performance-drift-detection.md) — broken ref path." },
|
|
"K9_noTimeSensitive": { "pass": false, "findings": ["Line 121: 'Microsoft EU Data Boundary ... Azure OpenAI (Sweden Central, West Europe)' — volatile region/availability claim in body (most genuine finding)", "Lines 81/121: 'Regulation 2024/1689' + 'Schrems II (C-311/18)' — legal identifiers (stable, borderline)", "§2.1/§6.2 AI Act obligation status presented without Digital Omnibus caveat — may date"] }
|
|
},
|
|
"ms-ai-infrastructure": {
|
|
"K1_triggerPrecision": { "provisional": true, "precision": 1.0, "notes": "20/20. Tightly scoped + named trigger phrases, low false-positive risk. Borderline (hybrid RAG, multi-region cost) need real-corpus validation. Operator must curate." },
|
|
"K4_noDuplication": { "score": 4, "pass": true, "evidence": "Largely summary/routing w/ '> Ref:' deferral. Minor: RTO/RPO table + SLA table put specific values in body that also live in ref files." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 imperative (concentrated in per-section directives)." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 'Last updated: 2026-02' + Status. NO source-URL in headers; identical batch date reads as generation timestamp, not per-file verification." },
|
|
"K9_noTimeSensitive": { "pass": false, "findings": ["Lines 93-98: SLA table w/ hardcoded percentages (99.9/99.999/99.95%) + 'Standard v2'", "Lines 149/158/184/186/249: Phi-3/Phi-4 model versions + param counts (3.8B/14B) in body", "Line 141: 'Azure Local (tidl. Azure Stack HCI)' rename note", "Line 87: hardcoded 'peak + 30% buffer'"] }
|
|
},
|
|
"ms-ai-security": {
|
|
"K1_triggerPrecision": { "provisional": true, "precision": 0.95, "notes": "19/20. Clean sibling separation (AI Act/DPIA/platform/RAG/BCDR not triggered). One ambiguity: model feature-comparison could be pulled by broad 'performance optimization for AI'. Operator must curate." },
|
|
"K4_noDuplication": { "score": 3, "pass": false, "evidence": "DUPLICATION-WITH-CONTRADICTION: body weighting table (L52-59 Standard: Identity 20/Network 15/Data 20/Content 20/Compliance 15/Monitoring 10) CONTRADICTS canonical rubric security-scoring-rubrics-6x5.md (Compliance 25/Data 20/Identity 20/Content 15/Network 10/Monitoring 10). Body scoring rule (weighted sum) also diverges from rubric (Ja-checkpoint count). Real correctness bug." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 imperative." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 dated headers; mostly no header source-URL (URLs in body cost-register rows)." },
|
|
"K9_noTimeSensitive": { "pass": false, "findings": ["Line 73: 'OWASP LLM Top 10 (2025)' — dated standard version", "Line 92: 'Defender ... GA for AI applications, Preview for AI agents' + 'ikke i Azure Government' — GA/preview + region status in body", "Line 138: 'GPT-4o mini vs GPT-4o' — model versions", "Lines 149-153: hardcoded perf/price figures (20-50ms, 5-10x, 50%/80%, Batch 50% @ 24h SLA)"] }
|
|
}
|
|
}
|