Kjell Tore Guttormsen
a6ff17fd45
Skjerper ms-ai-infrastructure-description for å bryte build↔operate-overlappet mot ms-ai-engineering (operatør-godkjent leaning, S38): "multi-region"→"cross-region" og "edge AI architecture"→"edge AI deployment". Fjerner de delte distinktive tokenene `multi` (vekt 0.5) og `architecture` (0.333) → eng↔infra combined 7.4167→6.5833 (< 7.0 terskel). Korpus-maks faller til 6.8333 (eng↔gov, pass). Begge edits sammen gir lavere worst-case enn edit #1 alene (6.9167), derfor robust margin. Resultat: K10-gulvet løftes for begge → eng 89→96, infra 89→96. ALLE 5 skills nå ≥90 (advisor 91, eng/gov/infra/sec 96). Korpus-invarianten (operatør-krav S38) oppfylt. Judge-cache (🔑-regel): description-edit invaliderer infra K1. Re-judget mot det operatør-kuraterte 20-settet — UENDRET 1.0/0.0/1.0 (edit fjerner kun tokens → kan ikke gi falske positive; in-domain #2/#6 trigger fortsatt på BCDR/hybrid-vokabular). K4/K7/K8/K9 uendret (body/refs urørt). Provenance oppdatert S11→S39. Test: K10-real-five-testen pinnet pre-fix-baseline (eng+infra FAIL) — oppdatert til å pinne korpus-invarianten (alle 5 pass, eng↔infra < 7.0). kb-eval 142/142, kb-update 316/316, validate 239/0, kb-integrity 192/0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
48 lines
7.2 KiB
JSON
48 lines
7.2 KiB
JSON
{
|
|
"_meta": {
|
|
"rubric": "scripts/kb-eval/judge-prompt.md",
|
|
"judge_model": "opus",
|
|
"method": "5 parallel adversarial LLM-judges, one per skill",
|
|
"note": "K1 is AUTHORITATIVE as of S11 — measured by blinded judges (one per skill) against the operator-curated 20-prompt set in data/k1-trigger-prompts.json. K4/K7/K8/K9 carry prior cold-re-judge provenance inline."
|
|
},
|
|
"ms-ai-advisor": {
|
|
"_updated": "S11 (2026-06-20) — K1 authoritative (blinded judge vs curated 20-set); description tightened against over-trigging",
|
|
"K1_triggerPrecision": { "provisional": false, "inDomainHitRate": 1.0, "outDomainFalsePositiveRate": 0.0, "precision": 1.0, "pass": true, "misclassified": [], "notes": "S11 authoritative: blinded judge vs operator-curated 20-set. 10/10 in-domain hits, 0/10 false positives. Over-trigging fix VALIDATED — judge cited the new 'NOT for build/secure/operate/legally-assess' exclusion when rejecting all four sibling-domain prompts (engineering/security/governance/infra)." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "Body = persona + 7-phase workflow + ref-index; no ref detail reproduced. Only internal MCP-table redundancy (SKILL-internal, not SKILL<->ref)." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative." },
|
|
"K8_sourceCitation": { "ratio": 0.8, "pass": true, "notes": "AT THRESHOLD: architecture/decision-trees.md lacks dated header (footer source only). Add dated header to harden margin." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": ["Only meta-instructions (preview/GA as dynamic-to-verify) + stable identifiers (M365 SKUs, MADR v3.0). No stale-able product claim in body."] }
|
|
},
|
|
"ms-ai-engineering": {
|
|
"_updated": "S11 (2026-06-20) — K1 authoritative (blinded judge vs curated 20-set); K9 prior S10",
|
|
"K1_triggerPrecision": { "provisional": false, "inDomainHitRate": 1.0, "outDomainFalsePositiveRate": 0.0, "precision": 1.0, "pass": true, "misclassified": [], "notes": "S11 authoritative: blinded judge vs operator-curated 20-set. 10/10 in-domain hits, 0/10 false positives. Clean sibling separation (advisor/governance/security/infra/off-topic not triggered)." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "S10 re-judge: 7 section intros are orientation prose routing to references/<domain>/ + named kjernefiler; the two body tables (RAG-vs-finetuning, MLOps test-types) have no verbatim row-match in refs. No duplication." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 dated headers across 5 domains; format inconsistent (EN/NO, month vs day granularity)." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": [] }
|
|
},
|
|
"ms-ai-governance": {
|
|
"_updated": "S11 (2026-06-20) — K1 authoritative (blinded judge vs curated 20-set); Schrems II trigger added; K4+K9 prior S9",
|
|
"K1_triggerPrecision": { "provisional": false, "inDomainHitRate": 1.0, "outDomainFalsePositiveRate": 0.0, "precision": 1.0, "pass": true, "misclassified": [], "notes": "S11 authoritative: blinded judge vs operator-curated 20-set. 10/10 in-domain hits, 0/10 false positives (was 0.85). Schrems II recall fix VALIDATED — judge triggered on Schrems II / overføring-til-tredjeland prompts via the two new trigger phrases." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "S9 FIX: §6.2 now a compact decision-flow with explicit pointers ('[full forbudsliste i ai-act-classification-methodology.md]', '[åtte kategorier ...]') — Art.5 + Annex III lists no longer enumerated in body; they live only in references/responsible-ai/ai-act-classification-methodology.md. §2.1 is the single 4-level overview table ('ikke gjenta dem her'). §6.1 (DPIA tree), §1.2 (Digdir table), §6.3/§6.4 are routing/orientation, not verbatim copies of ref files." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative/infinitive." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 sampled refs carry Last updated + Status + Category headers." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": [] }
|
|
},
|
|
"ms-ai-infrastructure": {
|
|
"_updated": "S39 (2026-06-23) — K1 RE-JUDGED after K10 description edit (multi-region→cross-region, edge AI architecture→deployment); K4/K7/K8/K9 unchanged (body/refs untouched), prior S10/S11",
|
|
"K1_triggerPrecision": { "provisional": false, "inDomainHitRate": 1.0, "outDomainFalsePositiveRate": 0.0, "precision": 1.0, "pass": true, "misclassified": [], "notes": "S39 re-judge vs operator-curated 20-set after K10 boundary-sharpening edit. 10/10 in-domain hits, 0/10 false positives — UNCHANGED from S11. Edit only removed tokens (multi, architecture) → cannot add false positives; in-domain #2 (multi-region failover) and #6 (hybrid arkitektur) still trigger on remaining BCDR/resilience/hybrid vocabulary. engineering/governance/security/advisor/cost still correctly excluded." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "S10 re-judge: consistent summary+pointer pattern; §1.2 RTO/RPO now ~2 lines delegating to bcdr/rto-rpo-planning-ai-services.md (265 lines). SLA table replaced by relative-guidance prose. No procedural duplication." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative/infinitive." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 'Last updated: 2026-02' + Status; no source-URL on header line." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": [] }
|
|
},
|
|
"ms-ai-security": {
|
|
"_updated": "S11 (2026-06-20) — K1 authoritative (blinded judge vs curated 20-set); K4+K9 prior S9",
|
|
"K1_triggerPrecision": { "provisional": false, "inDomainHitRate": 1.0, "outDomainFalsePositiveRate": 0.0, "precision": 1.0, "pass": true, "misclassified": [], "notes": "S11 authoritative: blinded judge vs operator-curated 20-set. 10/10 in-domain hits, 0/10 false positives. Borderline content-safety/TCO correctly included; sibling governance/engineering/infra/advisor correctly excluded." },
|
|
"K4_noDuplication": { "score": 5, "pass": true, "evidence": "S9 re-judge (cold, post-fix): (a) 6x5 weights — body L50 routes to security-scoring-rubrics-6x5.md ('Ikke dupliser vekttallene her'), no body numbers; (b) risk-classification thresholds — body L54 routes to same rubric ('Ikke dupliser terskeltallene her'), canonical mapping incl. 1.00-1.49 Uakseptabel lives only in rubric; (c) P10/P50/P90 — body L94 affirms 'per komponent (ikke flat multiplikator)' owned by deterministic-cost-calculation-model.md §3, concrete factors only in cost model — body affirms, does not contradict; (d) OWASP table + §3 perf are routing/orientation with explicit volatile-numbers-live-in-refs note. No duplication/contradiction." },
|
|
"K7_imperativeStyle": { "ratio": 1.0, "pass": true, "notes": "10/10 sampled instruction sentences imperative/infinitive." },
|
|
"K8_sourceCitation": { "ratio": 1.0, "pass": true, "notes": "5/5 dated headers + Status; 3/5 also carry Verified: MCP <date>." },
|
|
"K9_noTimeSensitive": { "pass": true, "findings": [] }
|
|
}
|
|
}
|