feat(ms-ai-architect): Fase 0 steg 3 — base-rate-rapport + GATE: BYGG Fase 3 (scoped/hybrid) [skip-docs]
compute-base-rate.mjs (tynn CLI over testet lib/base-rate.mjs) -> base-rate-report.{json,md}.
373 paastander: verifiserbar feilrate 13,4 % (40/299), Wilson [10,0 %, 17,7 %]; unsourced 19,8 %.
staleness-recall 0/40 = 0 % -> korrekthets-judge er eneste mekanisme som fanger feilene.
Feil konsentrert i fetchbare typer (sku 36 %, version 25 %, tpm 20 %); pris 74 % unsourced -> operatoer-gated.
GATE besluttet i plan-doc: BYGG Fase 3 scoped til fetchbare claim_types (taxonomy/status/version/tpm/sku/region); pris ikke maskinverifiserbar.
This commit is contained in:
parent
6985ed1f5d
commit
49fd18c6d2
4 changed files with 512 additions and 0 deletions
66
scripts/kb-eval/data/base-rate-report.md
Normal file
66
scripts/kb-eval/data/base-rate-report.md
Normal file
|
|
@ -0,0 +1,66 @@
|
|||
# Base-rate-rapport — Fase 0 (KB korrekthet)
|
||||
|
||||
_Generert deterministisk av `compute-base-rate.mjs` over `gold-correctness-set.json`. Tall fra testet `lib/base-rate.mjs` (15 tester). Ikke rediger for hånd — regenerer._
|
||||
|
||||
**Gull-sett:** 373 påstander · metode: se gold-correctness-set.json `_meta.method`
|
||||
|
||||
## Verdict-vokabular
|
||||
|
||||
- **correct** — en hentet learn.microsoft.com-side oppgir den påståtte verdien
|
||||
- **outdated** — hentet kilde viser en annen, erstattet verdi (tidsdrift)
|
||||
- **wrong** — hentet kilde motsier påstanden; den var aldri korrekt
|
||||
- **unsourced** — ingen hentbar MS Learn-side oppgir verdien (kan ikke verifiseres)
|
||||
|
||||
«Reelle feil» = outdated + wrong. **unsourced er IKKE en feil** — det er den uverifiserbare massen (priser på JS-rendrede Azure-sider som en fetch-basert judge heller ikke når). Den verifiserbare feilraten ekskluderer derfor unsourced fra nevneren.
|
||||
|
||||
## Overall
|
||||
|
||||
| metrikk | verdi |
|
||||
|---|---|
|
||||
| Påstander totalt | 373 |
|
||||
| correct / outdated / wrong / unsourced | 259 / 29 / 11 / 74 |
|
||||
| Reelle feil (outdated+wrong) | 40 |
|
||||
| Verifiserbare påstander (nevner) | 299 |
|
||||
| **Verifiserbar feilrate** | **40/299 = 13.4%** |
|
||||
| Wilson 95 % | [10.0%, 17.7%] |
|
||||
| Unsourced-andel | 74/373 = 19.8% |
|
||||
| Feil staleness-loopen fanger (lastmod_changed=true) | 0 |
|
||||
| **Feil kun en korrekthets-judge fanger (judge-unique)** | **40** |
|
||||
|
||||
> **Gate-kritisk:** `errorsJudgeUnique` = reelle feil hvis siterte kilde-lastmod IKKE endret seg etter fildato — den eneste klassen en korrekthets-judge fanger som den eksisterende staleness-loopen bommer på. Staleness-recall på de reelle feilene = 0/40 = 0.0%.
|
||||
|
||||
## Per stratum
|
||||
|
||||
### Stratum
|
||||
|
||||
| key | n | correct | outdated | wrong | unsourced | err-rate (verifiable) | Wilson 95% | judge-unique |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| volatile | 331 | 222 | 28 | 10 | 71 | 38/260 = 14.6% | [10.8%, 19.4%] | 38 |
|
||||
| control | 42 | 37 | 1 | 1 | 3 | 2/39 = 5.1% | [1.4%, 16.9%] | 2 |
|
||||
|
||||
## Per skill
|
||||
|
||||
### Skill
|
||||
|
||||
| key | n | correct | outdated | wrong | unsourced | err-rate (verifiable) | Wilson 95% | judge-unique |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| ms-ai-security | 96 | 65 | 7 | 1 | 23 | 8/73 = 11.0% | [5.7%, 20.2%] | 8 |
|
||||
| ms-ai-engineering | 86 | 54 | 9 | 4 | 19 | 13/67 = 19.4% | [11.7%, 30.4%] | 13 |
|
||||
| ms-ai-advisor | 79 | 59 | 7 | 2 | 11 | 9/68 = 13.2% | [7.1%, 23.3%] | 9 |
|
||||
| ms-ai-governance | 76 | 52 | 5 | 2 | 17 | 7/59 = 11.9% | [5.9%, 22.5%] | 7 |
|
||||
| ms-ai-infrastructure | 36 | 29 | 1 | 2 | 4 | 3/32 = 9.4% | [3.2%, 24.2%] | 3 |
|
||||
|
||||
## Per claim_type
|
||||
|
||||
### Claim type
|
||||
|
||||
| key | n | correct | outdated | wrong | unsourced | err-rate (verifiable) | Wilson 95% | judge-unique |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| taxonomy | 122 | 108 | 7 | 3 | 4 | 10/118 = 8.5% | [4.7%, 14.9%] | 10 |
|
||||
| price | 76 | 20 | 0 | 0 | 56 | 0/20 = 0.0% | [0.0%, 16.1%] | 0 |
|
||||
| status | 70 | 58 | 4 | 3 | 5 | 7/65 = 10.8% | [5.3%, 20.6%] | 7 |
|
||||
| version | 33 | 24 | 6 | 2 | 1 | 8/32 = 25.0% | [13.3%, 42.1%] | 8 |
|
||||
| tpm | 30 | 20 | 4 | 1 | 5 | 5/25 = 20.0% | [8.9%, 39.1%] | 5 |
|
||||
| sku | 22 | 14 | 6 | 2 | 0 | 8/22 = 36.4% | [19.7%, 57.0%] | 8 |
|
||||
| region | 20 | 15 | 2 | 0 | 3 | 2/17 = 11.8% | [3.3%, 34.3%] | 2 |
|
||||
|
||||
Loading…
Add table
Add a link
Reference in a new issue