feat(ms-ai-architect): G5 gull-friskhets-mikropass LUKKET — 2 gull-feil rettet (genaiops#2, model-selection#8), v3 REHABILITERT — slår v2 på fersk gull (P 89.7/R 92.1 vs 86.8/86.8); adopsjonsbeslutning reversert [skip-docs]
This commit is contained in:
parent
8ca8f33835
commit
bc3fc85b20
6 changed files with 598 additions and 9 deletions
|
|
@ -5,7 +5,11 @@
|
|||
"sample_frame": "fase0-sample-frame.json",
|
||||
"raw_returns": "fase0-returns/batch-*.json",
|
||||
"note_evidence_quote": "Per-claim evidence_quote omitted here to bound size; full verbatim quotes live in the session transcript. notes + evidence_url capture each verdict basis.",
|
||||
"claim_count": 373
|
||||
"claim_count": 373,
|
||||
"reconciliation_log": [
|
||||
"2026-06-30 Spor 2b: 12 judge-vs-gold-uenigheter adjudert mot live; 4 gull-feil rettet + 1 note-fiks. Logg: docs/ref-kb-gold-reconciliation-2026-06.md.",
|
||||
"2026-06-30 G5 friskhets-mikropass: 5 omstridte claims (gold=outdated, v3=grounded) re-adjudert mot live MS Learn. 2 stale gull rettet outdated->correct (genaiops-llm-specific-practices.md#2 '1600+' live 1900 tett nedre grense; model-selection-price-performance.md#8 Model Router GA nov 2025). 3 opprettholdt som outdated (judge-feil bekreftet: multi-model-strategy-costs.md#2, token-usage-tracking-attribution.md#3, ai-foundry-disaster-recovery-planning.md#9). Logg: docs/ref-kb-correctness-program-2026-06.md §8 G5."
|
||||
]
|
||||
},
|
||||
"claims": [
|
||||
{
|
||||
|
|
@ -1705,11 +1709,11 @@
|
|||
"stratum": "volatile",
|
||||
"claim": "Model Catalog 1600+ foundation models",
|
||||
"claim_type": "sku",
|
||||
"verdict": "outdated",
|
||||
"verdict": "correct",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/concepts/foundry-models-overview",
|
||||
"lastmod_changed": false,
|
||||
"file_last_updated": "2026-06-19",
|
||||
"notes": "Kilde over 1900. 1600+ utdatert lavere tall."
|
||||
"notes": "RECONCILED 2026-06-30 (G5 friskhets-mikropass): outdated->correct. Live (foundry-models-overview): 'There are over 1,900 models'. '1600+' er en SANN nedre grense (1900 >= 1600), understatement kun ~1,19x - innenfor den ratifiserte nedre-grense-policyen (grov = >~2x og beslutningsendrende). Gull var her for strengt: motsa egen policy. v3 grounded (R1) var korrekt. Fil kan friskes til '1900+' (Spor 0, valgfritt) men er IKKE feil som den star."
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-engineering/mlops-genaiops/genaiops-llm-specific-practices.md#3",
|
||||
|
|
@ -4008,11 +4012,11 @@
|
|||
"stratum": "volatile",
|
||||
"claim": "Model Router er GA-funksjonalitet i Microsoft Foundry",
|
||||
"claim_type": "status",
|
||||
"verdict": "outdated",
|
||||
"verdict": "correct",
|
||||
"evidence_url": "https://learn.microsoft.com/azure/foundry/foundry-models/how-to/model-choice-guide",
|
||||
"lastmod_changed": false,
|
||||
"file_last_updated": "2026-06-01",
|
||||
"notes": "Kilde lenker 'model router for Foundry (preview)'. Fortsatt preview, ikke GA."
|
||||
"notes": "RECONCILED 2026-06-30 (G5 friskhets-mikropass): outdated->correct. Model Router gikk GA nov 2025: whats-new-model-router har 'Model router GA version', kanonisk concept-side (foundry/openai/concepts/model-router) titulerer uten '(preview)' og lister v2025-11-18 som 'Active (latest)'. Gull var stale - basert pa en utdatert '(preview)'-lenkelabel pa siteringssiden (model-choice-guide); resterende '(preview)' gjelder enkelt-modeller (grok/DeepSeek/Claude-fotnoter), ikke ruter-tjenesten. v3 grounded (R7, fulgte til kanonisk side) var korrekt."
|
||||
},
|
||||
{
|
||||
"id": "ms-ai-security/cost-optimization/model-selection-price-performance.md#9",
|
||||
|
|
|
|||
234
scripts/kb-eval/data/judge-bakeoff-report-v2-g5gold.json
Normal file
234
scripts/kb-eval/data/judge-bakeoff-report-v2-g5gold.json
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
{
|
||||
"_meta": {
|
||||
"source": "gold-correctness-set.json + judge-bakeoff-results.json",
|
||||
"thresholds": {
|
||||
"minRecall": 0.7,
|
||||
"minPrecision": 0.6
|
||||
},
|
||||
"judged": 255
|
||||
},
|
||||
"population": {
|
||||
"total": 255,
|
||||
"verifiable": 240,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"unsourcedInP": 15
|
||||
},
|
||||
"arms": {
|
||||
"staleness": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 38,
|
||||
"tn": 202,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 0,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": null,
|
||||
"recallWilson": {
|
||||
"p": 0,
|
||||
"low": 0,
|
||||
"high": 0.09181293258383999
|
||||
},
|
||||
"precisionWilson": null
|
||||
},
|
||||
"judge": {
|
||||
"tp": 33,
|
||||
"fp": 5,
|
||||
"fn": 5,
|
||||
"tn": 197,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 38,
|
||||
"precision": 0.868421052631579,
|
||||
"recall": 0.868421052631579,
|
||||
"f1": 0.868421052631579,
|
||||
"recallWilson": {
|
||||
"p": 0.868421052631579,
|
||||
"low": 0.7267282994850112,
|
||||
"high": 0.9424621712426856
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.868421052631579,
|
||||
"low": 0.7267282994850112,
|
||||
"high": 0.9424621712426856
|
||||
}
|
||||
},
|
||||
"hybrid": {
|
||||
"tp": 33,
|
||||
"fp": 5,
|
||||
"fn": 5,
|
||||
"tn": 197,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 38,
|
||||
"precision": 0.868421052631579,
|
||||
"recall": 0.868421052631579,
|
||||
"f1": 0.868421052631579,
|
||||
"recallWilson": {
|
||||
"p": 0.868421052631579,
|
||||
"low": 0.7267282994850112,
|
||||
"high": 0.9424621712426856
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.868421052631579,
|
||||
"low": 0.7267282994850112,
|
||||
"high": 0.9424621712426856
|
||||
}
|
||||
}
|
||||
},
|
||||
"sourceSilent": {
|
||||
"onVerifiableNegative": 3,
|
||||
"onVerifiableError": 2,
|
||||
"agreesWithUnsourced": 5,
|
||||
"disagreesWithUnsourced": 10
|
||||
},
|
||||
"byClaimType": {
|
||||
"version": {
|
||||
"tp": 6,
|
||||
"fp": 0,
|
||||
"fn": 1,
|
||||
"tn": 21,
|
||||
"positives": 7,
|
||||
"negatives": 21,
|
||||
"flagged": 6,
|
||||
"precision": 1,
|
||||
"recall": 0.8571428571428571,
|
||||
"f1": 0.923076923076923,
|
||||
"recallWilson": {
|
||||
"p": 0.8571428571428571,
|
||||
"low": 0.4868654966809701,
|
||||
"high": 0.9743210440510252
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.6096569663469354,
|
||||
"high": 0.9999999999999999
|
||||
}
|
||||
},
|
||||
"tpm": {
|
||||
"tp": 4,
|
||||
"fp": 0,
|
||||
"fn": 1,
|
||||
"tn": 20,
|
||||
"positives": 5,
|
||||
"negatives": 20,
|
||||
"flagged": 4,
|
||||
"precision": 1,
|
||||
"recall": 0.8,
|
||||
"f1": 0.888888888888889,
|
||||
"recallWilson": {
|
||||
"p": 0.8,
|
||||
"low": 0.3755282641185388,
|
||||
"high": 0.9637768390302125
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.5100999795960008,
|
||||
"high": 1
|
||||
}
|
||||
},
|
||||
"region": {
|
||||
"tp": 1,
|
||||
"fp": 0,
|
||||
"fn": 0,
|
||||
"tn": 14,
|
||||
"positives": 1,
|
||||
"negatives": 14,
|
||||
"flagged": 1,
|
||||
"precision": 1,
|
||||
"recall": 1,
|
||||
"f1": 1,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.2065432914738929,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.2065432914738929,
|
||||
"high": 1
|
||||
}
|
||||
},
|
||||
"status": {
|
||||
"tp": 6,
|
||||
"fp": 1,
|
||||
"fn": 1,
|
||||
"tn": 45,
|
||||
"positives": 7,
|
||||
"negatives": 46,
|
||||
"flagged": 7,
|
||||
"precision": 0.8571428571428571,
|
||||
"recall": 0.8571428571428571,
|
||||
"f1": 0.8571428571428571,
|
||||
"recallWilson": {
|
||||
"p": 0.8571428571428571,
|
||||
"low": 0.4868654966809701,
|
||||
"high": 0.9743210440510252
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.8571428571428571,
|
||||
"low": 0.4868654966809701,
|
||||
"high": 0.9743210440510252
|
||||
}
|
||||
},
|
||||
"taxonomy": {
|
||||
"tp": 11,
|
||||
"fp": 3,
|
||||
"fn": 0,
|
||||
"tn": 83,
|
||||
"positives": 11,
|
||||
"negatives": 86,
|
||||
"flagged": 14,
|
||||
"precision": 0.7857142857142857,
|
||||
"recall": 1,
|
||||
"f1": 0.88,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.7411599827511859,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.7857142857142857,
|
||||
"low": 0.5241027622679172,
|
||||
"high": 0.9242875166308363
|
||||
}
|
||||
},
|
||||
"sku": {
|
||||
"tp": 5,
|
||||
"fp": 1,
|
||||
"fn": 2,
|
||||
"tn": 14,
|
||||
"positives": 7,
|
||||
"negatives": 15,
|
||||
"flagged": 6,
|
||||
"precision": 0.8333333333333334,
|
||||
"recall": 0.7142857142857143,
|
||||
"f1": 0.7692307692307692,
|
||||
"recallWilson": {
|
||||
"p": 0.7142857142857143,
|
||||
"low": 0.35892909014821267,
|
||||
"high": 0.9177828342909844
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.8333333333333334,
|
||||
"low": 0.43649056343635395,
|
||||
"high": 0.9699474141282697
|
||||
}
|
||||
}
|
||||
},
|
||||
"gate": {
|
||||
"pass": true,
|
||||
"recallOk": true,
|
||||
"precisionOk": true,
|
||||
"beatsStaleness": true,
|
||||
"thresholds": {
|
||||
"minRecall": 0.7,
|
||||
"minPrecision": 0.6
|
||||
},
|
||||
"reasons": [
|
||||
"all criteria met"
|
||||
]
|
||||
}
|
||||
}
|
||||
54
scripts/kb-eval/data/judge-bakeoff-report-v2-g5gold.md
Normal file
54
scripts/kb-eval/data/judge-bakeoff-report-v2-g5gold.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Judge bake-off-rapport — S1 (Fase 3 de-risk)
|
||||
|
||||
_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._
|
||||
|
||||
**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall.
|
||||
|
||||
## Evaluerings-populasjon (P)
|
||||
|
||||
Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
|
||||
|
||||
| metrikk | verdi |
|
||||
|---|---|
|
||||
| P totalt | 255 |
|
||||
| Verifiserbare (correct/outdated/wrong) | 240 |
|
||||
| Positive (reelle feil å fange) | 38 |
|
||||
| Negative (correct) | 202 |
|
||||
| Unsourced i P (kjørt, men utenfor P/R) | 15 |
|
||||
|
||||
## Arm-sammenligning (detektering over de 240 verifiserbare)
|
||||
|
||||
| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| staleness (billig baseline) | 0 | 0 | 38 | 202 | n/a | 0.0% | [0.0%, 9.2%] | n/a |
|
||||
| judge (per-påstand groundedness) | 33 | 5 | 5 | 197 | 86.8% | 86.8% | [72.7%, 94.2%] | 0.868 |
|
||||
| hybrid (union) | 33 | 5 | 5 | 197 | 86.8% | 86.8% | [72.7%, 94.2%] | 0.868 |
|
||||
|
||||
## Judge per claim_type (verifiserbar delmengde)
|
||||
|
||||
| claim_type | positive | TP | FP | FN | presisjon | recall |
|
||||
|---|---|---|---|---|---|---|
|
||||
| taxonomy | 11 | 11 | 3 | 0 | 78.6% | 100.0% |
|
||||
| version | 7 | 6 | 0 | 1 | 100.0% | 85.7% |
|
||||
| status | 7 | 6 | 1 | 1 | 85.7% | 85.7% |
|
||||
| sku | 7 | 5 | 1 | 2 | 83.3% | 71.4% |
|
||||
| tpm | 5 | 4 | 0 | 1 | 100.0% | 80.0% |
|
||||
| region | 1 | 1 | 0 | 0 | 100.0% | 100.0% |
|
||||
|
||||
## source_silent-diagnostikk
|
||||
|
||||
Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
|
||||
|
||||
| signal | antall | tolkning |
|
||||
|---|---|---|
|
||||
| På verifiserbar feil | 2 | judge-bom: reell feil oversett via «kan ikke verifisere» |
|
||||
| På verifiserbar correct | 3 | judge reproduserte ikke et korrekt faktum mennesket fant |
|
||||
| Enig med unsourced | 5 | judge reproduserer den uverifiserbare grensen (godt) |
|
||||
| Uenig med unsourced | 10 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
|
||||
|
||||
## GATE: ✅ PASS — bygg S3
|
||||
|
||||
- recall 0.868 ≥ 0.7? **ja**
|
||||
- presisjon 0.868 ≥ 0.6? **ja**
|
||||
- slår staleness (recall 0.000)? **ja**
|
||||
- begrunnelse: all criteria met
|
||||
234
scripts/kb-eval/data/judge-bakeoff-report-v3-g5gold.json
Normal file
234
scripts/kb-eval/data/judge-bakeoff-report-v3-g5gold.json
Normal file
|
|
@ -0,0 +1,234 @@
|
|||
{
|
||||
"_meta": {
|
||||
"source": "gold-correctness-set.json + judge-bakeoff-results.json",
|
||||
"thresholds": {
|
||||
"minRecall": 0.7,
|
||||
"minPrecision": 0.6
|
||||
},
|
||||
"judged": 255
|
||||
},
|
||||
"population": {
|
||||
"total": 255,
|
||||
"verifiable": 240,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"unsourcedInP": 15
|
||||
},
|
||||
"arms": {
|
||||
"staleness": {
|
||||
"tp": 0,
|
||||
"fp": 0,
|
||||
"fn": 38,
|
||||
"tn": 202,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 0,
|
||||
"precision": null,
|
||||
"recall": 0,
|
||||
"f1": null,
|
||||
"recallWilson": {
|
||||
"p": 0,
|
||||
"low": 0,
|
||||
"high": 0.09181293258383999
|
||||
},
|
||||
"precisionWilson": null
|
||||
},
|
||||
"judge": {
|
||||
"tp": 35,
|
||||
"fp": 4,
|
||||
"fn": 3,
|
||||
"tn": 198,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 39,
|
||||
"precision": 0.8974358974358975,
|
||||
"recall": 0.9210526315789473,
|
||||
"f1": 0.9090909090909091,
|
||||
"recallWilson": {
|
||||
"p": 0.9210526315789473,
|
||||
"low": 0.792003210797347,
|
||||
"high": 0.972785898605735
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.8974358974358975,
|
||||
"low": 0.7642084129775517,
|
||||
"high": 0.9593873444171301
|
||||
}
|
||||
},
|
||||
"hybrid": {
|
||||
"tp": 35,
|
||||
"fp": 4,
|
||||
"fn": 3,
|
||||
"tn": 198,
|
||||
"positives": 38,
|
||||
"negatives": 202,
|
||||
"flagged": 39,
|
||||
"precision": 0.8974358974358975,
|
||||
"recall": 0.9210526315789473,
|
||||
"f1": 0.9090909090909091,
|
||||
"recallWilson": {
|
||||
"p": 0.9210526315789473,
|
||||
"low": 0.792003210797347,
|
||||
"high": 0.972785898605735
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.8974358974358975,
|
||||
"low": 0.7642084129775517,
|
||||
"high": 0.9593873444171301
|
||||
}
|
||||
}
|
||||
},
|
||||
"sourceSilent": {
|
||||
"onVerifiableNegative": 0,
|
||||
"onVerifiableError": 0,
|
||||
"agreesWithUnsourced": 2,
|
||||
"disagreesWithUnsourced": 13
|
||||
},
|
||||
"byClaimType": {
|
||||
"version": {
|
||||
"tp": 7,
|
||||
"fp": 0,
|
||||
"fn": 0,
|
||||
"tn": 21,
|
||||
"positives": 7,
|
||||
"negatives": 21,
|
||||
"flagged": 7,
|
||||
"precision": 1,
|
||||
"recall": 1,
|
||||
"f1": 1,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.6456611570247934,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.6456611570247934,
|
||||
"high": 1
|
||||
}
|
||||
},
|
||||
"tpm": {
|
||||
"tp": 5,
|
||||
"fp": 0,
|
||||
"fn": 0,
|
||||
"tn": 20,
|
||||
"positives": 5,
|
||||
"negatives": 20,
|
||||
"flagged": 5,
|
||||
"precision": 1,
|
||||
"recall": 1,
|
||||
"f1": 1,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.5655085052479191,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.5655085052479191,
|
||||
"high": 1
|
||||
}
|
||||
},
|
||||
"region": {
|
||||
"tp": 1,
|
||||
"fp": 1,
|
||||
"fn": 0,
|
||||
"tn": 13,
|
||||
"positives": 1,
|
||||
"negatives": 14,
|
||||
"flagged": 2,
|
||||
"precision": 0.5,
|
||||
"recall": 1,
|
||||
"f1": 0.6666666666666666,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.2065432914738929,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.5,
|
||||
"low": 0.09452865480086614,
|
||||
"high": 0.9054713451991339
|
||||
}
|
||||
},
|
||||
"status": {
|
||||
"tp": 6,
|
||||
"fp": 3,
|
||||
"fn": 1,
|
||||
"tn": 43,
|
||||
"positives": 7,
|
||||
"negatives": 46,
|
||||
"flagged": 9,
|
||||
"precision": 0.6666666666666666,
|
||||
"recall": 0.8571428571428571,
|
||||
"f1": 0.75,
|
||||
"recallWilson": {
|
||||
"p": 0.8571428571428571,
|
||||
"low": 0.4868654966809701,
|
||||
"high": 0.9743210440510252
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 0.6666666666666666,
|
||||
"low": 0.3541973474990897,
|
||||
"high": 0.8794184013172571
|
||||
}
|
||||
},
|
||||
"taxonomy": {
|
||||
"tp": 9,
|
||||
"fp": 0,
|
||||
"fn": 2,
|
||||
"tn": 86,
|
||||
"positives": 11,
|
||||
"negatives": 86,
|
||||
"flagged": 9,
|
||||
"precision": 1,
|
||||
"recall": 0.8181818181818182,
|
||||
"f1": 0.9,
|
||||
"recallWilson": {
|
||||
"p": 0.8181818181818182,
|
||||
"low": 0.5230138624217553,
|
||||
"high": 0.9486333993289995
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.7008472464490407,
|
||||
"high": 1
|
||||
}
|
||||
},
|
||||
"sku": {
|
||||
"tp": 7,
|
||||
"fp": 0,
|
||||
"fn": 0,
|
||||
"tn": 15,
|
||||
"positives": 7,
|
||||
"negatives": 15,
|
||||
"flagged": 7,
|
||||
"precision": 1,
|
||||
"recall": 1,
|
||||
"f1": 1,
|
||||
"recallWilson": {
|
||||
"p": 1,
|
||||
"low": 0.6456611570247934,
|
||||
"high": 1
|
||||
},
|
||||
"precisionWilson": {
|
||||
"p": 1,
|
||||
"low": 0.6456611570247934,
|
||||
"high": 1
|
||||
}
|
||||
}
|
||||
},
|
||||
"gate": {
|
||||
"pass": true,
|
||||
"recallOk": true,
|
||||
"precisionOk": true,
|
||||
"beatsStaleness": true,
|
||||
"thresholds": {
|
||||
"minRecall": 0.7,
|
||||
"minPrecision": 0.6
|
||||
},
|
||||
"reasons": [
|
||||
"all criteria met"
|
||||
]
|
||||
}
|
||||
}
|
||||
54
scripts/kb-eval/data/judge-bakeoff-report-v3-g5gold.md
Normal file
54
scripts/kb-eval/data/judge-bakeoff-report-v3-g5gold.md
Normal file
|
|
@ -0,0 +1,54 @@
|
|||
# Judge bake-off-rapport — S1 (Fase 3 de-risk)
|
||||
|
||||
_Generert deterministisk av `run-judge-bakeoff.mjs` over `gold-correctness-set.json` + `judge-bakeoff-results.json`. Tall fra testet `lib/judge-bakeoff.mjs`. Ikke rediger for hånd — regenerer._
|
||||
|
||||
**Forhåndsregistrert gate (låst FØR fan-out):** recall ≥ 0.7, presisjon ≥ 0.6, OG judge-recall > staleness-recall.
|
||||
|
||||
## Evaluerings-populasjon (P)
|
||||
|
||||
Volatil stratum + fetchbare claim_types (price ekskludert) — der feilene bor; unngår «invertert leverage».
|
||||
|
||||
| metrikk | verdi |
|
||||
|---|---|
|
||||
| P totalt | 255 |
|
||||
| Verifiserbare (correct/outdated/wrong) | 240 |
|
||||
| Positive (reelle feil å fange) | 38 |
|
||||
| Negative (correct) | 202 |
|
||||
| Unsourced i P (kjørt, men utenfor P/R) | 15 |
|
||||
|
||||
## Arm-sammenligning (detektering over de 240 verifiserbare)
|
||||
|
||||
| arm | TP | FP | FN | TN | presisjon | recall | recall Wilson 95% | F1 |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| staleness (billig baseline) | 0 | 0 | 38 | 202 | n/a | 0.0% | [0.0%, 9.2%] | n/a |
|
||||
| judge (per-påstand groundedness) | 35 | 4 | 3 | 198 | 89.7% | 92.1% | [79.2%, 97.3%] | 0.909 |
|
||||
| hybrid (union) | 35 | 4 | 3 | 198 | 89.7% | 92.1% | [79.2%, 97.3%] | 0.909 |
|
||||
|
||||
## Judge per claim_type (verifiserbar delmengde)
|
||||
|
||||
| claim_type | positive | TP | FP | FN | presisjon | recall |
|
||||
|---|---|---|---|---|---|---|
|
||||
| taxonomy | 11 | 9 | 0 | 2 | 100.0% | 81.8% |
|
||||
| version | 7 | 7 | 0 | 0 | 100.0% | 100.0% |
|
||||
| status | 7 | 6 | 3 | 1 | 66.7% | 85.7% |
|
||||
| sku | 7 | 7 | 0 | 0 | 100.0% | 100.0% |
|
||||
| tpm | 5 | 5 | 0 | 0 | 100.0% | 100.0% |
|
||||
| region | 1 | 1 | 1 | 0 | 50.0% | 100.0% |
|
||||
|
||||
## source_silent-diagnostikk
|
||||
|
||||
Judgen hentet siden men fant ikke verdien. Diagnostisk, ikke et flagg.
|
||||
|
||||
| signal | antall | tolkning |
|
||||
|---|---|---|
|
||||
| På verifiserbar feil | 0 | judge-bom: reell feil oversett via «kan ikke verifisere» |
|
||||
| På verifiserbar correct | 0 | judge reproduserte ikke et korrekt faktum mennesket fant |
|
||||
| Enig med unsourced | 2 | judge reproduserer den uverifiserbare grensen (godt) |
|
||||
| Uenig med unsourced | 13 | judge hevdet grunnet/ugrunnet der mennesket ikke fant kilde |
|
||||
|
||||
## GATE: ✅ PASS — bygg S3
|
||||
|
||||
- recall 0.921 ≥ 0.7? **ja**
|
||||
- presisjon 0.897 ≥ 0.6? **ja**
|
||||
- slår staleness (recall 0.000)? **ja**
|
||||
- begrunnelse: all criteria met
|
||||
Loading…
Add table
Add a link
Reference in a new issue