feat(ms-ai-architect): v3.1 fan-out resume-prep — deterministisk payload-generator (build-judge-payloads.mjs) + selvbærende runbook; torsdag = kun spawn+score [skip-docs]
This commit is contained in:
parent
045db566ba
commit
52e822376b
3 changed files with 179 additions and 0 deletions
4
.gitignore
vendored
4
.gitignore
vendored
|
|
@ -34,6 +34,10 @@ scripts/kb-eval/data/skill-lifecycle-report.json
|
|||
# regenerated by `score-skill.mjs --write` and the apply-skill-op gate. Consumed by
|
||||
# the STEG C SessionStart surfacing; not committed (avoids churn in the public repo).
|
||||
scripts/kb-eval/data/skill-score-report.json
|
||||
# Generated judge bake-off fan-out payloads (45 per-file prompts, ~760KB) — derived
|
||||
# deterministically from judge-bakeoff-claims.json + the judge prompt by
|
||||
# build-judge-payloads.mjs; regenerated on demand, not committed (bulk + churn).
|
||||
scripts/kb-eval/data/judge-bakeoff-payloads-*.json
|
||||
.kb-backup/
|
||||
.rollback-in-progress
|
||||
|
||||
|
|
|
|||
86
docs/v3.1-fanout-runbook.md
Normal file
86
docs/v3.1-fanout-runbook.md
Normal file
|
|
@ -0,0 +1,86 @@
|
|||
# Runbook — v3.1 judge bake-off fan-out (resume-ready)
|
||||
|
||||
_Opprettet 2026-06-30. Selvbærende oppskrift for å kjøre v3.1-bake-offen (programdok §8 G1, option A) i en fersk sesjon. Deterministisk der det går; den ene LLM-tunge delen (45 subagenter) er isolert og eksplisitt. Forutsetter INGEN kontekst utover denne fila + de refererte artefaktene._
|
||||
|
||||
## Hvorfor / hva dette avgjør
|
||||
|
||||
v3 er adoptert baseline, målt **P 100,0 % / R 92,9 % / 0 FP** på G5b-korrigert gull (`scripts/kb-eval/data/judge-bakeoff-report-v3-g5bgold.{json,md}`). v3.1 (`scripts/kb-eval/judge-claim-prompt-v3.1.md`) er en ren recall-hardning av de 3 gjenstående FN (R1 øvre/nedre-grense-skille, R7 last-bærende-streng-carve-out, ny R8 fler-delt-fullstendighet). Denne kjøringen måler om v3.1 fanger de 3 FN UTEN å innføre nye FP.
|
||||
|
||||
**Forhåndsregistrert adopsjonsgate (låst FØR fan-out):** adopter v3.1 KUN hvis den **holder P = 100 % OG løfter R over 92,9 %** (mot G5b-gull). Enhver ny FP feller den → behold v3. Rapportens egen «GATE: PASS» (R≥0,70/P≥0,60) er kun gulvet, IKKE adopsjonsbaren.
|
||||
|
||||
## Forutsetninger (verifiser FØRST — premiss-sjekk)
|
||||
|
||||
```bash
|
||||
cd /Users/ktg/repos/ktg-plugin-marketplace/ms-ai-architect
|
||||
# 1. Manifest + gull fortsatt i sync (skal si 255 claims / 45 files):
|
||||
node scripts/kb-eval/extract-judge-claims.mjs # -> "blind manifest: 255 claims across 45 files"
|
||||
# 2. Suite + gull-lint grønt:
|
||||
node --test tests/kb-update/*.test.mjs tests/kb-eval/*.test.mjs # -> pass 641
|
||||
node scripts/kb-eval/lint-gold-consistency.mjs # -> "0 uwaived ... (373 claims)"
|
||||
```
|
||||
|
||||
Hvis manifestet IKKE er 255/45: gullet er endret siden 2026-06-30 → regenerer manifest (`node scripts/kb-eval/extract-judge-claims.mjs --write`) og re-mål v3-baseline mot oppdatert gull FØR du måler v3.1 (baren må være fersk — [[gold-freshness-can-invert-adoption]]).
|
||||
|
||||
## Steg 1 — generer payloads (deterministisk, ingen LLM)
|
||||
|
||||
```bash
|
||||
node scripts/kb-eval/build-judge-payloads.mjs \
|
||||
--prompt judge-claim-prompt-v3.1.md \
|
||||
--out judge-bakeoff-payloads-v3.1.json --write
|
||||
# -> "45 per-file payloads, 255 claims total, prompt=judge-claim-prompt-v3.1.md"
|
||||
```
|
||||
|
||||
Output: `scripts/kb-eval/data/judge-bakeoff-payloads-v3.1.json` (gitignored, regenererbar) = array av `{file, claim_count, prompt}`. Hver `prompt` er v3.1-malen med `<FILE>` + `<CLAIMS>` allerede substituert — klar til å dispatche ordrett.
|
||||
|
||||
## Steg 2 — fan-out (DEN dyre delen: 45 Opus-4.8-subagenter)
|
||||
|
||||
For HVER av de 45 payload-oppføringene, spawn ÉN subagent (inline `Agent`-verktøy):
|
||||
- **model:** `opus` (4.8), reasoning **xhigh** ([[subagent-model-opus-xhigh]]). Aldri Sonnet/Haiku.
|
||||
- **prompt:** payload-oppføringens `prompt`-felt, ordrett (ingen tillegg).
|
||||
- **subagent_type:** `general-purpose` (trenger `microsoft_docs_fetch`/`microsoft_docs_search`).
|
||||
- **blind:** payloaden inneholder ALDRI gull-verdikt/notes (manifestet er strippet) — ikke lekk dem.
|
||||
- **read-only:** subagenter skriver ALDRI til disk; hovedkonteksten aggregerer.
|
||||
- Kjør i batcher (f.eks. 8–10 samtidige) for å holde kontekst håndterbar; 45 totalt.
|
||||
|
||||
Hver subagent returnerer streng JSON: `{"file":"...","results":[{"id","judge_verdict","rule","evidence_url","evidence_quote","reason"},...]}`.
|
||||
|
||||
**Aggreger** alle 45 `results`-arrays til én fil. Behold MINST `id` + `judge_verdict` (+ `rule` anbefalt; scoreren bruker kun `judge_verdict`). Skriv:
|
||||
|
||||
`scripts/kb-eval/data/judge-bakeoff-results-v3.1.json` = `{"results":[ {"id":..., "judge_verdict":..., "rule":...}, … 255 totalt ]}`
|
||||
|
||||
(Format = identisk med `judge-bakeoff-results-v3.json` — sjekk den som mal.)
|
||||
|
||||
## Steg 3 — score (deterministisk) + anvend gaten
|
||||
|
||||
```bash
|
||||
node scripts/kb-eval/run-judge-bakeoff.mjs \
|
||||
--min-recall 0.70 --min-precision 0.60 \
|
||||
--results judge-bakeoff-results-v3.1.json \
|
||||
--report-prefix judge-bakeoff-report-v3.1 --write
|
||||
```
|
||||
|
||||
- Scoreren FEILER hardt hvis < 255 verdikt (ufullstendig fan-out) — fix manglende ids og kjør på nytt.
|
||||
- Les `data/judge-bakeoff-report-v3.1.json` → `arms.judge`: `precision`, `recall`, `fp`, `fn`, `tp`, `tn`.
|
||||
- **Adopsjon:** sammenlign mot v3 (P 100 / R 92,9, FP 0). Adopter v3.1 KUN hvis `precision == 1.0` (FP 0) OG `recall > 0.929`. Ellers: behold v3, dokumenter hvilke nye FP/regresjoner v3.1 innførte (input til en evt. v3.2).
|
||||
- Hvis v3.1 fanget noen men ikke alle 3 FN, eller innførte FP: det er et MÅLT resultat — før det i §8 G1-raden, ikke en feil.
|
||||
|
||||
## Steg 4 — uansett utfall
|
||||
|
||||
- Oppdater programdok §8 G1-rad + lukke-logg med målt P/R for v3.1 + adopsjonsbeslutning.
|
||||
- **Hvis adoptert:** den adopterte prompten (v3 eller v3.1) blir input til **G2** (wire inn i `scripts/kb-update/lib/transform.mjs`-judge-passet, Port 2 born-verified, + kadens-runner Port 3).
|
||||
- Forventede artefakter etterpå: `judge-bakeoff-results-v3.1.json` (commit), `judge-bakeoff-report-v3.1.{json,md}` (commit). Payloads-fila er gitignored.
|
||||
- Commit (`[skip-docs]`, ingen AI-trailers, Forgejo `origin`).
|
||||
|
||||
## De 3 FN v3.1 sikter på (forventet flip grounded→not_grounded)
|
||||
|
||||
| Claim | Feilmodus | v3.1-regel | Forventet |
|
||||
|---|---|---|---|
|
||||
| `ms-ai-security/cost-optimization/multi-model-strategy-costs.md#2` | «opptil 18» tak brutt av live 28 | R1 (øvre grense) | not_grounded |
|
||||
| `ms-ai-governance/monitoring-observability/token-usage-tracking-attribution.md#3` | metrikk-navn `PromptTokens`/`CompletionTokens` finnes ikke (live: `ProcessedPromptTokens`/`InputTokens`/`GeneratedTokens`/`OutputTokens`) | R7 (last-bærende streng) | not_grounded |
|
||||
| `ms-ai-infrastructure/bcdr/ai-foundry-disaster-recovery-planning.md#9` | Norway East er Global (ikke-residency) trenings-region, ikke regional | R8 (fler-delt) | not_grounded |
|
||||
|
||||
Disse er gull=`outdated` (positive). Fanger v3.1 alle 3 uten ny FP → R 92,9 → 100 ved P 100. Det er max-utfallet.
|
||||
|
||||
## Bakgrunn (les ved tvil)
|
||||
|
||||
`docs/ref-kb-correctness-program-2026-06.md` §8 (G1/G5/G5b) · `scripts/kb-eval/judge-claim-prompt-v3.1.md` (R1–R8) · `docs/ref-kb-gold-reconciliation-2026-06.md` (gull-flip-ledger) · STATE.md «👉 NESTE».
|
||||
89
scripts/kb-eval/build-judge-payloads.mjs
Normal file
89
scripts/kb-eval/build-judge-payloads.mjs
Normal file
|
|
@ -0,0 +1,89 @@
|
|||
#!/usr/bin/env node
|
||||
// build-judge-payloads.mjs — deterministic fan-out prep for the per-claim
|
||||
// groundedness judge bake-off. Turns the blind claim manifest + a judge prompt
|
||||
// template into 45 ready-to-dispatch subagent payloads (one per file), so the
|
||||
// fan-out is reproducible instead of hand-assembled in main context.
|
||||
//
|
||||
// The v3 fan-out grouped claims by file MANUALLY at dispatch time; that made the
|
||||
// exact payloads non-reproducible. This script fixes the construction: same prompt,
|
||||
// same per-file claim grouping, every run — so v3.1 (and any future arm) is
|
||||
// apples-to-apples and a fresh session can resume with one command, no improvising.
|
||||
//
|
||||
// Pure string assembly — no LLM, no network, no math. Reads:
|
||||
// data/judge-bakeoff-claims.json (blind manifest from extract-judge-claims.mjs)
|
||||
// <--prompt> (judge-claim-prompt-vN.md, with <FILE>/<CLAIMS>)
|
||||
// Writes (with --write):
|
||||
// data/<--out> (array of {file, claim_count, prompt})
|
||||
//
|
||||
// Usage:
|
||||
// node scripts/kb-eval/build-judge-payloads.mjs --prompt judge-claim-prompt-v3.1.md \
|
||||
// --out judge-bakeoff-payloads-v3.1.json [--write]
|
||||
// (default: print per-file claim counts + a sanity sample; --write persists)
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const DATA = path.join(__dirname, 'data');
|
||||
|
||||
function flag(name, def) {
|
||||
const i = process.argv.indexOf(name);
|
||||
return i >= 0 ? process.argv[i + 1] : def;
|
||||
}
|
||||
|
||||
const promptName = flag('--prompt');
|
||||
if (!promptName) {
|
||||
console.error('error: --prompt <judge-claim-prompt-vN.md> is required');
|
||||
process.exit(2);
|
||||
}
|
||||
const promptPath = path.isAbsolute(promptName) ? promptName : path.join(__dirname, promptName);
|
||||
if (!fs.existsSync(promptPath)) {
|
||||
console.error(`error: prompt template not found: ${promptPath}`);
|
||||
process.exit(2);
|
||||
}
|
||||
const template = fs.readFileSync(promptPath, 'utf8');
|
||||
if (!template.includes('<FILE>') || !template.includes('<CLAIMS>')) {
|
||||
console.error('error: prompt template must contain both <FILE> and <CLAIMS> placeholders');
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const manifest = JSON.parse(fs.readFileSync(path.join(DATA, 'judge-bakeoff-claims.json'), 'utf8'));
|
||||
const claims = manifest.claims || [];
|
||||
|
||||
// Group by file, preserving manifest order (deterministic).
|
||||
const byFile = new Map();
|
||||
for (const c of claims) {
|
||||
if (!byFile.has(c.file)) byFile.set(c.file, []);
|
||||
byFile.get(c.file).push({
|
||||
id: c.id,
|
||||
claim: c.claim,
|
||||
claim_type: c.claim_type,
|
||||
evidence_url: c.evidence_url || null,
|
||||
});
|
||||
}
|
||||
|
||||
const payloads = [];
|
||||
for (const [file, fileClaims] of byFile) {
|
||||
const claimsBlock = JSON.stringify(fileClaims, null, 2);
|
||||
const prompt = template.split('<FILE>').join(file).split('<CLAIMS>').join(claimsBlock);
|
||||
payloads.push({ file, claim_count: fileClaims.length, prompt });
|
||||
}
|
||||
|
||||
const totalClaims = payloads.reduce((s, p) => s + p.claim_count, 0);
|
||||
|
||||
if (process.argv.includes('--write')) {
|
||||
const outName = flag('--out', `judge-bakeoff-payloads-${path.basename(promptName).replace(/^judge-claim-prompt-/, '').replace(/\.md$/, '')}.json`);
|
||||
const outPath = path.join(DATA, outName);
|
||||
fs.writeFileSync(outPath, JSON.stringify(payloads, null, 2) + '\n');
|
||||
console.log(`wrote ${outPath}`);
|
||||
console.log(`${payloads.length} per-file payloads, ${totalClaims} claims total, prompt=${promptName}`);
|
||||
} else {
|
||||
console.log(`prompt template: ${promptName}`);
|
||||
console.log(`${payloads.length} files, ${totalClaims} claims total`);
|
||||
console.log('per-file claim counts:');
|
||||
for (const p of payloads) console.log(` ${p.claim_count.toString().padStart(2)} ${p.file}`);
|
||||
console.log(`\nsample payload[0] head (file=${payloads[0].file}):`);
|
||||
console.log(payloads[0].prompt.slice(0, 200) + ' …');
|
||||
console.log('\n(dry run — pass --write to persist)');
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue