open/ms-ai-architect
1
0
Fork 0
Code Issues Releases 1 Activity

feat(ms-ai-architect): G7-ankere kan baere sin egen fil — klasse-entryen for de 42 er skrevet, og #17 og #14 er begge oppfylt

Browse source 
Operator ratifiserte utvidelsen 2026-08-11. Beslutning #17 (én klasse-entry)
og grunnen bak #14 (ingen lokator usynlig for gaten) var begge ekte og
kolliderte i skjemaet. De kolliderer ikke lenger.

SKJEMAET: et anker er enten en ordrett STRENG, sjekket mot entry.file som foer,
eller et { file, text }-PAR sjekket mot SIN EGEN fil. Bakoverkompatibelt — alle
47 eksisterende entries er uroert. Et misformet par er en schema-feil, ikke et
anker som stille matcher ingenting.

TDD, og testene maatte skjerpes foer de var ekte: fem tester skrevet og kjoert
FOERST. To av dem PASSERTE mot gammel kode — av feil grunn: den gamle koden
stringify-er ankerobjektet inn i drift-meldingen, saa /other\.md/ traff
tilfeldig. En test som ikke kan feile beviser ingenting. Begge fikk en
diskriminator (meldingen skal IKKE navne entry.file) og feilet deretter.
5 feilende -> implementasjon -> 20/20 i fila, 1052/1052 i suiten.

idx-26an: 42 { file, text }-ankere, ett per medlem, hvert re-verifisert ordrett
OG unikt i sin egen fil med split-telling foer skriving.

MEKANISMEN ER BEVIST, IKKE ANTATT: kjoert mot den ekte validatoren med ett
medlems anker fjernet i en lesestubb — ok=false, ett anchor_drift, riktig
entry-id, og meldingen navner agent-evaluation-testing-frameworks.md. Det er
nettopp garantien #14 fantes for, naa baaret av én entry i stedet for 42.

MAALT HASARD BOKFOERT I ENTRYEN: to av de 42 ankertekstene er strenge
prefikser av andre medlemmers ankere («**Total MCP calls:** 6» i
ai-services-cost-optimization.md, «**MCP Calls:** 3» i
reserved-capacity-planning.md). Hver er unik i SIN fil, saa koen er trygg — en
kryss-fil search-and-replace er det ikke. Slett per fil, og rest-soek etter
hver edit.

Koen: 47 -> 48 entries (28 open, 20 resolved). Bokfoeringen av #17 er dermed
komplett: 1 klasse-entry + 16 individuelle = 17. Ingen korpusfil roert.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 22:23:13 +02:00
commit 8ea54ec00e
3 changed files with 287 additions and 12 deletions
Download patch file Download diff file Expand all files Collapse all files

179
scripts/kb-eval/data/g7-review-queue.json
View file

@ -645,6 +645,185 @@
"- 9 unique source URLs from Microsoft Learn",
"- Coverage: Prompt Shields, Security Benchmark (AI-2, AI-3), LLM Security Planning, Content Filtering"
]
},
{
"id": "idx-26an",
"file": "scripts/kb-eval/data/r11-footer-class-2026-08-11.json",
"class": "multi-locator",
"status": "open",
"raised": "2026-08-11",
"summary": "CLASS-WIDE ENTRY. 42 members in 42 files, all line-form, none flagged, none carrying an open decision - the remainder of the footer counting class after the 16 that needed their own entry (idx-26x, idx-26y, idx-26z, idx-26aa..idx-26am). Measured in docs sec. 9.16: the full class is 58 members in 58 files, dataset scripts/kb-eval/data/r11-footer-class-2026-08-11.json (tracked), buckets 30 konsistent / 11 inkonsistent / 1 tvetydig / 16 ikke sjekkbar.\nTHE FORM APPLIES TO THESE WITHOUT FURTHER JUDGEMENT. Ratified on idx-26j 2026-08-09: current provenance, one referent for the whole footer block, and the unverifiable MCP-calls line is DELETED rather than corrected to the sum. THE GROUND IS UNVERIFIABILITY, NOT ARITHMETIC - no artifact in the repo records MCP calls per file, so the number is unadjudicable under both readings of the referent. The consistency bucket therefore governs BOOKKEEPING ONLY and never whether a line is deleted; a konsistent member goes for the same reason an inkonsistent one does.\nWHY ONE ENTRY AND NOT 42. Operator decision #17, 2026-08-11, overriding decision #14 for exactly this subset: #14 (\"one entry per file\") was taken when the class was believed to be 14 files, and 42 near-identical entries are the mechanism sec. 9.15 showed lets a mismeasurement hide behind its own copies - seven entries there cited each other. #14 also had a REAL reason, re-measured and honoured here rather than discarded: the queue schema carried one `file` per entry, so a class-wide entry would have left these files invisible to check-g7-queue.mjs - the 795d494 failure, a locator alive only in prose while both gates go green. Measured empirically against the live validator before this entry was written: a class entry with 42 string anchors returned ok=false with 41 of 42 reported as anchor_drift. Both reasons were true and they collided.\nRESOLVED BY EXTENDING THE SCHEMA, operator-ratified 2026-08-11: an anchor may now be a { file, text } pair checked against ITS OWN file, alongside the existing string form checked against entry.file. Backwards compatible - every pre-existing entry is untouched - and it satisfies both constraints at once: one record, and all 42 locators still machine-checked. Five tests written FIRST and observed failing; two of them initially passed against the old code for the wrong reason (it serialised the anchor object into the drift message, so the file-name regex matched by accident) and were tightened until they discriminated.\nANCHORS: 42 { file, text } pairs, one per member, each re-verified verbatim AND unique within its own file by split-count before writing - the gate proves presence, never uniqueness.\nHAZARD FOR WHOEVER APPLIES THE DELETIONS, measured not assumed: two of the 42 anchor texts are STRICT PREFIXES of other members' anchors - \"**Total MCP calls:** 6\" in ai-services-cost-optimization.md is a prefix of the longer lines in multi-agent-orchestration-patterns.md and continuous-improvement-feedback-loops.md, and \"**MCP Calls:** 3\" in reserved-capacity-planning.md is a prefix of the line in prompt-testing-and-evaluation.md. Each is unique inside its own file, so this queue is safe; a cross-file search-and-replace is NOT. Delete per file, and run a rest-search in each file after its edit, because a deletion can leave the number alive elsewhere in the same file (idx-26ab is the measured instance).\nNOT MACHINE-APPLIABLE. Nothing in this queue is, by definition. apply-o2-ratified.mjs is delete-only and aborts here. Closing this entry means recording, in the resolution, which files were edited and what was verified after each edit.",
"evidence": "scripts/kb-eval/data/r11-footer-class-2026-08-11.json (the 42 are the members with form=line and flags=[]); docs sec. 9.16 (the measurement: four sweeps with different blind spots, residual of each read by hand, label vocabulary enumerated at 68 distinct labels / 141 lines); idx-26j resolution (the ratified form); idx-26q (the #14 reasoning and the 795d494 failure class); docs sec. 9.15 (duplication as the place a mismeasurement hides); commit 1c07afd (the 16 individual entries)",
"anchors": [
{
"file": "skills/ms-ai-advisor/references/copilot-extensibility/adaptive-cards-copilot-responses.md",
"text": "**MCP calls:** 3 docs_search, 2 docs_fetch, 1 code_sample_search"
},
{
"file": "skills/ms-ai-advisor/references/copilot-extensibility/copilot-api-rate-limiting-resilience.md",
"text": "**MCP Calls:** 6 (3 searches, 2 fetches, 1 code sample search)"
},
{
"file": "skills/ms-ai-advisor/references/copilot-extensibility/declarative-agents-grounding-strategies.md",
"text": "**MCP-kall:** 7 (3 search, 3 fetch, 1 code sample search)"
},
{
"file": "skills/ms-ai-advisor/references/copilot-extensibility/sharepoint-copilot-agents.md",
"text": "**MCP-calls:** 5 (3 search + 2 fetch)."
},
{
"file": "skills/ms-ai-advisor/references/copilot-extensibility/teams-copilot-message-extensions.md",
"text": "**MCP-kall utført:** 6 (3 search, 2 fetch, 1 code sample search)"
},
{
"file": "skills/ms-ai-advisor/references/prompt-engineering/multi-turn-conversation-management.md",
"text": "**MCP calls:** 5 (search + fetch)"
},
{
"file": "skills/ms-ai-advisor/references/prompt-engineering/prompt-testing-and-evaluation.md",
"text": "**MCP Calls:** 3 (microsoft_docs_search × 2, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)"
},
{
"file": "skills/ms-ai-engineering/references/agent-orchestration/agent-evaluation-testing-frameworks.md",
"text": "- **MCP calls:** 3 (microsoft_docs_search) + 2 (microsoft_docs_fetch) + 1 (microsoft_code_sample_search) = 6"
},
{
"file": "skills/ms-ai-engineering/references/agent-orchestration/agent-memory-and-context-management.md",
"text": "**MCP calls**: 6 (3x microsoft_docs_search, 2x microsoft_docs_fetch, 1x microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/agent-orchestration/foundry-workflows-visual-orchestration.md",
"text": "**MCP calls**: 4 (2x docs_search, 2x docs_fetch)"
},
{
"file": "skills/ms-ai-engineering/references/agent-orchestration/multi-agent-orchestration-patterns.md",
"text": "**Total MCP calls:** 6 (3 microsoft_docs_search + 2 microsoft_docs_fetch + 1 microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/ai-services-api-best-practices.md",
"text": "**MCP call summary:** 7 microsoft_docs_search + 4 microsoft_docs_fetch + 1 microsoft_code_sample_search = 12 total MCP calls"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/ai-services-cost-optimization.md",
"text": "**Total MCP calls:** 6"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/ai-services-governance-compliance.md",
"text": "**Total antall MCP-kall:** 8 (4 docs_search + 4 docs_fetch)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/ai-services-networking-security.md",
"text": "**MCP calls:** 7 (microsoft_docs_search, microsoft_docs_fetch, microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/azure-ai-vision-image-analysis.md",
"text": "**MCP-kall totalt:** 4 (3 docs_search + 1 code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/document-intelligence-prebuilt-models.md",
"text": "**Totalt MCP-kall:** 5 (3× search, 2× fetch, 1× code samples)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/speech-services-text-to-speech.md",
"text": "**Totalt antall MCP-kall:** 7 (4 × docs_search, 3 × docs_fetch, 1 × code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/azure-ai-services/translator-document-translation.md",
"text": "**Total MCP calls:** 4 (docs_search) + 3 (docs_fetch) = **7**"
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/data-drift-monitoring-detection.md",
"text": "**MCP Calls:** 5 (3 × microsoft_docs_search, 1 × microsoft_docs_fetch, 1 × microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/genaiops-llm-specific-practices.md",
"text": "**Totalt:** 18 kilder, 8 MCP-kall."
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/mlops-security-access-control.md",
"text": "**MCP Calls:** 8 (microsoft-learn docs search + fetch, code samples)"
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/mlops-teams-collaboration-tools.md",
"text": "**Totalt antall MCP-kall:** 6 (3x search, 2x fetch, 1x code samples)"
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/model-deployment-strategies-azure.md",
"text": "**MCP-kall utført:** 8 (microsoft_docs_search × 5, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)"
},
{
"file": "skills/ms-ai-engineering/references/mlops-genaiops/responsible-ai-mlops-integration.md",
"text": "**MCP-calls brukt:** 6 (microsoft_docs_search x 3, microsoft_docs_fetch x 2, microsoft_code_sample_search x 1)"
},
{
"file": "skills/ms-ai-engineering/references/rag-architecture/rag-cost-optimization.md",
"text": "**MCP calls:** 3 (search) + 2 (fetch) = 5 total"
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/real-time-streaming-monitoring.md",
"text": "**MCP calls:** 6 (3 × search, 2 × fetch, 1 × code search)"
},
{
"file": "skills/ms-ai-governance/references/monitoring-observability/response-quality-metrics-rag.md",
"text": "**MCP research calls:** 3 (microsoft_docs_search × 3, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)"
},
{
"file": "skills/ms-ai-governance/references/responsible-ai/continuous-improvement-feedback-loops.md",
"text": "**Total MCP calls:** 6 (3 searches + 2 fetches + 1 code sample search)"
},
{
"file": "skills/ms-ai-governance/references/responsible-ai/human-in-the-loop-oversight.md",
"text": "**MCP Calls:** 6 (3 searches + 2 fetches + 1 code sample search)"
},
{
"file": "skills/ms-ai-governance/references/responsible-ai/responsible-ai-policy-development.md",
"text": "**MCP Calls:** 4 (microsoft_docs_search x3, microsoft_docs_fetch x2)"
},
{
"file": "skills/ms-ai-security/references/ai-security-engineering/ai-security-scoring-framework.md",
"text": "**MCP calls:** 5 (3 søk + 2 fetch)"
},
{
"file": "skills/ms-ai-security/references/ai-security-engineering/content-safety-filter-calibration.md",
"text": "**MCP-kall:** 6 (3x microsoft_docs_search, 2x microsoft_docs_fetch, 1x microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-security/references/ai-security-engineering/norwegian-content-safety.md",
"text": "**MCP-kall:** 6 (microsoft_docs_search x6)"
},
{
"file": "skills/ms-ai-security/references/ai-security-engineering/output-validation-grounding-verification.md",
"text": "**MCP-kall utført:** 4 (2x docs_search, 1x code_sample_search, 2x docs_fetch)"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/azure-ai-foundry-cost-governance.md",
"text": "**Total MCP Calls:** 4 (3x microsoft_docs_search, 1x microsoft_docs_fetch, 1x microsoft_code_sample_search)"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/budget-forecasting-ai-projects.md",
"text": "**Total MCP calls:** 3 (docs_search) + 2 (docs_fetch) + 1 (code_sample_search) = 6"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/inference-endpoint-cost-optimization.md",
"text": "**Totalt MCP-kall:** 3 (microsoft_docs_search) + 2 (microsoft_docs_fetch) + 1 (microsoft_code_sample_search) = 6"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/model-selection-price-performance.md",
"text": "**MCP-kall brukt:** 6 (4x docs_search, 2x docs_fetch)"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/reserved-capacity-planning.md",
"text": "**MCP Calls:** 3"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/small-language-models-economics.md",
"text": "**Total MCP-kall:** 4 (3x search, 2x fetch, 1x code samples)"
},
{
"file": "skills/ms-ai-security/references/cost-optimization/token-counting-optimization.md",
"text": "**MCP Calls:** 4 (microsoft_docs_search × 3, microsoft_docs_fetch × 2, microsoft_code_sample_search × 1)"
}
]
}
]
}

56
scripts/kb-eval/lib/g7-queue.mjs
View file

@ -23,6 +23,9 @@ export const ENTRY_STATES = new Set(['open', 'resolved']);
const REQUIRED = ['id', 'file', 'class', 'status', 'raised', 'summary', 'evidence', 'anchors'];
const isAnchor = (a) =>
typeof a === 'string' || (a !== null && typeof a === 'object' && typeof a.file === 'string' && typeof a.text === 'string');
/**
* @param {unknown} entries queue entries
* @param {(path: string) => string} readFile
@ -78,21 +81,50 @@ export function validateQueue(entries, readFile) {
continue;
}
let text;
try {
text = readFile(entry.file);
} catch {
findings.push({ id, kind: 'file_unreadable', message: `cannot read ${entry.file}` });
// An anchor is either a verbatim string, checked against entry.file, or a
// { file, text } pair checked against ITS OWN file. The pair form exists for
// class-wide entries — one record for a defect that lives in many files.
// Measured 2026-08-11: booking such a class as a single-file entry hid 41 of
// 42 locators from this check, which is the 795d494 failure, a locator alive
// only in prose while both gates go green. A malformed pair is a schema fault
// rather than an anchor that quietly matches nothing.
const malformed = entry.anchors.filter((a) => !isAnchor(a));
if (malformed.length > 0) {
findings.push({
id,
kind: 'schema',
message: `anchor must be a verbatim string or a { file, text } pair: ${malformed
.map((m) => JSON.stringify(m))
.join(', ')}`,
});
continue;
}
const gone = entry.anchors.filter((a) => !text.includes(a));
if (gone.length > 0) {
findings.push({
id,
kind: 'anchor_drift',
message: `anchor no longer occurs in ${entry.file}: ${gone.map((g) => JSON.stringify(g)).join(', ')}`,
});
const byFile = new Map();
for (const a of entry.anchors) {
const target = typeof a === 'string' ? entry.file : a.file;
const text = typeof a === 'string' ? a : a.text;
if (!byFile.has(target)) byFile.set(target, []);
byFile.get(target).push(text);
}
for (const [target, texts] of byFile) {
let text;
try {
text = readFile(target);
} catch {
findings.push({ id, kind: 'file_unreadable', message: `cannot read ${target}` });
continue;
}
const gone = texts.filter((t) => !text.includes(t));
if (gone.length > 0) {
findings.push({
id,
kind: 'anchor_drift',
message: `anchor no longer occurs in ${target}: ${gone.map((g) => JSON.stringify(g)).join(', ')}`,
});
}
}
}

64
tests/kb-eval/test-g7-queue.test.mjs
View file

@ -119,6 +119,70 @@ test('a non-array queue is rejected without throwing', () => {
assert.match(r.findings[0].message, /array/i);
});
// A class-wide entry books ONE finding that lives in many files. Booking it as a
// single-file entry was measured to hide 41 of 42 locators from the gate — the
// 795d494 failure, a locator alive only in prose while both gates go green. So an
// anchor may also be a { file, text } pair, checked against ITS OWN file.
test('a { file, text } anchor is checked against its own file, not entry.file', () => {
const e = { ...okEntry(), anchors: [{ file: 'skills/x/references/other.md', text: 'over here' }] };
const r = validateQueue(
[e],
stub({ 'skills/x/references/y.md': 'nothing matching', 'skills/x/references/other.md': 'aaa over here bbb' }),
);
assert.equal(r.ok, true);
assert.deepEqual(r.findings, []);
});
test('a { file, text } anchor absent from its own file DRIFTS, and the finding names that file', () => {
const e = { ...okEntry(), anchors: [{ file: 'skills/x/references/other.md', text: 'over here' }] };
const r = validateQueue(
[e],
stub({ 'skills/x/references/y.md': 'over here', 'skills/x/references/other.md': 'empty' }),
);
assert.equal(r.ok, false);
assert.equal(r.findings[0].kind, 'anchor_drift');
assert.match(r.findings[0].message, /other\.md/);
// Discriminating: entry.file DOES contain the text, so a message blaming y.md
// would mean the anchor was looked up in the wrong file. Without this line the
// test passes against the old code, which serialises the object into the
// message and matches /other\.md/ by accident.
assert.doesNotMatch(r.findings[0].message, /y\.md/);
});
test('string and { file, text } anchors mix in one entry, and BOTH are checked', () => {
const e = { ...okEntry(), anchors: ['the exact text', { file: 'skills/x/references/other.md', text: 'gone' }] };
const r = validateQueue(
[e],
stub({ 'skills/x/references/y.md': 'the exact text', 'skills/x/references/other.md': 'not it' }),
);
assert.equal(r.ok, false);
assert.equal(r.findings.length, 1);
assert.match(r.findings[0].message, /gone/);
// Discriminating, for the same reason: the surviving string anchor is present
// in entry.file, so the drift must be reported against other.md.
assert.match(r.findings[0].message, /other\.md/);
assert.doesNotMatch(r.findings[0].message, /y\.md/);
});
test('an unreadable per-anchor file is reported, not thrown, and names that file', () => {
const e = { ...okEntry(), anchors: [{ file: 'skills/x/references/missing.md', text: 'whatever' }] };
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'the exact text' }));
assert.equal(r.ok, false);
assert.equal(r.findings[0].kind, 'file_unreadable');
assert.match(r.findings[0].message, /missing\.md/);
});
// Without this, a typo'd anchor object would be checked against nothing and the
// entry would pass — a defect leaving the programme unnoticed, which is the one
// outcome the queue exists to prevent.
test('a malformed anchor object is a schema fault, never a silent pass', () => {
const e = { ...okEntry(), anchors: [{ file: 'skills/x/references/other.md' }] };
const r = validateQueue([e], stub({ 'skills/x/references/y.md': 'x', 'skills/x/references/other.md': 'x' }));
assert.equal(r.ok, false);
assert.equal(r.findings[0].kind, 'schema');
assert.match(r.findings[0].message, /anchor/i);
});
test('the real queue file validates against the live corpus', async () => {
const { readFileSync } = await import('node:fs');
const queue = JSON.parse(readFileSync('scripts/kb-eval/data/g7-review-queue.json', 'utf8'));