open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

fix(ms-ai-architect): G7 idx-27b + idx-26f lukket — scorecard-punkt 2/5 til kildeordlyd, transparensmelding kopiert

Browse source 
idx-27b: Moenster 3-bulleten paastod en 'Powered by AI'-disclosure; erstattet med
den allerede ratifiserte standardmeldingen fra Copilot Studio-seksjonen, grep-
verifisert byte-for-byte. Ingen audience-tiering lagt til (scope-klassen fra 66fb567).

idx-26f: punkt 2 og 5 omskrevet til kildens ordlyd i idx-26d-dialekten framfor ren
sletting av navngitt spesifisitet - den minimale diffen ville etterlatt 'Dataset
statistics' (ogsaa ukildet) og 'across sensitive groups' (gammel dialekt), og dermed
arvet begge defektklassene. Form (b), stempel-innsnevring i fila, avvist av operatoer.

idx-26d-resolutionens sluttklausul supersedert med datert tillegg; idx-27c/27d
bokfoert for to naboer som ikke kunne repareres ved kopi.

Ko: 6 aapne / 8 resolved. Suite 1047/1047.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 22:26:49 +02:00
commit 9013d250bd
3 changed files with 104 additions and 7 deletions
Download patch file Download diff file Expand all files Collapse all files

71
docs/r11-pilot-results.md
View file

@ -1108,6 +1108,77 @@ the named axis well.
**Queue state: 6 open, 6 resolved.** Suite 1047/1047.
### 9.10 idx-27b and idx-26f closed — the minimal diff was the unsound one
Both entries lived in `transparency-documentation-standards.md`, so they were taken
together and grep-checked as a pair, the same way idx-26b/26c and idx-26d/27 were.
**idx-27b was the near-mechanical one and behaved like it.** The Mønster 3
"Azure implementasjon" bullet asserted a Copilot Studio `"Powered by AI"` disclosure —
the same imprecision idx-27 had just removed from the Copilot Studio section. Because
idx-27 had already ratified a replacement wording a few hundred lines below, the repair
was a *copy of a ratified formulation*, not a fresh judgement: the bullet now carries
the standard transparency message verbatim, and the quote was grep-verified
byte-for-byte against that row **after** the edit. Hand-copying a verbatim string is
where quote-style drift enters, and nothing in the suite would have caught it.
What was deliberately not carried over matters more than what was. The bullet sits
under an *audience-layering* table; the ratified row sits under *Built-in disclosures*.
Restating the claim in layered-disclosure vocabulary would have added reach the source
does not state — which is exactly the scope class caught in `66fb567` one session
earlier. The repair for a scope defect is the first place that scope defect can recur.
**idx-26f carried a real choice, and the obvious form of it was wrong.** Items 2 and 5
of the scorecard enumeration carried specificity the live source does not state, under
a **Confidence:** Verified stamp. The apparent repair — delete the named specificity —
does not survive contact with the source:
| Item | Anchor | What deleting only the named specificity leaves | Why that fails |
|---|---|---|---|
| 2 | `... across sensitive groups (gender, ethnicity, age)` | `... across sensitive groups` | Drops the *you-choose-them* agency the source states ("target values **you set** for **your desired** sensitive groups"), and strands item 2 in the pre-idx-26d dialect while items 1/3/6/7 now carry "du har satt" |
| 5 | `Dataset statistics, missing values, outlier analysis` | `Dataset statistics` | The source says only "shows you characteristics of your data" — `Dataset statistics` is *also* unstated, and unlike item 4 this entry never adjudicated it checked-and-clear |
So the minimal diff would have re-created both defects it was raised against: unsourced
content under the stamp, and the mixed dialect idx-26d had just canonicalised — **in the
file idx-26d had just canonicalised it in.** Both descriptions were instead rewritten to
the source's own wording in the established dialect. Item 4 was left untouched, per the
entry's own adjudication that it is a recognisable paraphrase.
**Form (b) — narrowing the stamp's stated reach inside the file — was put to the
operator and declined.** The entry's own text supplies the reason: the stamp is honest
today *only because* idx-26d's resolution writes down an exception, and removing the
need for that written-down exception is what closing the entry means. Annotating the
file instead relocates the annotation from the queue into the corpus; it does not close
the defect as booked.
**A closure can falsify prose in a tracked artefact.** idx-26d's resolution ended
"Items 2, 4 and 5 descriptions are NOT covered by that widening" — true when written,
false the moment idx-26f closed, and checked by nothing. Same class as the false
"linje 129" locator caught last session. It was handled by appending a dated
supersession clause rather than rewriting the original, and idx-26f's resolution now
states positively what the stamp covers, so no exception has to be written down
anywhere for it to stay honest.
**Two neighbours booked, not swept: idx-27c and idx-27d.** The Foundry agent-transparency
bullet asserts a `"This chatbot uses AI"` embeddable component; the scenario 1 tooling
line names a "Copilot Studio disclosure widget" where this same file documents a
pre-built "AI disclosure" *topic*. Neither can be repaired by copying the ratified
Copilot Studio wording — one needs its own live fetch against Foundry docs, the other is
intra-file name drift inside a worked scenario rather than a source-attribution error.
Booking them was not optional bookkeeping: idx-27b's resolution *asserts* they exist, so
leaving them unwritten would have planted the same false-reference defect this section
documents catching.
Cross-file grep before editing put all three replaced strings at zero occurrences
elsewhere in the corpus.
**The transferable part: the cheapest true edit is rarely the smallest diff.** A repair
scoped to exactly the words an entry names will silently adopt whatever unmeasured
content sits beside them — and adoption under a verification stamp is indistinguishable,
to a reader, from verification.
**Queue state: 6 open, 8 resolved.** Suite 1047/1047.
## Appendix A — the 15 admitted proposals, hand-verified
Every proposal the classifier (§4 + context condition) admitted over the whole

34
scripts/kb-eval/data/g7-review-queue.json
View file

@ -121,19 +121,20 @@
"anchors": [
"1. **Model overview**: Architecture, training data, intended use"
],
"resolution": "Operator-ratified 2026-08-03, full canonicalisation. The source was re-fetched live before writing (how-to-responsible-ai-scorecard) and enumerates seven segments: summary/model overview, data analysis, model performance, cohorts, top important factors, fairness insights, causal insights. All five original member NAMES were renamed to the source segment names, closing the defect this entry was actually booked under - that the list mixed two dialects after idx-26c added two source-named members. Rewriting item 1 alone was rejected for exactly that reason: it removes the false attribution but leaves the booked defect standing. DESCRIPTIONS were rewritten for two members only: item 1 (was \"Architecture, training data, intended use\", which is Model Card content - Model details / Intended use / Training data are canonical Model Card sections listed in this same file at lines 71-76 - now \"Modelloversikt og de target-verdiene du har satt\", matching the source summary segment) and item 3 (was \"Feature importance (global/local explanations)\", RAI *dashboard* vocabulary inside a scorecard enumeration, contradicting this file's own callout that dashboard components are not scorecard segments - now \"Faktorene som påvirker modellens prediksjoner mest\"). Descriptions of items 2, 4 and 5 were deliberately left UNTOUCHED even though they carry specifics the source does not state, because that is a distinct defect class (unsourced specificity under a verification marker) booked as idx-26f; rewriting them here would have silently resolved an entry raised the same session and left the queue incoherent. Source order was NOT imposed: idx-26c established the list carries no ordering claim and the five were already not in source order, so reordering would be a larger diff with no measured defect behind it. Cross-file check run before editing: the five names occur elsewhere in the corpus, but in dashboard/interpretability contexts, not as a mirror of this enumeration - except stakeholder-communication-ai-decisions.md:57-62, which is a parallel five-member list in the same paraphrase dialect and is booked as idx-26e rather than swept in. What the **Confidence:** Verified (MCP: microsoft-learn, 2026-08-03) stamp closing §3 now vouches for, stated explicitly rather than left implicit: a seven-member enumeration whose NAMES are all the source's own segment names and whose descriptions for items 1, 3, 6 and 7 were verified against the live source on the date the stamp already carries. Items 2, 4 and 5 descriptions are NOT covered by that widening - idx-26f is open against them."
"resolution": "Operator-ratified 2026-08-03, full canonicalisation. The source was re-fetched live before writing (how-to-responsible-ai-scorecard) and enumerates seven segments: summary/model overview, data analysis, model performance, cohorts, top important factors, fairness insights, causal insights. All five original member NAMES were renamed to the source segment names, closing the defect this entry was actually booked under - that the list mixed two dialects after idx-26c added two source-named members. Rewriting item 1 alone was rejected for exactly that reason: it removes the false attribution but leaves the booked defect standing. DESCRIPTIONS were rewritten for two members only: item 1 (was \"Architecture, training data, intended use\", which is Model Card content - Model details / Intended use / Training data are canonical Model Card sections listed in this same file at lines 71-76 - now \"Modelloversikt og de target-verdiene du har satt\", matching the source summary segment) and item 3 (was \"Feature importance (global/local explanations)\", RAI *dashboard* vocabulary inside a scorecard enumeration, contradicting this file's own callout that dashboard components are not scorecard segments - now \"Faktorene som påvirker modellens prediksjoner mest\"). Descriptions of items 2, 4 and 5 were deliberately left UNTOUCHED even though they carry specifics the source does not state, because that is a distinct defect class (unsourced specificity under a verification marker) booked as idx-26f; rewriting them here would have silently resolved an entry raised the same session and left the queue incoherent. Source order was NOT imposed: idx-26c established the list carries no ordering claim and the five were already not in source order, so reordering would be a larger diff with no measured defect behind it. Cross-file check run before editing: the five names occur elsewhere in the corpus, but in dashboard/interpretability contexts, not as a mirror of this enumeration - except stakeholder-communication-ai-decisions.md:57-62, which is a parallel five-member list in the same paraphrase dialect and is booked as idx-26e rather than swept in. What the **Confidence:** Verified (MCP: microsoft-learn, 2026-08-03) stamp closing §3 now vouches for, stated explicitly rather than left implicit: a seven-member enumeration whose NAMES are all the source's own segment names and whose descriptions for items 1, 3, 6 and 7 were verified against the live source on the date the stamp already carries. Items 2, 4 and 5 descriptions are NOT covered by that widening - idx-26f is open against them. [Superseded 2026-08-03 by idx-26f's closure: items 2 and 5 were rewritten to source wording and item 4 adjudicated a recognisable paraphrase; see idx-26f's resolution for what the stamp covers now.]"
},
{
"id": "idx-27b",
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
"class": "replacement",
"status": "open",
"status": "resolved",
"raised": "2026-08-03",
"summary": "Surfaced by the neighbourhood sweep for idx-27. The Mønster 3 \"Azure implementasjon\" list asserts a Copilot Studio \"Powered by AI\" disclosure in the chat interface - the same claim idx-27 removed from the Copilot Studio section, and false for the same reason: faqs-generative-orchestration documents a default transparency message (\"Just so you are aware, I sometimes use AI to answer your questions.\"), not a badge. Different section, outside idx-27's anchors, so booked separately rather than swept in. Repair is a replacement, outside the delete-only envelope. Note for whoever repairs this: after idx-27 the corrected wording already exists a few hundred lines below, so the repair is a copy of an already-ratified formulation rather than a fresh judgement.",
"evidence": "docs/r11-pilot-results.md §9.6; https://learn.microsoft.com/microsoft-copilot-studio/faqs-generative-orchestration",
"anchors": [
"- **Copilot Studio**: \"Powered by AI\" disclosure i chat interface"
]
],
"resolution": "Operator-ratified 2026-08-03. The Moenster 3 \"Azure implementasjon\" bullet was replaced with exactly what the already-ratified row at the Copilot Studio section asserts: the standard transparency message \"Just so you are aware, I sometimes use AI to answer your questions.\" delivered in the chat interface. This is a copy of a formulation ratified earlier the same session (idx-27), not a fresh judgement, and the quote was grep-verified byte-for-byte against that row after the edit - hand-copying a verbatim string is where quote-style drift enters. Deliberately NOT carried over: any audience-tiering or layered-disclosure language from the surrounding Moenster 3 pattern. The bullet sits under an audience-layering table while the source row sits under \"Built-in disclosures\"; adding reach the source does not state is exactly the scope class caught in 66fb567, and this repair would have inherited it. Standing matches idx-27's addendum: the standard message is documented in both faqs-generative-orchestration and faqs-generative-answers (the latter with the broader \"best practice to communicate to users that the agent uses artificial intelligence\" framing), so it carries no scope qualifier - unlike the confirmation safeguard, which does. Cross-file grep before editing: \"Powered by AI\" occurred nowhere else in the corpus. Two neighbours were checked and deliberately NOT swept in - the Foundry agent-transparency bullet asserting a \"This chatbot uses AI\" embeddable component, and the scenario-1 tooling line naming a \"Copilot Studio disclosure widget\" where this file's own Copilot Studio section documents a pre-built \"AI disclosure\" topic in the generative AI toolkit, not a widget. Both are unverified product artefacts in other sections against other sources; they are booked as idx-27c and idx-27d rather than repaired here."
},
{
"id": "idx-26e",
@ -152,13 +153,38 @@
"id": "idx-26f",
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
"class": "replacement",
"status": "open",
"status": "resolved",
"raised": "2026-08-03",
"summary": "Surfaced while writing idx-26d, and deliberately excluded from it. Items 2 and 5 of the scorecard enumeration carry specifics the live source does not state: the fairness insights segment is described with \"(gender, ethnicity, age)\" where the source says only \"your desired sensitive groups\", and the data analysis segment with \"missing values, outlier analysis\" where the source says only that it \"shows you characteristics of your data\". Neither is false on its face - both are plausible instances - but both sit under the **Confidence:** Verified (MCP: microsoft-learn, 2026-08-03) stamp, which now vouches for an enumeration re-verified against the live source. This is a distinct class from idx-26d's name drift and idx-26b's cross-attribution: unsourced SPECIFICITY under a verification marker, where the defect is the marker's reach rather than the sentence. idx-26d's resolution states explicitly that its widening of the stamp does not cover items 2, 4 and 5, so the stamp is currently honest only because that exclusion is written down - closing this entry is what would make it honest on its own terms. Item 4's description was checked in the same pass and is a recognisable paraphrase of the source, not added specificity, so it is named here as checked-and-clear rather than left ambiguous.",
"evidence": "docs/r11-pilot-results.md §9.8; https://learn.microsoft.com/azure/machine-learning/how-to-responsible-ai-scorecard",
"anchors": [
"2. **Fairness insights**: Performance disparities across sensitive groups (gender, ethnicity, age)",
"5. **Data analysis**: Dataset statistics, missing values, outlier analysis"
],
"resolution": "Operator-ratified 2026-08-03, form (a): descriptions rewritten to the source's own wording, NOT surgical deletion of the named parentheticals. The source was re-fetched live before writing (how-to-responsible-ai-scorecard) and states only \"how well your model is satisfying the fairness target values you set for your desired sensitive groups\" and \"The data analysis segment shows you characteristics of your data\". Item 2 is now \"Hvor godt modellen moeter fairness-target values du har satt for de sensitive gruppene du velger\" and item 5 \"Karakteristikker ved dataene dine\" (both written with correct Norwegian diacritics in the corpus file). Deleting only the specificity this entry NAMED was rejected on two counts, each of which would have made the repair inherit the defect class it was raised against: it leaves item 5 as \"Dataset statistics\", which the source does not state either and which this entry did NOT adjudicate as checked-and-clear the way it did item 4; and it leaves item 2 as \"across sensitive groups\", dropping the you-choose-them agency and stranding item 2 in the pre-idx-26d dialect while items 1, 3, 6 and 7 carry \"du har satt\" - reopening the mixed-dialect defect idx-26d was booked under, in the file idx-26d had just canonicalised. Form (b) - narrowing the stamp's stated reach in the file instead - was put to the operator and declined: it is honest-by-annotation relocated from this queue's resolution field into the corpus, i.e. the same mechanism this entry exists to remove the need for. Item 4 was left untouched per this entry's own adjudication. Cross-file grep before editing: \"gender, ethnicity, age\", \"outlier analysis\" and \"Dataset statistics\" occurred nowhere else in the corpus. The enumeration was re-read whole after the edit for coherence, not just for the two edited strings. SUPERSEDES the closing clause of idx-26d's resolution: what the **Confidence:** Verified (MCP: microsoft-learn, 2026-08-03) stamp vouches for is now a seven-member enumeration whose names are all the source's own segment names, whose descriptions for items 1, 2, 3, 5, 6 and 7 were verified against the live source on the date the stamp carries, and whose item 4 description was adjudicated a recognisable paraphrase of the source's causal-insights passage. No exception is written down anywhere for the stamp to remain honest."
},
{
"id": "idx-27c",
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
"class": "replacement",
"status": "open",
"raised": "2026-08-03",
"summary": "Surfaced by the neighbourhood sweep for idx-27b and deliberately excluded from it. The Microsoft Foundry \"Agent transparency\" list asserts \"Disclosure widgets: \\\"This chatbot uses AI\\\" embeddable component\" - a named product artefact carrying a verbatim-looking user-facing string. Same class as idx-27 and idx-27b (a plausible disclosure phrasing asserted as a built-in surface), but a different platform section against a different source, so it cannot be repaired by copying the ratified Copilot Studio formulation. Needs its own live fetch against Foundry documentation to establish whether an embeddable disclosure widget exists and, if so, what string it actually carries. Repair is a replacement, outside the delete-only envelope.",
"evidence": "docs/r11-pilot-results.md §9.10",
"anchors": [
"- Disclosure widgets: \"This chatbot uses AI\" embeddable component"
]
},
{
"id": "idx-27d",
"file": "skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md",
"class": "replacement",
"status": "open",
"raised": "2026-08-03",
"summary": "Surfaced by the neighbourhood sweep for idx-27b and deliberately excluded from it. The scenario 1 tooling line names a \"Copilot Studio disclosure widget\", but this same file's Copilot Studio section documents a pre-built \"AI disclosure\" TOPIC in the generative AI toolkit - not a widget. This is intra-file name drift of the same shape as idx-26d, not a source-attribution error: the file contradicts itself about what the artefact is called. Note the context differs from idx-27/idx-27b - this line sits inside a worked scenario recommendation rather than a product-capability claim, so the repair should decide whether to name the toolkit topic or drop the artefact name, not merely restate a source. Repair is a replacement, outside the delete-only envelope.",
"evidence": "docs/r11-pilot-results.md §9.10",
"anchors": [
"**Tooling:** Azure OpenAI Transparency Note + Copilot Studio disclosure widget"
]
}
]

6
skills/ms-ai-governance/references/responsible-ai/transparency-documentation-standards.md
View file

@ -112,10 +112,10 @@ PDF-rapport designet for å dele model- og data-innsikter mellom tekniske og ikk
**Komponenter i Scorecard:**
1. **Summary / model overview**: Modelloversikt og de target-verdiene du har satt
2. **Fairness insights**: Performance disparities across sensitive groups (gender, ethnicity, age)
2. **Fairness insights**: Hvor godt modellen møter fairness-target values du har satt for de sensitive gruppene du velger
3. **Top important factors**: Faktorene som påvirker modellens prediksjoner mest
4. **Causal insights**: Causal vs correlational relationships i features
5. **Data analysis**: Dataset statistics, missing values, outlier analysis
5. **Data analysis**: Karakteristikker ved dataene dine
6. **Model performance**: Modellens viktigste metrikker og prediksjonskarakteristikker, målt mot target values du har satt
7. **Cohorts**: Best og dårligst presterende datakohorter og subgrupper, automatisk uttrukket av scorecard-en
@ -215,7 +215,7 @@ Tilby ulike nivåer av transparency basert på audience:
**Azure implementasjon:**
- **Azure OpenAI**: Content Safety labels i API response
- **Copilot Studio**: "Powered by AI" disclosure i chat interface
- **Copilot Studio**: Standard transparensmelding i chat interface: "Just so you are aware, I sometimes use AI to answer your questions."
- **Azure Portal**: Model catalog med filterable model cards
---