34 KiB
R11 pilot results — measured, 2026-08-03
The §10 acceptance measurement of docs/r11-tiered-fix-design.md, run against
the live ledger. No KB file was edited and no ledger record was written.
Instrument: scripts/kb-eval/lib/fix-op.mjs (+ tests/kb-eval/test-fix-op-classify.test.mjs,
49 tests after §4b) driven by scripts/kb-eval/classify-fix-ops.mjs. The classifier is
the O1 driver with writes disabled — it constructs the swap and checks the §4
invariant, so measurements 1 and 3 come out of the mechanism that would later
touch the corpus, not out of a proxy heuristic.
Artefact: scripts/kb-eval/data/r11-pilot-classification.json (untracked,
regenerable; per-flag records so the run can be re-analysed without re-running).
It holds the pilot run — node scripts/kb-eval/classify-fix-ops.mjs --write.
Every corpus-wide figure below is from --threshold 1, and the per-table
reproduce command is stated where it is used.
Two different 202s. This population is 202 flags. §3's "202 flags whose claim and quote contain a numeric token" is a different 202, measured over the full 712-flag population. They are unrelated.
1. The headline
Nine provable, correct value swaps exist in the entire 776-flag not_grounded
population — 1.2 %. The machine half of the R11 tiering buys nine edits. Every
other flag needs a human.
| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|---|---|---|---|---|
Pilot (not_grounded ≥ 7) |
24 | 202 | 2 | 2 |
Whole not_grounded corpus |
218 | 776 | 15 | 9 |
These are numeric-path figures, and they stay that way. §4b (the status synonym table) was implemented afterwards and adds a separate class with its own hand-verification — see §8. A run today prints O1 = 7 (pilot) and 23 (corpus) because the status proposals are included in the total; the numeric line above is unchanged and is still what
s4_as_writtencompares against.
Reproduce: node scripts/kb-eval/classify-fix-ops.mjs (pilot) and
node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 (corpus). The nine are
enumerated with verdicts in appendix A — that hand-verification is the only thing
separating 9 from 15, so it is recorded rather than left in a session transcript.
This is the answer §10 asked for, and it is materially worse than the design assumed: "If the split is materially worse than assumed, that is known after one session rather than after ten."
The O2 half, measured afterwards (§9), does not rescue the number. Of the 46 R8 multi-part claims that have the O2 shape, 17 are candidates and 2 are clean subtractions on the available evidence; all 29 non-candidates fail because the source supplies a corrected value, which makes them swaps or rewrites. O2's value in R11 is triage — it tells a human which 17 to look at first — not automation.
2. §4 as written is not sufficient — measured, not argued
§4 claims its invariant is "deliberately stronger than human review at scale." It is not. Run exactly as specified over the pilot, it admitted 6 swaps, of which 4 were wrong — precision 2/6:
| Proposed swap | Why it is wrong |
|---|---|
30-dagers → 24-dagers |
Unit crossing. The quote says 24 hours. |
3000 requests/sekund → 50 |
Metric crossing. The quote is a query throttle per index; the claim is an indexing rate per replica. |
Microsoft Agent 365 → Agent 7 |
Identifier mutilated. The 7 was harvested out of E7. |
text-embedding-ada-002 → ada-2 |
Identifier mutilated. The 2 came from a dimensions column. |
The defect is structural, not incidental. §4 constrains where the new value came from (verbatim in the cited quote) and what the edit looks like (one line, rest byte-identical). It constrains nothing about whether the two tokens denote the same quantity. Same-type-and-provenance is not same-referent.
The added condition
contextCorresponds() requires the token to sit under the same label or the
same trailing unit on both sides. It is deliberately lexical, with no
translation table: dokumenter is not taught to equal documents, because a
synonym table introduces a new fact source and is an operator decision, not an
engineering one. Consequence, measured: a swap is provable essentially only where
the context is language-neutral — a URL, a code sample, a parameter key.
3. The condition is necessary but still not sufficient
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15 splits them cleanly by token type:
Reproduce: node scripts/kb-eval/classify-fix-ops.mjs --threshold 1
(the persisted artefact is the pilot run — the corpus-wide tables in §1 and §3
come from this threshold-1 run). Per-proposal verdicts: appendix A.
| Token type | Proposals | Correct | Wrong | Unverified | Failure mode |
|---|---|---|---|---|---|
iso_date |
9 | 9 | 0 | 0 | — every one is an api-version= bump in a URL or code sample |
number |
5 | 0 | 3 | 2 | AI-900 → AI-901, gpt-4o → gpt-5.1o (×2) |
version |
1 | 0 | 1 | 0 | Java agent 3.7.5 → 3.4.0 — a downgrade |
A matching identifier prefix (AI-, gpt-) satisfies the context condition
while the digit is part of a name, not a quantity. Only iso_date survives
hand-verification, and the report marks it as the sole recommended class
(o1_recommended). number and version proposals must not be applied.
4. The four §10 measurements
-
O1 / O2 / O3 split. O1 = 2/202 on the pilot (9/776 corpus-wide, safe class only).
O2 is undetermined — it does not exist as a class until §5 is ratified, so every non-O1 item is O3 by design.§5 was ratified 2026-08-03, and O2 has since been measured over the class where it can exist at all — see measurement 2 and §9. It remains unmeasured outside R8 ∧MULTI_PART_CLAIM; the classifier still routes every non-O1 item to O3, so O3 ≥ 200/202 stands as the machine's own partition. -
How much of R8 resolves as O2. MEASURED 2026-08-03 — see §9. R8 is 87/202 on the pilot (366/776 corpus-wide) and yields zero O1. Of the pilot's 87, 46 are structural enumerations (R8 ∧
MULTI_PART_CLAIM) — the O2 candidate shape. Which of them subtract cleanly turns on the judge's prosereason, and no regex reads prose, so this was done by prose classification: 17 of 46 (37 %) are O2 candidates, 29 are O3. All 29 are foreclosed by condition 3. -
O1 abort rate: 99 % (200/202). Typed, because "99 %" alone is not actionable:
Code Pilot Class MULTI_PART_CLAIM96 (47.5 %) intrinsic — not a value swap at all MULTI_VALUE_TOKEN29 intrinsic NO_VALUE_TOKEN28 intrinsic — the claim asserts prose STATUS_SYNONYM15 operator question (§6.2) NOT_VERBATIM15 intrinsic LOCATOR_AMBIGUOUS7 fixable engineering gap MULTI_REPLACEMENT6 intrinsic CONTEXT_MISMATCH4 intrinsic — these are the 4 wrong edits above Only 7 of 200 aborts (3.5 %) are a fixable engineering gap. More locator engineering cannot move the O1 number materially.
-
Review throughput per class. NOT MEASURED. It requires human review sessions, which have not happened. Recording it as measured would be false.
5. Two further findings
F1 — subtraction can leave a misleading remainder. §5 argues O2 "cannot introduce a new error, because it asserts strictly less." True of the sentence, false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
- Bing) er GA (juni 2025)"* where the source says the tool is deprecated.
Subtracting
er GA (juni 2025)leaves the tool standing in a list of available tools. Strictly less asserted; still misleading. O2 therefore still requires a human to look at the remainder — cheaper than O3 (no fact-finding) but not mechanical.
F2 — subtraction can destroy true information. Real case: a list of seven
prebuilt model IDs where the judge found six correct and prebuilt-check wrong —
the real ID is prebuilt-check.us. Subtraction drops a model that exists; the
correct fix is a swap. Subtraction is not the safe default everywhere.
F3 — disposition carries zero information. It is outdated on 202 of
202 flags. docs/r11-flag-format-2026-07.md specifies not_grounded → {outdated, wrong} with "the human assigns which at R11", but the pass
hard-assigned outdated. Do not use it as a classifier signal. Spec/data
divergence, recorded.
F4 — claims are not file text. claim is an LLM-extracted, translated
restatement: 0 of 202 match their file line verbatim, and 188 share no 40-char
run with it. For table claims, line points at the header, not the value.
This is why the locator exists at all, and why it searches the enclosing block
rather than the line.
6. Operator decisions — ALL THREE RATIFIED 2026-08-03
All three were put to the operator with the recommendations below and all three
were accepted as recommended. The contract text now lives in
docs/r11-tiered-fix-design.md §4a/§4b/§5; this section records what was asked
and what the answer was.
None of the three is implemented yet. The classifier still aborts
STATUS_SYNONYM and still routes every non-O1 item to O3. A later session builds
against the ratified contract — it must not assume the code already honours it.
- Ratify O2 (§5)? → RATIFIED, with the remainder check (not as a blanket rule), exactly as F1/F2 above argued. Contract: design doc §5, three conditions, human-confirmed.
- Amend §4 with a ratified synonym table? → RATIFIED, narrow and closed. Contract: design doc §4b — four label rows, closed table, complete-label-only, corpus-side value with the file's own markup preserved. Unlocks up to 54 corpus-wide flags.
- Is O1 worth building at all? → KEPT, locked to
iso_date. Contract: design doc §4a condition 5 — a driver may applyiso_dateproposals and must never applynumberorversionones. Nine edits corpus-wide.
7. What this does not change
The design's core reading survives: the expensive half (locating the source, reading it, extracting the deciding passage) was already paid for by the judge pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot falsifies is the assumption that a meaningful share of that evidence converts into machine-provable edits. It does not. R11 is a human review programme with a nine-item machine assist, and its leverage lies entirely in the O2 decision.
8. §4b implemented — the status class measured, 2026-08-03
The ratified synonym table (docs/r11-tiered-fix-design.md §4b) is implemented in
fix-op.mjs and the class is measured. Reproduce:
node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 (status_synonym block).
| Population | STATUS_SYNONYM flags | Proven by §4b | Still aborting |
|---|---|---|---|
Pilot (not_grounded ≥ 7) |
15 | 5 | 10 |
Whole not_grounded corpus |
54 | 8 | 46 |
Why the other 46 abort, corpus-wide — this sub-distribution is the actionable
part, because the top-level STATUS_SYNONYM count alone says nothing:
| Reason | N | What it means |
|---|---|---|
NO_COMPLETE_FILE_LABEL |
27 | the file writes the status inside a sentence — (preview) in a list item, **DSPM (preview):**, "[Preview]: …" in a JSON string. Constraint 2 refuses these, correctly. |
NO_SOURCE_STATUS |
15 | the cited quote carries no listed lifecycle phrasing at all — the flag was never a status swap. |
SOURCE_STATUS_AMBIGUOUS |
2 | the quote asserts two different rows (e.g. "…is now generally available. Partner solutions remain in preview."). |
FILE_ALREADY_MATCHES |
2 | file and source agree; the mismatch was in the LLM-extracted claim, not in the corpus. |
The class is REVIEW-grade, not apply-grade — 5 of 8 correct. All eight were
hand-judged against the cited source (appendix B). Three defects, all one family:
§4b binds the table, the completeness of the file label and the written value, and
nothing about whether the source phrasing refers to the row's own subject.
That is the same provenance-without-referent defect that falsified §4 (§2), now
reproduced in the status class. status is therefore deliberately absent from
o1_recommended: the machine writes nothing, and every proposal reaches a human.
Two candidate conditions, costed over the eight
Neither is implemented — extending a table the operator ratified as closed is an operator decision, exactly as condition 5 was in §4a. Both are pure gain on this population (they kill wrong proposals and no correct one), which is the number the decision needs:
| Candidate | Kills | Correct proposals lost |
|---|---|---|
A — count a bare GA in the quote as a GA-row source phrasing, so a quote saying both GA and "public preview" becomes ambiguous |
1 (onelake:198) |
0 |
| B — abort when the quote is a multi-entity enumeration (≥ 2 pipes, or a numbered list) | 2 (onelake:198, owasp:79) |
0 |
B subsumes A on these eight. Neither catches security-copilot-integration.md:94,
where the quote is prose and the (Preview) marker simply belongs to a different
agent. A stricter referent-name condition (the row's subject must appear in the
quote) would catch it — and would also kill two correct proposals (:83, :93),
where the source names the capability rather than the agent. That trade is real and
is why this is put to the operator rather than shipped.
9. §10 measurement #2 — how much of R8 resolves as O2, measured 2026-08-03
The last machine-answerable pilot measurement. §4.2 recorded it as "not
answered, and not answerable by machine" — true of a regex, not of prose
classification, which is what this ran. Reproduce the verification and the
tally: node scripts/kb-eval/check-o2-returns.mjs.
Population. The 46 pilot flags that are R8 ∧ MULTI_PART_CLAIM — the O2
candidate shape, re-derived from the ledger, not read from a plan
(classify-fix-ops.mjs --threshold 7). 17 distinct files.
Method. Eight subagents, six items each, classifying against §5's three
conditions. Constraints, all deliberate: read-only (no writes, no commits); no
web or MCP lookups — O2 is defined by requiring no new fact-finding, so the
evidence_quote is the only source evidence a classifier may use; and every
proposal must be obtainable from the file text by deleting characters only.
Because claim matches its file line verbatim in 0 of 202 cases (F4), each
classifier had to open the actual file and locate the real text rather than edit
the restatement. All 46 located it; locator_failed is 0.
| Verdict | N | Share |
|---|---|---|
| O2 candidate | 17 | 37 % |
| O3 | 29 | 63 % |
The result that matters is not the split — it is what blocks the other 29.
| Blocking condition(s) | N |
|---|---|
| condition 2 + condition 3 | 21 |
| condition 3 alone | 3 |
| all three | 5 |
All 29 are foreclosed by condition 3: the source supplies a corrected value,
so the fix is a swap or a rewrite and subtraction would destroy true
information. Condition 1 — "asserts strictly less", the one that sounds like
the hard one — blocks only 5, and never alone. This is F2 (§5) reproduced at
scale: prebuilt-check.us, prebuilt-mortgage.us.closingDisclosure,
Set-DlpCompliancePolicy, jensen_shannon_distance, F300 = 384 GB, DurationMs
/ ResultSignature, Claude 4.5 → 4.6. R8's failing multi-part claims are
predominantly a wrong-value class, not a surplus-specificity class. The design's
reading of R8 in §3 — "where the grounded part stands on its own, the fix is O2"
— holds for a minority of the class.
Machine verification of the returns (V1/V2/V2b, §9.1). 46 of 46 pass V1: the
quoted file text occurs verbatim in the named file, æ/ø/å and markup intact.
16 of the 17 O2 proposals are deletion-only; one is flagged
(feedback-loops-continuous-improvement.md:555, where Automatically add →
Add recapitalises rather than merely deletes). That is a text change, not a
subtraction, and it goes to a human as such.
The 17 are candidates, not admitted edits — and the split inside them is the honest number:
| N | |
|---|---|
conditions 2 and 3 both affirmatively yes |
2 (idx 8, 17) |
at least one condition marked human_must_confirm |
15 |
classifier confidence high |
1 |
So the machine's own reading is that 2 of 46 (4 %) are clean subtractions on
the evidence available, and 15 more are worth a human's time. This is a triage,
not a machine assist. It is not measurement #4 — review throughput still
requires human review sessions that have not happened (§4.4) — but it is the
input #4 needs: it says how many items enter review and in what state, which is
the half of throughput that does not require a stopwatch. The recurring reason for human_must_confirm on condition 3 is
structural and worth recording: the removed material is often true of something
else (Purview really does classify data; the Communication Compliance template
really exists; Redis really is in Norway West) — it is merely false of the
subject the row names. Deleting it is defensible; relocating it may be better.
That is a judgement about the corpus, not about the source, which is exactly why
§5 put conditions 2 and 3 in human hands.
Extrapolation, flagged as such. MULTI_PART_CLAIM is 161 corpus-wide under
R8. At the pilot's 37 % that is ~60 O2 candidates and ~6 clean ones. This is an
extrapolation from one measured sample, not a measurement, and the pilot was
deliberately drawn from the densest files.
9.1 What the machine checks, and what it deliberately does not
scripts/kb-eval/lib/o2-return-check.mjs (25 tests). The checks do not decide
O2 — conditions 2 and 3 stay human by ratified contract. They bound the two
failure modes a human reviewing 46 proposals cannot catch cheaply:
- V1 — the quoted file text must occur verbatim in the file. Catches invented text and silent æ/ø/å transliteration. Applied to every row, not just the O2 ones: an O3 verdict resting on invented text is equally wrong, merely wrong in the safe direction.
- V2 — the remainder must be obtainable by deleting characters only.
- V2b — word-level and case-sensitive, because V2 alone is too weak: deleting a leading word and recapitalising the next passes the character test, since the capital already existed inside the deleted word. This check was added after wave 1 produced exactly that case.
- V3 — schema completeness and verdict/condition coherence.
The raw returns are committed under scripts/kb-eval/data/r11-o2-returns/ —
they are evidence, not regenerable output, same discipline as appendices A and B.
9.2 The two affirmative candidates, hand-verified — one of them fails
Before putting anything in front of a human ratifier, the two candidates the classifier marked affirmative on both human conditions (idx 8, 17) were checked by hand, 2026-08-03. The check was not a re-reading of the source: it was a check of the classifier's own cond-2 reasoning, which cites other lines in the same file as its justification. V1/V2/V2b never touch those citations — they bound the quoted block and the remainder string, nothing else.
Anchoring, verified for all 17. Each file_text_verbatim occurs exactly
once in its file (17/17), so text-anchored editing is unambiguous. The line
numbers are not: line differs from real_line in 9 of 17 records. Any
edit must be anchored on the verbatim block, never on the line number.
idx 17 — holds. Both cross-references check out: line 164 does carry
Start med 0.15, and the policy sample does use score-threshold="0.15" —
at line 238, not 239 as the classifier wrote (an off-by-one in the citation;
the substance stands). The three bands appear nowhere else in the file
(grep for the band values and their labels returns only lines 254-256, the
block itself), so deleting them leaves nothing dangling internally. Two cosmetic
residues for the ratifier, both already visible in the record: the heading keeps
a now-trailing colon, and the **Verified** (Microsoft Learn - Enable semantic caching for LLM APIs) stamp on line 258 afterwards stamps only the direction
statement. Neither makes the remainder false.
idx 8 — cond 2 does not hold, and the candidate is not ratifiable as written. The classifier justified cond 2 on one of the two sub-deletions and never checked the other:
- Sub-deletion (b), the
Optional: Application Insightsbullet: verified. Lines 202-203 do cover Application Insights on their own terms (**Azure Monitor + Application Insights** (Verified)/ "Drift metrics emitteres til Application Insights"), so removing the prerequisite bullet leaves no false implication. - Sub-deletion (a),
eller managed compute cluster: fails. The same file documents**Managed Compute Cluster** (for store volumer)as a real compute option at lines 263-265, with pricing and a usage recommendation. Deleting the alternative from the prerequisites leaves the file asserting serverless Spark as the only compute requirement seventy lines above a cost section that prices the alternative. That is precisely a misleading remainder.
There is a further limit the O2 envelope cannot resolve: cond 3 passes for (a) only because the source does not mention managed compute — not-mentioned passes cond 3 by construction. Whether Azure ML actually permits managed compute for monitoring is a fact question, and O2 is defined by zero fact-gathering. It is an operator call, not a gap to be read harder.
A reduced subtraction — sub-deletion (b) alone — would still be deletion-only
and is not defeated by anything found here. But it is a different remainder
string than the one V2b attested, so it is not machine-clean until
check-o2-returns.mjs is re-run against it. The same rule governs any operator
amendment, including dropping idx 17's trailing colon: amended remainder →
re-run the check before it counts as verified.
The generalisable finding. The classifier judged cond 2 against the source
and against citations it chose itself. It did not systematically judge it
against the rest of the same file. Internal consistency is a cond-2
dimension the wave prompts never assigned, and idx 8 — the single high
confidence record in the set — is the proof that it bites. The remaining 15
candidates have not had this check. It must be run per candidate before any of
them reaches a ratifier; expect it to move some of them.
Appendix A — the 15 admitted proposals, hand-verified
Every proposal the classifier (§4 + context condition) admitted over the whole
not_grounded population, with the verdict that produced §3's table. A later run
that admits a 16th can diff against this list; without it, "9 of 15" is an
unreproducible claim.
| # | File:line | Swap | Type | Verdict |
|---|---|---|---|---|
| 1 | agent-orchestration/agent-evaluation-testing-frameworks.md:56 |
4.1 → 5 (gpt-4.1-mini → gpt-5-mini) |
number | unverified — model identifier; the result is a real model name, but not checked against the source. Not applied. |
| 2 | api-management/logging-analytics-ai-traffic.md:49 |
2023-09-01 → 2025-09-01 |
iso_date | correct — ARM loggers@ api-version bump |
| 3 | azure-ai-services/translator-document-translation.md:162 |
40 → 10 (MB) |
number | unverified — matched on the unit MB, but sync/async limits differ; metric-crossing risk. Not applied. |
| 4 | monitoring-observability/log-analytics-kql-ai-queries.md:617 |
2025-09-01 → 2026-04-01 |
iso_date | correct — api-version= inside a KQL string literal |
| 5 | responsible-ai/responsible-ai-training-awareness.md:77 |
900 → 901 (AI-900 → AI-901) |
number | wrong — certification identifier mutilated |
| 6 | bcdr/cost-analysis-dr-configurations.md:120 |
4 → 5.1 (GPT-4o → GPT-5.1o) |
number | wrong — model identifier mutilated |
| 7 | bcdr/multi-region-azure-openai-deployment.md:316 |
2024-06-01 → 2024-10-01 |
iso_date | correct — api-version= in a management URL |
| 8 | ai-security-engineering/ai-prompt-shield-network.md:309 |
2024-09-01 → 2024-09-15 |
iso_date | correct — Content Safety api-version |
| 9 | ai-security-engineering/content-safety-filter-calibration.md:277 |
2024-10-01 → 2024-10-21 |
iso_date | correct — Azure OpenAI api-version in a curl sample |
| 10 | ai-security-engineering/jailbreak-prevention-production.md:305 |
2024-09-01 → 2024-09-15 |
iso_date | correct — Content Safety api-version in a curl sample |
| 11 | cost-optimization/observability-cost-reduction.md:114 |
3.7.5 → 3.4.0 (Java Agent) |
version | wrong — a downgrade; the quote's version is not the claim's referent |
| 12 | cost-optimization/vector-storage-cost-optimization.md:266 |
2025-09-01 → 2026-04-01 |
iso_date | correct — AI Search api-version |
| 13 | cost-optimization/vector-storage-cost-optimization.md:318 |
2024-02-01 → 2024-10-21 |
iso_date | correct — embeddings api-version |
| 14 | performance-scalability/response-chunking-strategies.md:56 |
4 → 5.1 (gpt-4o → gpt-5.1o) |
number | wrong — model identifier mutilated |
| 15 | performance-scalability/token-per-second-optimization.md:295 |
2024-12-01 → 2025-01-01 |
iso_date | correct — Azure OpenAI api-version |
9 correct · 4 wrong · 2 unverified. All nine correct are iso_date; every
wrong one is a digit inside a product, model or certification identifier, where a
matching prefix (AI-, gpt-, Agent ) satisfies the context condition while
the digit is part of a name rather than a quantity. The two unverified are also
number and are excluded by the same class rule — verifying them costs a source
fetch each and would move the total to at most 11.
Appendix B — the 8 §4b status proposals, hand-verified
Every status proposal the classifier admits over the whole not_grounded
population, judged against the cited source. Same discipline as appendix A: a
later run that admits a ninth can diff against this list, and "5 of 8" is
otherwise an unreproducible claim.
Reproduce: node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 --write, then
read the proposal.type === 'status' items in
scripts/kb-eval/data/r11-pilot-classification.json.
| # | File:line | Swap | Verdict |
|---|---|---|---|
| 1 | agent-orchestration/foundry-agent-service-ga.md:68 |
**Preview** → **GA** |
correct — quote: "hosted agents are generally available"; the row's subject is Hosted agents |
| 2 | agent-orchestration/foundry-agent-service-ga.md:72 |
**GA** → **Preview** |
correct — quote: "Trigger an agent by using Logic Apps (preview)"; the row's subject is the Logic Apps trigger |
| 3 | ai-security-engineering/security-copilot-integration.md:83 |
Public Preview → GA |
correct — quote: "Email and collaboration alert triage capabilities are already generally available (GA)"; the row is the phishing/email triage agent |
| 4 | ai-security-engineering/security-copilot-integration.md:93 |
GA → Preview |
correct — quote is from the agent's own doc page: "This feature is in public preview" |
| 5 | ai-security-engineering/entra-agent-id-zero-trust.md:439 |
Public Preview → GA |
correct — quote: "The Microsoft Entra Agent ID platform is now generally available"; the row's subject is Entra Agent ID (kjerne) |
| 6 | ai-security-engineering/security-copilot-integration.md:94 |
GA → Preview |
unproven — the quote's (Preview) marker belongs to Identity Risk Management Agent, not to Access Review Agent. The judge's prose reason does support preview from a what's-new post, so the outcome is plausibly right; the cited evidence does not establish it. Not applied. |
| 7 | data-engineering/onelake-data-strategy.md:198 |
GA → Preview |
wrong — the quote says Lakehouse | Yes | GA. "Public preview" in the same quote belongs to Eventhouse. Killed by candidate A and B. |
| 8 | ai-security-engineering/owasp-llm-top10-azure-mitigations.md:79 |
GA → Preview |
wrong — the source marks only Response Completeness as preview; the row covers the groundedness/completeness pair, so the edit makes the groundedness half false. The correct fix is to split the row (O2/O3). Killed by candidate B. |
5 correct · 1 unproven · 2 wrong. All five correct ones carry the source phrasing on the row's own subject; all three defects are the referent gap described in §8. No proposal was applied — §4b output is a human review list.
Appendix C — the 17 O2 candidates
Every proposal the prose classification admitted over the 46, with the two
human-judged conditions as the classifier left them. Same discipline as
appendices A and B: without this list, "17 of 46" is an unreproducible claim.
confirm = the classifier marked the condition human_must_confirm, i.e. it
could not settle it on the evidence available and is handing it over — not a
defect, it is the contract.
The full records, including each proposal's verbatim file text and the exact
remainder, are in scripts/kb-eval/data/r11-o2-returns/.
| # | File:line | Failing sub-assertion | Cond 2 | Cond 3 | Confidence |
|---|---|---|---|---|---|
| 7 | ms-ai-engineering/azure-ai-services/document-intelligence-prebuilt-models.md:79 |
The third table row presenting prebuilt-document (General Document) as a current basic model — the source states the general document model is no l… |
yes | confirm | medium |
| 8 | ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:191 |
Two sub-assertions: (a) eller managed compute cluster as an alternative compute option — the how-to page and the monitor schema require a Spark poo… |
yes | yes | high |
| 9 | ms-ai-engineering/mlops-genaiops/data-drift-monitoring-detection.md:218 |
The second sentence, Støtter også drift detection for grounding data i RAG scenarios. — the canonical observability page lists only Evaluation, Mon… |
yes | confirm | medium |
| 14 | ms-ai-engineering/mlops-genaiops/feedback-loops-continuous-improvement.md:555 |
Two sub-assertions: 'SharePoint' as a feedback storage service, and the word 'Automatically' in 'Automatically add reviewed samples to training set' … | confirm | yes | medium |
| 17 | ms-ai-engineering/rag-architecture/rag-caching-optimization.md:254 |
The three-band rubric (0.1-0.2 strict / 0.3-0.5 balanced / 0.6-0.8 liberal) — undocumented, and the two upper bands contradict the source's warning t… | yes | yes | medium |
| 18 | ms-ai-engineering/rag-architecture/rag-caching-optimization.md:29 |
The list item 'Azure AI Search (built-in caching av search results)' — the source states each query operates on the current index view with no cachin… | confirm | yes | medium |
| 19 | ms-ai-engineering/rag-architecture/rag-caching-optimization.md:297 |
The bullet "Automatic indexing av vectors" — the judge states vector indexes must be declared explicitly in the indexing policy (only at container cr… | confirm | yes | medium |
| 26 | ms-ai-governance/responsible-ai/transparency-documentation-standards.md:117 |
Items 4 (Error analysis) and 5 (Counterfactual analysis) are listed as Responsible AI Scorecard components, but the canonical scorecard segment enume… | confirm | yes | medium |
| 27 | ms-ai-governance/responsible-ai/transparency-documentation-standards.md:426 |
The 'Chat interface' row (a "Powered by AI" badge in the chat window) and the 'Plugin actions' row (confirmation prompts before sensitive actions) ar… | yes | confirm | medium |
| 28 | ms-ai-governance/responsible-ai/transparency-documentation-standards.md:83 |
The second and third bullets — that Hugging Face model cards are synchronised automatically, and that a template exists for generating model cards fo… | yes | confirm | medium |
| 31 | ms-ai-security/ai-security-engineering/ai-incident-response-procedures.md:139 |
The legalHold object is given a field named "enabled"; the Storage API's LegalHold model exposes tags and hasLegalHold, so the literal field name "en… | yes | confirm | medium |
| 33 | ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:211 |
Two parts attributed to Defender for Cloud AI Security Posture Management that the AISPM page does not support: the discovery mechanism "(via Azure R… | yes | confirm | medium |
| 36 | ms-ai-security/ai-security-engineering/ai-threat-modeling-stride.md:38 |
The "/indiscriminate" qualifier, which extends the Tampering placement and the Critical severity to indiscriminate data poisoning; the source gives t… | yes | confirm | medium |
| 38 | ms-ai-security/ai-security-engineering/data-leakage-prevention-ai.md:396 |
The listing of 'DSPM for AI - Unethical behavior in AI apps' and 'DSPM for AI - Protect sensitive data from Copilot processing' as Insider Risk Manag… | yes | confirm | medium |
| 40 | ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:133 |
The third bullet 'CVE severity mapping' presented as a category of alert that dependency scanning generates; severity is a property of an alert, … | yes | confirm | medium |
| 42 | ms-ai-security/ai-security-engineering/supply-chain-security-ai-models.md:200 |
The second bullet presenting the HuggingFace Registry as a Microsoft channel for verified models with provenance tracking; the source calls it a comm… | yes | confirm | medium |
| 45 | ms-ai-security/cost-optimization/semantic-caching-patterns.md:436 |
The '/West' half of the region pair, i.e. the standing implication that Azure OpenAI can be deployed in Norway West. | yes | confirm | medium |
2 affirmative on both conditions (8, 17) · 15 needing a human call. Item 14 additionally carries a machine flag: its remainder recapitalises rather than deletes (§9.1, V2b), so it is a text change and must be reviewed as one.
⚠️ The cond 2 = yes column is the classifier's claim, not a verified fact.
Hand-checking the two affirmative rows (§9.2) falsified idx 8: half its cond-2
justification was never checked and does not hold, because the same file
documents the deleted alternative in its cost section. Read every yes in this
column as "the classifier believed this", and verify against the rest of the
file before ratifying. Only idx 17 has survived that check so far.
Reproduce the tally and the machine checks:
node scripts/kb-eval/check-o2-returns.mjs.