ms-ai-architect/docs/r11-pilot-results.md

R11 pilot results — measured, 2026-08-03

The §10 acceptance measurement of docs/r11-tiered-fix-design.md, run against the live ledger. No KB file was edited and no ledger record was written.

Instrument: scripts/kb-eval/lib/fix-op.mjs (+ tests/kb-eval/test-fix-op-classify.test.mjs, 49 tests after §4b) driven by scripts/kb-eval/classify-fix-ops.mjs. The classifier is the O1 driver with writes disabled — it constructs the swap and checks the §4 invariant, so measurements 1 and 3 come out of the mechanism that would later touch the corpus, not out of a proxy heuristic.

Artefact: scripts/kb-eval/data/r11-pilot-classification.json (untracked, regenerable; per-flag records so the run can be re-analysed without re-running). It holds the pilot run — node scripts/kb-eval/classify-fix-ops.mjs --write. Every corpus-wide figure below is from --threshold 1, and the per-table reproduce command is stated where it is used.

Two different 202s. This population is 202 flags. §3's "202 flags whose claim and quote contain a numeric token" is a different 202, measured over the full 712-flag population. They are unrelated.

---

1. The headline

Nine provable, correct value swaps exist in the entire 776-flag not_grounded population — 1.2 %. The machine half of the R11 tiering buys nine edits. Every other flag needs a human.

Population	Files	Flags	O1 admitted	O1 hand-verified correct
Pilot (not_grounded ≥ 7)	24	202	2	2
Whole not_grounded corpus	218	776	15	9

These are numeric-path figures, and they stay that way. §4b (the status synonym table) was implemented afterwards and adds a separate class with its own hand-verification — see §8. A run today prints O1 = 7 (pilot) and 23 (corpus) because the status proposals are included in the total; the numeric line above is unchanged and is still what s4_as_written compares against.

Reproduce: node scripts/kb-eval/classify-fix-ops.mjs (pilot) and node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 (corpus). The nine are enumerated with verdicts in appendix A — that hand-verification is the only thing separating 9 from 15, so it is recorded rather than left in a session transcript.

This is the answer §10 asked for, and it is materially worse than the design assumed: "If the split is materially worse than assumed, that is known after one session rather than after ten."

2. §4 as written is not sufficient — measured, not argued

§4 claims its invariant is "deliberately stronger than human review at scale." It is not. Run exactly as specified over the pilot, it admitted 6 swaps, of which 4 were wrong — precision 2/6:

Proposed swap	Why it is wrong
30-dagers → 24-dagers	Unit crossing. The quote says 24 hours.
3000 requests/sekund → 50	Metric crossing. The quote is a query throttle per index; the claim is an indexing rate per replica.
Microsoft Agent 365 → Agent 7	Identifier mutilated. The 7 was harvested out of E7.
text-embedding-ada-002 → ada-2	Identifier mutilated. The 2 came from a dimensions column.

The defect is structural, not incidental. §4 constrains where the new value came from (verbatim in the cited quote) and what the edit looks like (one line, rest byte-identical). It constrains nothing about whether the two tokens denote the same quantity. Same-type-and-provenance is not same-referent.

The added condition

contextCorresponds() requires the token to sit under the same label or the same trailing unit on both sides. It is deliberately lexical, with no translation table: dokumenter is not taught to equal documents, because a synonym table introduces a new fact source and is an operator decision, not an engineering one. Consequence, measured: a swap is provable essentially only where the context is language-neutral — a URL, a code sample, a parameter key.

3. The condition is necessary but still not sufficient

Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15 splits them cleanly by token type:

Reproduce: node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 (the persisted artefact is the pilot run — the corpus-wide tables in §1 and §3 come from this threshold-1 run). Per-proposal verdicts: appendix A.

Token type	Proposals	Correct	Wrong	Unverified	Failure mode
iso_date	9	9	0	0	— every one is an api-version= bump in a URL or code sample
number	5	0	3	2	AI-900 → AI-901, gpt-4o → gpt-5.1o (×2)
version	1	0	1	0	Java agent 3.7.5 → 3.4.0 — a downgrade

A matching identifier prefix (AI-, gpt-) satisfies the context condition while the digit is part of a name, not a quantity. Only iso_date survives hand-verification, and the report marks it as the sole recommended class (o1_recommended). number and version proposals must not be applied.

4. The four §10 measurements

1. O1 / O2 / O3 split. O1 = 2/202 on the pilot (9/776 corpus-wide, safe class only). O2 is undetermined — it does not exist as a class until §5 is ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.

2. How much of R8 resolves as O2. Not answered, and not answerable by machine. R8 is 87/202 on the pilot (366/776 corpus-wide) and yields zero O1. Of the pilot's 87, 46 are structural enumerations — the O2 candidate shape. Which of them subtract cleanly turns on the judge's prose reason, and no regex reads prose. This needs prose classification (see §6).

3. O1 abort rate: 99 % (200/202). Typed, because "99 %" alone is not actionable:

   Code	Pilot	Class
   MULTI_PART_CLAIM	96 (47.5 %)	intrinsic — not a value swap at all
   MULTI_VALUE_TOKEN	29	intrinsic
   NO_VALUE_TOKEN	28	intrinsic — the claim asserts prose
   STATUS_SYNONYM	15	operator question (§6.2)
   NOT_VERBATIM	15	intrinsic
   LOCATOR_AMBIGUOUS	7	fixable engineering gap
   MULTI_REPLACEMENT	6	intrinsic
   CONTEXT_MISMATCH	4	intrinsic — these are the 4 wrong edits above

   Only 7 of 200 aborts (3.5 %) are a fixable engineering gap. More locator engineering cannot move the O1 number materially.

4. Review throughput per class. NOT MEASURED. It requires human review sessions, which have not happened. Recording it as measured would be false.

5. Two further findings

F1 — subtraction can leave a misleading remainder. §5 argues O2 "cannot introduce a new error, because it asserts strictly less." True of the sentence, false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research

• Bing) er GA (juni 2025)"* where the source says the tool is deprecated. Subtracting er GA (juni 2025) leaves the tool standing in a list of available tools. Strictly less asserted; still misleading. O2 therefore still requires a human to look at the remainder — cheaper than O3 (no fact-finding) but not mechanical.

F2 — subtraction can destroy true information. Real case: a list of seven prebuilt model IDs where the judge found six correct and prebuilt-check wrong — the real ID is prebuilt-check.us. Subtraction drops a model that exists; the correct fix is a swap. Subtraction is not the safe default everywhere.

F3 — disposition carries zero information. It is outdated on 202 of 202 flags. docs/r11-flag-format-2026-07.md specifies not_grounded → {outdated, wrong} with "the human assigns which at R11", but the pass hard-assigned outdated. Do not use it as a classifier signal. Spec/data divergence, recorded.

F4 — claims are not file text. claim is an LLM-extracted, translated restatement: 0 of 202 match their file line verbatim, and 188 share no 40-char run with it. For table claims, line points at the header, not the value. This is why the locator exists at all, and why it searches the enclosing block rather than the line.

6. Operator decisions — ALL THREE RATIFIED 2026-08-03

All three were put to the operator with the recommendations below and all three were accepted as recommended. The contract text now lives in docs/r11-tiered-fix-design.md §4a/§4b/§5; this section records what was asked and what the answer was.

None of the three is implemented yet. The classifier still aborts STATUS_SYNONYM and still routes every non-O1 item to O3. A later session builds against the ratified contract — it must not assume the code already honours it.

1. Ratify O2 (§5)? → RATIFIED, with the remainder check (not as a blanket rule), exactly as F1/F2 above argued. Contract: design doc §5, three conditions, human-confirmed.
2. Amend §4 with a ratified synonym table? → RATIFIED, narrow and closed. Contract: design doc §4b — four label rows, closed table, complete-label-only, corpus-side value with the file's own markup preserved. Unlocks up to 54 corpus-wide flags.
3. Is O1 worth building at all? → KEPT, locked to iso_date. Contract: design doc §4a condition 5 — a driver may apply iso_date proposals and must never apply number or version ones. Nine edits corpus-wide.

7. What this does not change

The design's core reading survives: the expensive half (locating the source, reading it, extracting the deciding passage) was already paid for by the judge pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot falsifies is the assumption that a meaningful share of that evidence converts into machine-provable edits. It does not. R11 is a human review programme with a nine-item machine assist, and its leverage lies entirely in the O2 decision.

---

8. §4b implemented — the status class measured, 2026-08-03

The ratified synonym table (docs/r11-tiered-fix-design.md §4b) is implemented in fix-op.mjs and the class is measured. Reproduce: node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 (status_synonym block).

Population	STATUS_SYNONYM flags	Proven by §4b	Still aborting
Pilot (not_grounded ≥ 7)	15	5	10
Whole not_grounded corpus	54	8	46

Why the other 46 abort, corpus-wide — this sub-distribution is the actionable part, because the top-level STATUS_SYNONYM count alone says nothing:

Reason	N	What it means
NO_COMPLETE_FILE_LABEL	27	the file writes the status inside a sentence — (preview) in a list item, **DSPM (preview):**, "[Preview]: …" in a JSON string. Constraint 2 refuses these, correctly.
NO_SOURCE_STATUS	15	the cited quote carries no listed lifecycle phrasing at all — the flag was never a status swap.
SOURCE_STATUS_AMBIGUOUS	2	the quote asserts two different rows (e.g. "…is now generally available. Partner solutions remain in preview.").
FILE_ALREADY_MATCHES	2	file and source agree; the mismatch was in the LLM-extracted claim, not in the corpus.

The class is REVIEW-grade, not apply-grade — 5 of 8 correct. All eight were hand-judged against the cited source (appendix B). Three defects, all one family: §4b binds the table, the completeness of the file label and the written value, and nothing about whether the source phrasing refers to the row's own subject. That is the same provenance-without-referent defect that falsified §4 (§2), now reproduced in the status class. status is therefore deliberately absent from o1_recommended: the machine writes nothing, and every proposal reaches a human.

Two candidate conditions, costed over the eight

Neither is implemented — extending a table the operator ratified as closed is an operator decision, exactly as condition 5 was in §4a. Both are pure gain on this population (they kill wrong proposals and no correct one), which is the number the decision needs:

Candidate	Kills	Correct proposals lost
A — count a bare GA in the quote as a GA-row source phrasing, so a quote saying both GA and "public preview" becomes ambiguous	1 (onelake:198)	0
B — abort when the quote is a multi-entity enumeration (≥ 2 pipes, or a numbered list)	2 (onelake:198, owasp:79)	0

B subsumes A on these eight. Neither catches security-copilot-integration.md:94, where the quote is prose and the (Preview) marker simply belongs to a different agent. A stricter referent-name condition (the row's subject must appear in the quote) would catch it — and would also kill two correct proposals (:83, :93), where the source names the capability rather than the agent. That trade is real and is why this is put to the operator rather than shipped.

Appendix A — the 15 admitted proposals, hand-verified

Every proposal the classifier (§4 + context condition) admitted over the whole not_grounded population, with the verdict that produced §3's table. A later run that admits a 16th can diff against this list; without it, "9 of 15" is an unreproducible claim.

#	File:line	Swap	Type	Verdict
1	agent-orchestration/agent-evaluation-testing-frameworks.md:56	4.1 → 5 (gpt-4.1-mini → gpt-5-mini)	number	unverified — model identifier; the result is a real model name, but not checked against the source. Not applied.
2	api-management/logging-analytics-ai-traffic.md:49	2023-09-01 → 2025-09-01	iso_date	correct — ARM loggers@ api-version bump
3	azure-ai-services/translator-document-translation.md:162	40 → 10 (MB)	number	unverified — matched on the unit MB, but sync/async limits differ; metric-crossing risk. Not applied.
4	monitoring-observability/log-analytics-kql-ai-queries.md:617	2025-09-01 → 2026-04-01	iso_date	correct — api-version= inside a KQL string literal
5	responsible-ai/responsible-ai-training-awareness.md:77	900 → 901 (AI-900 → AI-901)	number	wrong — certification identifier mutilated
6	bcdr/cost-analysis-dr-configurations.md:120	4 → 5.1 (GPT-4o → GPT-5.1o)	number	wrong — model identifier mutilated
7	bcdr/multi-region-azure-openai-deployment.md:316	2024-06-01 → 2024-10-01	iso_date	correct — api-version= in a management URL
8	ai-security-engineering/ai-prompt-shield-network.md:309	2024-09-01 → 2024-09-15	iso_date	correct — Content Safety api-version
9	ai-security-engineering/content-safety-filter-calibration.md:277	2024-10-01 → 2024-10-21	iso_date	correct — Azure OpenAI api-version in a curl sample
10	ai-security-engineering/jailbreak-prevention-production.md:305	2024-09-01 → 2024-09-15	iso_date	correct — Content Safety api-version in a curl sample
11	cost-optimization/observability-cost-reduction.md:114	3.7.5 → 3.4.0 (Java Agent)	version	wrong — a downgrade; the quote's version is not the claim's referent
12	cost-optimization/vector-storage-cost-optimization.md:266	2025-09-01 → 2026-04-01	iso_date	correct — AI Search api-version
13	cost-optimization/vector-storage-cost-optimization.md:318	2024-02-01 → 2024-10-21	iso_date	correct — embeddings api-version
14	performance-scalability/response-chunking-strategies.md:56	4 → 5.1 (gpt-4o → gpt-5.1o)	number	wrong — model identifier mutilated
15	performance-scalability/token-per-second-optimization.md:295	2024-12-01 → 2025-01-01	iso_date	correct — Azure OpenAI api-version

9 correct · 4 wrong · 2 unverified. All nine correct are iso_date; every wrong one is a digit inside a product, model or certification identifier, where a matching prefix (AI-, gpt-, Agent ) satisfies the context condition while the digit is part of a name rather than a quantity. The two unverified are also number and are excluded by the same class rule — verifying them costs a source fetch each and would move the total to at most 11.

Appendix B — the 8 §4b status proposals, hand-verified

Every status proposal the classifier admits over the whole not_grounded population, judged against the cited source. Same discipline as appendix A: a later run that admits a ninth can diff against this list, and "5 of 8" is otherwise an unreproducible claim.

Reproduce: node scripts/kb-eval/classify-fix-ops.mjs --threshold 1 --write, then read the proposal.type === 'status' items in scripts/kb-eval/data/r11-pilot-classification.json.

#	File:line	Swap	Verdict
1	agent-orchestration/foundry-agent-service-ga.md:68	**Preview** → **GA**	correct — quote: "hosted agents are generally available"; the row's subject is Hosted agents
2	agent-orchestration/foundry-agent-service-ga.md:72	**GA** → **Preview**	correct — quote: "Trigger an agent by using Logic Apps (preview)"; the row's subject is the Logic Apps trigger
3	ai-security-engineering/security-copilot-integration.md:83	Public Preview → GA	correct — quote: "Email and collaboration alert triage capabilities are already generally available (GA)"; the row is the phishing/email triage agent
4	ai-security-engineering/security-copilot-integration.md:93	GA → Preview	correct — quote is from the agent's own doc page: "This feature is in public preview"
5	ai-security-engineering/entra-agent-id-zero-trust.md:439	Public Preview → GA	correct — quote: "The Microsoft Entra Agent ID platform is now generally available"; the row's subject is Entra Agent ID (kjerne)
6	ai-security-engineering/security-copilot-integration.md:94	GA → Preview	unproven — the quote's (Preview) marker belongs to Identity Risk Management Agent, not to Access Review Agent. The judge's prose reason does support preview from a what's-new post, so the outcome is plausibly right; the cited evidence does not establish it. Not applied.
7	data-engineering/onelake-data-strategy.md:198	GA → Preview	wrong — the quote says Lakehouse | Yes | GA. "Public preview" in the same quote belongs to Eventhouse. Killed by candidate A and B.
8	ai-security-engineering/owasp-llm-top10-azure-mitigations.md:79	GA → Preview	wrong — the source marks only Response Completeness as preview; the row covers the groundedness/completeness pair, so the edit makes the groundedness half false. The correct fix is to split the row (O2/O3). Killed by candidate B.

5 correct · 1 unproven · 2 wrong. All five correct ones carry the source phrasing on the row's own subject; all three defects are the referent gap described in §8. No proposal was applied — §4b output is a human review list.