docs(ms-ai-architect): R11 pilot — appendiks A med de 15 hånd-dommene + reprodusér-kommandoer [skip-docs]

To durabilitets-hull i resultatdokumentet, begge funnet ved gjennomgang:

1. Korpus-tallene (218 filer / 776 flagg / 15 sluppet / 9 korrekte) kom fra en
   --threshold 1-kjøring, men det persisterte artefaktet er PILOT-kjøringen. En
   senere økt kunne ikke regenerere §1/§3 uten å vite om flagget. Reprodusér-
   kommando står nå ved hver tabell.

2. Hånd-verifiseringen av de 15 var det eneste som skiller 9 fra 15, og den lå
   kun i en økt-transkripsjon. Appendiks A fører hver enkelt: file:line, swap,
   type, dom, begrunnelse. En senere kjøring som slipper gjennom en 16. kan nå
   diffes mot lista.

RETTELSE i §3-tabellen: `number`-raden sto med «Correct 2». Det var ikke hånd-
verifisert — de to (gpt-4.1-mini → gpt-5-mini, 40 → 10 MB) er UVERIFISERTE, ikke
bekreftet korrekte. Riktig fordeling: 9 korrekte · 4 gale · 2 uverifiserte.
Hovedtallet (9) er uendret; det var alltid iso_date alene.

docs/r11-pilot-results.md

@@ -11,6 +11,9 @@ touch the corpus, not out of a proxy heuristic.
 
 Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
 regenerable; per-flag records so the run can be re-analysed without re-running).
+It holds the **pilot** run — `node scripts/kb-eval/classify-fix-ops.mjs --write`.
+Every corpus-wide figure below is from `--threshold 1`, and the per-table
+reproduce command is stated where it is used.
 
 > **Two different 202s.** This population is 202 flags. §3's "202 flags whose
 > claim *and* quote contain a numeric token" is a different 202, measured over
@@ -29,6 +32,11 @@ other flag needs a human.
 | Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
 | Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
 
+Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
+`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
+enumerated with verdicts in appendix A — that hand-verification is the only thing
+separating 9 from 15, so it is recorded rather than left in a session transcript.
+
 This is the answer §10 asked for, and it is materially worse than the design
 assumed: *"If the split is materially worse than assumed, that is known after one
 session rather than after ten."*
@@ -65,11 +73,15 @@ the context is language-neutral — a URL, a code sample, a parameter key.
 Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
 splits them cleanly by token type:
 
-| Token type | Proposals | Correct | Failure mode |
+Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1`
-|---|---|---|---|
+(the persisted artefact is the **pilot** run — the corpus-wide tables in §1 and §3
-| `iso_date` | 9 | **9** | — every one is an `api-version=` bump in a URL or code sample |
+come from this threshold-1 run). Per-proposal verdicts: appendix A.
-| `number` | 5 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
+
-| `version` | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
+| Token type | Proposals | Correct | Wrong | Unverified | Failure mode |
+|---|---|---|---|---|---|
+| `iso_date` | 9 | **9** | 0 | 0 | — every one is an `api-version=` bump in a URL or code sample |
+| `number` | 5 | 0 | 3 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
+| `version` | 1 | 0 | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
 
 A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
 while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
@@ -153,3 +165,37 @@ pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
 falsifies is the assumption that a meaningful share of that evidence converts into
 machine-provable edits. It does not. R11 is a human review programme with a
 nine-item machine assist, and its leverage lies entirely in the O2 decision.
+
+---
+
+## Appendix A — the 15 admitted proposals, hand-verified
+
+Every proposal the classifier (§4 + context condition) admitted over the whole
+`not_grounded` population, with the verdict that produced §3's table. A later run
+that admits a 16th can diff against this list; without it, "9 of 15" is an
+unreproducible claim.
+
+| # | File:line | Swap | Type | Verdict |
+|---|---|---|---|---|
+| 1 | `agent-orchestration/agent-evaluation-testing-frameworks.md:56` | `4.1` → `5` (`gpt-4.1-mini` → `gpt-5-mini`) | number | **unverified** — model identifier; the result is a real model name, but not checked against the source. Not applied. |
+| 2 | `api-management/logging-analytics-ai-traffic.md:49` | `2023-09-01` → `2025-09-01` | iso_date | **correct** — ARM `loggers@` api-version bump |
+| 3 | `azure-ai-services/translator-document-translation.md:162` | `40` → `10` (MB) | number | **unverified** — matched on the unit `MB`, but sync/async limits differ; metric-crossing risk. Not applied. |
+| 4 | `monitoring-observability/log-analytics-kql-ai-queries.md:617` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — `api-version=` inside a KQL string literal |
+| 5 | `responsible-ai/responsible-ai-training-awareness.md:77` | `900` → `901` (`AI-900` → `AI-901`) | number | **wrong** — certification identifier mutilated |
+| 6 | `bcdr/cost-analysis-dr-configurations.md:120` | `4` → `5.1` (`GPT-4o` → `GPT-5.1o`) | number | **wrong** — model identifier mutilated |
+| 7 | `bcdr/multi-region-azure-openai-deployment.md:316` | `2024-06-01` → `2024-10-01` | iso_date | **correct** — `api-version=` in a management URL |
+| 8 | `ai-security-engineering/ai-prompt-shield-network.md:309` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version |
+| 9 | `ai-security-engineering/content-safety-filter-calibration.md:277` | `2024-10-01` → `2024-10-21` | iso_date | **correct** — Azure OpenAI api-version in a curl sample |
+| 10 | `ai-security-engineering/jailbreak-prevention-production.md:305` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version in a curl sample |
+| 11 | `cost-optimization/observability-cost-reduction.md:114` | `3.7.5` → `3.4.0` (Java Agent) | version | **wrong** — a downgrade; the quote's version is not the claim's referent |
+| 12 | `cost-optimization/vector-storage-cost-optimization.md:266` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — AI Search api-version |
+| 13 | `cost-optimization/vector-storage-cost-optimization.md:318` | `2024-02-01` → `2024-10-21` | iso_date | **correct** — embeddings api-version |
+| 14 | `performance-scalability/response-chunking-strategies.md:56` | `4` → `5.1` (`gpt-4o` → `gpt-5.1o`) | number | **wrong** — model identifier mutilated |
+| 15 | `performance-scalability/token-per-second-optimization.md:295` | `2024-12-01` → `2025-01-01` | iso_date | **correct** — Azure OpenAI api-version |
+
+**9 correct · 4 wrong · 2 unverified.** All nine correct are `iso_date`; every
+wrong one is a digit inside a product, model or certification identifier, where a
+matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
+the digit is part of a name rather than a quantity. The two unverified are also
+`number` and are excluded by the same class rule — verifying them costs a source
+fetch each and would move the total to at most 11.