open/ms-ai-architect
1
0
Fork 0
Code Issues Releases Activity

docs(ms-ai-architect): R11 pilot — appendiks A med de 15 hånd-dommene + reprodusér-kommandoer [skip-docs]

Browse source 
To durabilitets-hull i resultatdokumentet, begge funnet ved gjennomgang:

1. Korpus-tallene (218 filer / 776 flagg / 15 sluppet / 9 korrekte) kom fra en
   --threshold 1-kjøring, men det persisterte artefaktet er PILOT-kjøringen. En
   senere økt kunne ikke regenerere §1/§3 uten å vite om flagget. Reprodusér-
   kommando står nå ved hver tabell.

2. Hånd-verifiseringen av de 15 var det eneste som skiller 9 fra 15, og den lå
   kun i en økt-transkripsjon. Appendiks A fører hver enkelt: file:line, swap,
   type, dom, begrunnelse. En senere kjøring som slipper gjennom en 16. kan nå
   diffes mot lista.

RETTELSE i §3-tabellen: `number`-raden sto med «Correct 2». Det var ikke hånd-
verifisert — de to (gpt-4.1-mini → gpt-5-mini, 40 → 10 MB) er UVERIFISERTE, ikke
bekreftet korrekte. Riktig fordeling: 9 korrekte · 4 gale · 2 uverifiserte.
Hovedtallet (9) er uendret; det var alltid iso_date alene.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 16:32:14 +02:00
commit 064078b8c2
1 changed files with 51 additions and 5 deletions
Download patch file Download diff file Expand all files Collapse all files

56
docs/r11-pilot-results.md
View file

@ -11,6 +11,9 @@ touch the corpus, not out of a proxy heuristic.
Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
regenerable; per-flag records so the run can be re-analysed without re-running).
It holds the **pilot** run — `node scripts/kb-eval/classify-fix-ops.mjs --write`.
Every corpus-wide figure below is from `--threshold 1`, and the per-table
reproduce command is stated where it is used.
> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
> claim *and* quote contain a numeric token" is a different 202, measured over
@ -29,6 +32,11 @@ other flag needs a human.
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs` (pilot) and
`node scripts/kb-eval/classify-fix-ops.mjs --threshold 1` (corpus). The nine are
enumerated with verdicts in appendix A — that hand-verification is the only thing
separating 9 from 15, so it is recorded rather than left in a session transcript.
This is the answer §10 asked for, and it is materially worse than the design
assumed: *"If the split is materially worse than assumed, that is known after one
session rather than after ten."*
@ -65,11 +73,15 @@ the context is language-neutral — a URL, a code sample, a parameter key.
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
splits them cleanly by token type:
| Token type | Proposals | Correct | Failure mode |
|---|---|---|---|
| `iso_date` | 9 | **9** | — every one is an `api-version=` bump in a URL or code sample |
| `number` | 5 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
| `version` | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
Reproduce: `node scripts/kb-eval/classify-fix-ops.mjs --threshold 1`
(the persisted artefact is the **pilot** run — the corpus-wide tables in §1 and §3
come from this threshold-1 run). Per-proposal verdicts: appendix A.
| Token type | Proposals | Correct | Wrong | Unverified | Failure mode |
|---|---|---|---|---|---|
| `iso_date` | 9 | **9** | 0 | 0 | — every one is an `api-version=` bump in a URL or code sample |
| `number` | 5 | 0 | 3 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
| `version` | 1 | 0 | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
@ -153,3 +165,37 @@ pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
falsifies is the assumption that a meaningful share of that evidence converts into
machine-provable edits. It does not. R11 is a human review programme with a
nine-item machine assist, and its leverage lies entirely in the O2 decision.
---
## Appendix A — the 15 admitted proposals, hand-verified
Every proposal the classifier (§4 + context condition) admitted over the whole
`not_grounded` population, with the verdict that produced §3's table. A later run
that admits a 16th can diff against this list; without it, "9 of 15" is an
unreproducible claim.
| # | File:line | Swap | Type | Verdict |
|---|---|---|---|---|
| 1 | `agent-orchestration/agent-evaluation-testing-frameworks.md:56` | `4.1` → `5` (`gpt-4.1-mini` → `gpt-5-mini`) | number | **unverified** — model identifier; the result is a real model name, but not checked against the source. Not applied. |
| 2 | `api-management/logging-analytics-ai-traffic.md:49` | `2023-09-01` → `2025-09-01` | iso_date | **correct** — ARM `loggers@` api-version bump |
| 3 | `azure-ai-services/translator-document-translation.md:162` | `40` → `10` (MB) | number | **unverified** — matched on the unit `MB`, but sync/async limits differ; metric-crossing risk. Not applied. |
| 4 | `monitoring-observability/log-analytics-kql-ai-queries.md:617` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — `api-version=` inside a KQL string literal |
| 5 | `responsible-ai/responsible-ai-training-awareness.md:77` | `900` → `901` (`AI-900` → `AI-901`) | number | **wrong** — certification identifier mutilated |
| 6 | `bcdr/cost-analysis-dr-configurations.md:120` | `4` → `5.1` (`GPT-4o` → `GPT-5.1o`) | number | **wrong** — model identifier mutilated |
| 7 | `bcdr/multi-region-azure-openai-deployment.md:316` | `2024-06-01` → `2024-10-01` | iso_date | **correct** — `api-version=` in a management URL |
| 8 | `ai-security-engineering/ai-prompt-shield-network.md:309` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version |
| 9 | `ai-security-engineering/content-safety-filter-calibration.md:277` | `2024-10-01` → `2024-10-21` | iso_date | **correct** — Azure OpenAI api-version in a curl sample |
| 10 | `ai-security-engineering/jailbreak-prevention-production.md:305` | `2024-09-01` → `2024-09-15` | iso_date | **correct** — Content Safety api-version in a curl sample |
| 11 | `cost-optimization/observability-cost-reduction.md:114` | `3.7.5` → `3.4.0` (Java Agent) | version | **wrong** — a downgrade; the quote's version is not the claim's referent |
| 12 | `cost-optimization/vector-storage-cost-optimization.md:266` | `2025-09-01` → `2026-04-01` | iso_date | **correct** — AI Search api-version |
| 13 | `cost-optimization/vector-storage-cost-optimization.md:318` | `2024-02-01` → `2024-10-21` | iso_date | **correct** — embeddings api-version |
| 14 | `performance-scalability/response-chunking-strategies.md:56` | `4` → `5.1` (`gpt-4o` → `gpt-5.1o`) | number | **wrong** — model identifier mutilated |
| 15 | `performance-scalability/token-per-second-optimization.md:295` | `2024-12-01` → `2025-01-01` | iso_date | **correct** — Azure OpenAI api-version |
**9 correct · 4 wrong · 2 unverified.** All nine correct are `iso_date`; every
wrong one is a digit inside a product, model or certification identifier, where a
matching prefix (`AI-`, `gpt-`, `Agent `) satisfies the context condition while
the digit is part of a name rather than a quantity. The two unverified are also
`number` and are excluded by the same class rule — verifying them costs a source
fetch each and would move the total to at most 11.