feat(ms-ai-architect): R11 pilot kjørt — §4-invarianten falsifisert, 9 provbare swaps i hele korpuset [skip-docs]
§10-målingen er gjennomført mot live ledger (243 records). Ingen KB-fil er redigert og ingen ledger-record er skrevet — §8s single-writer-state er urørt. Instrumentet ER O1-driveren med writes av (scripts/kb-eval/lib/fix-op.mjs, 30 tester). Måling #1 og #3 kommer dermed ut av mekanismen som senere skal ta på korpuset, ikke ut av en proxy-heuristikk. HOVEDFUNN — §4 som skrevet er utilstrekkelig, målt: Kjørt eksakt som spesifisert slapp den gjennom 6 swaps på piloten, hvorav 4 er GALE editer (presisjon 2/6): - 30-dagers → 24-dagers (enhets-kryssing: kilden sier 24 HOURS) - 3000 req/sek → 50 (metrikk-kryssing: query-throttle vs indexing-rate) - Microsoft Agent 365 → 7 (identifikator lemlestet, «7» høstet fra «E7») - text-embedding-ada-002 → ada-2 (identifikator lemlestet) §4 binder proveniensen til verdien og formen på editen — ingenting om at de to tokenene betegner SAMME STØRRELSE. Påstanden om at invarianten er «deliberately stronger than human review at scale» holder ikke. TILLEGG: contextCorresponds() krever samme label eller samme enhet på begge sider. Bevisst leksikalsk, UTEN oversettelsestabell — «dokumenter» læres ikke å være «documents», fordi en synonymtabell innfører en ny faktakilde og er en operatørbeslutning. Konsekvensen er målt: swap er provbar praktisk talt bare der konteksten er språknøytral (URL, kodeeksempel, parameternøkkel). TALLENE: - Pilot (≥7): 24 filer / 202 flagg → O1 = 2 (1,0 %), O3 = 200 (99,0 %) - Hele korpuset: 218 filer / 776 flagg → 15 sluppet gjennom, 9 korrekte - Kun iso_date (api-version-bump) overlever hånd-verifisering: 9/9. number/version lemlester identifikatorer (AI-900 → AI-901, gpt-4o → gpt-5.1o ×2, Java-agent 3.7.5 → 3.4.0 = nedgradering) og skal IKKE påføres. - Kun 7 av 200 aborter (3,5 %) er en fiksbar engineering-gap. Mer locator- arbeid kan ikke flytte O1-tallet vesentlig. Måling #2 (R8 → O2) er IKKE besvart og kan ikke besvares maskinelt: R8 gir null O1, og hvilke av de 46 enumerasjonene som subtraherer rent avhenger av dommerens PROSA-reason. Måling #4 (review-throughput) er ikke målt — det krever menneskelige review-økter som ikke har skjedd. Begge står som ikke-målt, ikke som antatt. VIDERE FUNN: subtraksjon kan etterlate en misvisende rest (§5 sier den «cannot introduce a new error» — sant om setningen, usant om leserens slutning), og kan ødelegge sann informasjon (prebuilt-check → finnes, heter prebuilt-check.us). `disposition` er `outdated` på 202/202 og bærer null informasjon, i strid med flagg-formatspesifikasjonen. `claim` matcher fillinjen ordrett i 0 av 202. Full oppskrift og åpne operatørbeslutninger: docs/r11-pilot-results.md
This commit is contained in:
parent
4fae3c46b2
commit
b0b5890703
5 changed files with 637 additions and 0 deletions
155
docs/r11-pilot-results.md
Normal file
155
docs/r11-pilot-results.md
Normal file
|
|
@ -0,0 +1,155 @@
|
||||||
|
# R11 pilot results — measured, 2026-08-03
|
||||||
|
|
||||||
|
**The §10 acceptance measurement of `docs/r11-tiered-fix-design.md`, run against
|
||||||
|
the live ledger. No KB file was edited and no ledger record was written.**
|
||||||
|
|
||||||
|
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
|
||||||
|
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
|
||||||
|
the O1 driver with writes disabled — it constructs the swap and checks the §4
|
||||||
|
invariant, so measurements 1 and 3 come out of the mechanism that would later
|
||||||
|
touch the corpus, not out of a proxy heuristic.
|
||||||
|
|
||||||
|
Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
|
||||||
|
regenerable; per-flag records so the run can be re-analysed without re-running).
|
||||||
|
|
||||||
|
> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
|
||||||
|
> claim *and* quote contain a numeric token" is a different 202, measured over
|
||||||
|
> the full 712-flag population. They are unrelated.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. The headline
|
||||||
|
|
||||||
|
**Nine provable, correct value swaps exist in the entire 776-flag `not_grounded`
|
||||||
|
population — 1.2 %.** The machine half of the R11 tiering buys nine edits. Every
|
||||||
|
other flag needs a human.
|
||||||
|
|
||||||
|
| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
|
||||||
|
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
|
||||||
|
|
||||||
|
This is the answer §10 asked for, and it is materially worse than the design
|
||||||
|
assumed: *"If the split is materially worse than assumed, that is known after one
|
||||||
|
session rather than after ten."*
|
||||||
|
|
||||||
|
## 2. §4 as written is not sufficient — measured, not argued
|
||||||
|
|
||||||
|
§4 claims its invariant is *"deliberately stronger than human review at scale."*
|
||||||
|
It is not. Run exactly as specified over the pilot, it admitted **6 swaps, of
|
||||||
|
which 4 were wrong** — precision **2/6**:
|
||||||
|
|
||||||
|
| Proposed swap | Why it is wrong |
|
||||||
|
|---|---|
|
||||||
|
| `30-dagers` → `24-dagers` | **Unit crossing.** The quote says 24 **hours**. |
|
||||||
|
| `3000 requests/sekund` → `50` | **Metric crossing.** The quote is a *query* throttle per index; the claim is an *indexing* rate per replica. |
|
||||||
|
| `Microsoft Agent 365` → `Agent 7` | **Identifier mutilated.** The `7` was harvested out of `E7`. |
|
||||||
|
| `text-embedding-ada-002` → `ada-2` | **Identifier mutilated.** The `2` came from a dimensions column. |
|
||||||
|
|
||||||
|
The defect is structural, not incidental. §4 constrains **where the new value
|
||||||
|
came from** (verbatim in the cited quote) and **what the edit looks like** (one
|
||||||
|
line, rest byte-identical). It constrains nothing about whether the two tokens
|
||||||
|
**denote the same quantity**. Same-type-and-provenance is not same-referent.
|
||||||
|
|
||||||
|
### The added condition
|
||||||
|
|
||||||
|
`contextCorresponds()` requires the token to sit under **the same label or the
|
||||||
|
same trailing unit on both sides**. It is deliberately lexical, with **no
|
||||||
|
translation table**: `dokumenter` is not taught to equal `documents`, because a
|
||||||
|
synonym table introduces a new fact source and is an operator decision, not an
|
||||||
|
engineering one. Consequence, measured: a swap is provable essentially only where
|
||||||
|
the context is language-neutral — a URL, a code sample, a parameter key.
|
||||||
|
|
||||||
|
## 3. The condition is necessary but still not sufficient
|
||||||
|
|
||||||
|
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
|
||||||
|
splits them cleanly by token type:
|
||||||
|
|
||||||
|
| Token type | Proposals | Correct | Failure mode |
|
||||||
|
|---|---|---|---|
|
||||||
|
| `iso_date` | 9 | **9** | — every one is an `api-version=` bump in a URL or code sample |
|
||||||
|
| `number` | 5 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
|
||||||
|
| `version` | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
|
||||||
|
|
||||||
|
A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
|
||||||
|
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
|
||||||
|
hand-verification**, and the report marks it as the sole recommended class
|
||||||
|
(`o1_recommended`). `number` and `version` proposals must not be applied.
|
||||||
|
|
||||||
|
## 4. The four §10 measurements
|
||||||
|
|
||||||
|
1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
|
||||||
|
class only). O2 is **undetermined** — it does not exist as a class until §5 is
|
||||||
|
ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.
|
||||||
|
2. **How much of R8 resolves as O2.** **Not answered, and not answerable by
|
||||||
|
machine.** R8 is 87/202 on the pilot (366/776 corpus-wide) and yields **zero**
|
||||||
|
O1. Of the pilot's 87, **46 are structural enumerations** — the O2 candidate
|
||||||
|
shape. Which of them subtract cleanly turns on the judge's prose `reason`, and
|
||||||
|
no regex reads prose. This needs prose classification (see §6).
|
||||||
|
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
|
||||||
|
actionable:
|
||||||
|
|
||||||
|
| Code | Pilot | Class |
|
||||||
|
|---|---|---|
|
||||||
|
| `MULTI_PART_CLAIM` | 96 (47.5 %) | intrinsic — not a value swap at all |
|
||||||
|
| `MULTI_VALUE_TOKEN` | 29 | intrinsic |
|
||||||
|
| `NO_VALUE_TOKEN` | 28 | intrinsic — the claim asserts prose |
|
||||||
|
| `STATUS_SYNONYM` | 15 | **operator question** (§6.2) |
|
||||||
|
| `NOT_VERBATIM` | 15 | intrinsic |
|
||||||
|
| `LOCATOR_AMBIGUOUS` | 7 | **fixable engineering gap** |
|
||||||
|
| `MULTI_REPLACEMENT` | 6 | intrinsic |
|
||||||
|
| `CONTEXT_MISMATCH` | 4 | intrinsic — these are the 4 wrong edits above |
|
||||||
|
|
||||||
|
**Only 7 of 200 aborts (3.5 %) are a fixable engineering gap.** More locator
|
||||||
|
engineering cannot move the O1 number materially.
|
||||||
|
4. **Review throughput per class. NOT MEASURED.** It requires human review
|
||||||
|
sessions, which have not happened. Recording it as measured would be false.
|
||||||
|
|
||||||
|
## 5. Two further findings
|
||||||
|
|
||||||
|
**F1 — subtraction can leave a misleading remainder.** §5 argues O2 *"cannot
|
||||||
|
introduce a new error, because it asserts strictly less."* True of the sentence,
|
||||||
|
false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
|
||||||
|
+ Bing) er GA (juni 2025)"* where the source says the tool is **deprecated**.
|
||||||
|
Subtracting `er GA (juni 2025)` leaves the tool standing in a list of available
|
||||||
|
tools. Strictly less asserted; still misleading. O2 therefore still requires a
|
||||||
|
human to look at the remainder — cheaper than O3 (no fact-finding) but not
|
||||||
|
mechanical.
|
||||||
|
|
||||||
|
**F2 — subtraction can destroy true information.** Real case: a list of seven
|
||||||
|
prebuilt model IDs where the judge found six correct and `prebuilt-check` wrong —
|
||||||
|
the real ID is `prebuilt-check.us`. Subtraction drops a model that **exists**; the
|
||||||
|
correct fix is a swap. Subtraction is not the safe default everywhere.
|
||||||
|
|
||||||
|
**F3 — `disposition` carries zero information.** It is `outdated` on **202 of
|
||||||
|
202** flags. `docs/r11-flag-format-2026-07.md` specifies `not_grounded →
|
||||||
|
{outdated, wrong}` with *"the human assigns which at R11"*, but the pass
|
||||||
|
hard-assigned `outdated`. Do not use it as a classifier signal. Spec/data
|
||||||
|
divergence, recorded.
|
||||||
|
|
||||||
|
**F4 — claims are not file text.** `claim` is an LLM-extracted, translated
|
||||||
|
restatement: **0 of 202** match their file line verbatim, and 188 share no 40-char
|
||||||
|
run with it. For table claims, `line` points at the **header**, not the value.
|
||||||
|
This is why the locator exists at all, and why it searches the enclosing block
|
||||||
|
rather than the line.
|
||||||
|
|
||||||
|
## 6. Open operator decisions
|
||||||
|
|
||||||
|
1. **Ratify O2 (§5)?** Until then O2 does not exist and everything is O3. F1/F2
|
||||||
|
above argue for ratifying it **with a remainder check** rather than as a blanket
|
||||||
|
rule.
|
||||||
|
2. **Amend §4 with a ratified synonym table?** 15 pilot flags (54 corpus-wide) are
|
||||||
|
`STATUS_SYNONYM`: the corpus writes `**Preview**` / `**GA**`, the source writes
|
||||||
|
*"generally available"*. Either a small ratified equivalence table admits them,
|
||||||
|
or they are permanently O3.
|
||||||
|
3. **Is O1 worth building at all?** Nine edits corpus-wide, all `api-version`
|
||||||
|
bumps. The alternative is to drop the O1 driver and treat those nine as O3.
|
||||||
|
|
||||||
|
## 7. What this does not change
|
||||||
|
|
||||||
|
The design's core reading survives: the expensive half (locating the source,
|
||||||
|
reading it, extracting the deciding passage) was already paid for by the judge
|
||||||
|
pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
|
||||||
|
falsifies is the assumption that a meaningful share of that evidence converts into
|
||||||
|
machine-provable edits. It does not. R11 is a human review programme with a
|
||||||
|
nine-item machine assist, and its leverage lies entirely in the O2 decision.
|
||||||
|
|
@ -204,6 +204,16 @@ Measure and record:
|
||||||
Scale to the remaining files only on measured numbers. If the split is materially
|
Scale to the remaining files only on measured numbers. If the split is materially
|
||||||
worse than assumed, that is known after one session rather than after ten.
|
worse than assumed, that is known after one session rather than after ten.
|
||||||
|
|
||||||
|
> **RUN 2026-08-03 — results in `docs/r11-pilot-results.md`.** The split is
|
||||||
|
> materially worse than assumed: **9 provable, correct value swaps in the whole
|
||||||
|
> 776-flag `not_grounded` population (1.2 %)**, all of them `api-version` bumps.
|
||||||
|
> The pilot also falsifies §4 as written — run exactly as specified it admitted 6
|
||||||
|
> swaps of which **4 were wrong edits** (unit crossing, metric crossing, two
|
||||||
|
> mutilated identifiers), so the invariant is *not* "stronger than human review at
|
||||||
|
> scale". A context-correspondence condition was added; read §4 together with the
|
||||||
|
> results doc, not on its own. §5 (O2) and the `STATUS_SYNONYM` class are the open
|
||||||
|
> operator decisions, and they now carry the whole programme's leverage.
|
||||||
|
|
||||||
## 11. Out of scope
|
## 11. Out of scope
|
||||||
|
|
||||||
- **Rebuild instead of repair.** Regenerating flagged files from source rather
|
- **Rebuild instead of repair.** Regenerating flagged files from source rather
|
||||||
|
|
|
||||||
175
scripts/kb-eval/classify-fix-ops.mjs
Normal file
175
scripts/kb-eval/classify-fix-ops.mjs
Normal file
|
|
@ -0,0 +1,175 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// classify-fix-ops.mjs — R11 pilot runner (docs/r11-tiered-fix-design.md §10).
|
||||||
|
//
|
||||||
|
// Runs the fix-operation classifier over the pilot population: the files
|
||||||
|
// carrying >= 7 `not_grounded` flags, the densest available sample. Produces the
|
||||||
|
// four §10 measurements — the O1/O3 split, the R8 breakdown, the typed abort
|
||||||
|
// distribution, and the per-flag record needed to re-analyse without re-running.
|
||||||
|
//
|
||||||
|
// READ-ONLY over the corpus and the ledger. It never edits a KB file and never
|
||||||
|
// touches judge-pass-manifest.json — §8's single-writer state is untouched. The
|
||||||
|
// only write is its own report, and only with --write.
|
||||||
|
//
|
||||||
|
// Usage: node scripts/kb-eval/classify-fix-ops.mjs [--write] [--threshold N] [--examples N]
|
||||||
|
|
||||||
|
import fs from 'node:fs';
|
||||||
|
import path from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
import { ABORT_CODES, classifyFlag } from './lib/fix-op.mjs';
|
||||||
|
|
||||||
|
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||||
|
const REPO = path.resolve(__dirname, '..', '..');
|
||||||
|
const DATA = path.join(__dirname, 'data');
|
||||||
|
|
||||||
|
const argv = process.argv.slice(2);
|
||||||
|
const flagArg = (name, fallback) => {
|
||||||
|
const i = argv.indexOf(name);
|
||||||
|
return i === -1 ? fallback : Number(argv[i + 1]);
|
||||||
|
};
|
||||||
|
const THRESHOLD = flagArg('--threshold', 7);
|
||||||
|
const EXAMPLES = flagArg('--examples', 3);
|
||||||
|
|
||||||
|
const ledger = JSON.parse(fs.readFileSync(path.join(DATA, 'judge-pass-manifest.json'), 'utf8'));
|
||||||
|
|
||||||
|
const ng = (rec) => (rec.flags || []).filter((f) => f.judge_verdict === 'not_grounded');
|
||||||
|
const population = ledger.files.filter((rec) => ng(rec).length >= THRESHOLD);
|
||||||
|
|
||||||
|
// Two passes over the same population. The canonical one applies the context
|
||||||
|
// condition; the §4-only pass exists purely to MEASURE what that condition
|
||||||
|
// removes — it is never a source of proposals, because four of the six swaps it
|
||||||
|
// admits on this population are wrong edits (see lib/fix-op.mjs).
|
||||||
|
const items = [];
|
||||||
|
const s4Only = [];
|
||||||
|
for (const rec of population) {
|
||||||
|
const text = fs.readFileSync(path.join(REPO, rec.file), 'utf8');
|
||||||
|
for (const flag of ng(rec)) {
|
||||||
|
const verdict = classifyFlag(flag, text);
|
||||||
|
s4Only.push(classifyFlag(flag, text, { contextCheck: false }));
|
||||||
|
items.push({
|
||||||
|
id: flag.id,
|
||||||
|
file: flag.file,
|
||||||
|
line: flag.line,
|
||||||
|
rule: flag.rule || '(none)',
|
||||||
|
claim: flag.claim,
|
||||||
|
evidence_url: flag.evidence_url,
|
||||||
|
evidence_quote: flag.evidence_quote,
|
||||||
|
reason: flag.reason,
|
||||||
|
op: verdict.op,
|
||||||
|
code: verdict.code,
|
||||||
|
detail: verdict.detail,
|
||||||
|
proposal: verdict.proposal,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// ------------------------------------------------------------------ measurements
|
||||||
|
|
||||||
|
const tally = (rows, key) =>
|
||||||
|
rows.reduce((acc, r) => {
|
||||||
|
const k = typeof key === 'function' ? key(r) : r[key];
|
||||||
|
acc[k] = (acc[k] || 0) + 1;
|
||||||
|
return acc;
|
||||||
|
}, {});
|
||||||
|
|
||||||
|
const o1 = items.filter((i) => i.op === 'O1');
|
||||||
|
const o3 = items.filter((i) => i.op === 'O3');
|
||||||
|
const byCode = tally(o3, 'code');
|
||||||
|
const byRule = tally(items, 'rule');
|
||||||
|
const r8 = items.filter((i) => i.rule === 'R8');
|
||||||
|
|
||||||
|
// LOCATOR_MISS / LOCATOR_AMBIGUOUS are a FIXABLE engineering gap (the locator did
|
||||||
|
// not find the value the claim asserts). Every other abort is intrinsic to the
|
||||||
|
// flag: no swappable value, no same-type replacement in the cited quote, or a
|
||||||
|
// claim that is not a value swap at all. The distinction is what tells the
|
||||||
|
// operator whether more engineering would move the O1 number.
|
||||||
|
const LOCATOR_CODES = new Set([ABORT_CODES.LOCATOR_MISS, ABORT_CODES.LOCATOR_AMBIGUOUS]);
|
||||||
|
const locatorAborts = o3.filter((i) => LOCATOR_CODES.has(i.code)).length;
|
||||||
|
|
||||||
|
const s4O1 = s4Only.filter((v) => v.op === 'O1').length;
|
||||||
|
|
||||||
|
// O1 precision is NOT uniform across token types, and this split is the pilot's
|
||||||
|
// operational conclusion. Hand-verified over the whole not_grounded population:
|
||||||
|
// every iso_date swap is an `api-version=` bump in a URL or code sample and all
|
||||||
|
// were correct; the number/version swaps mutilated identifiers instead
|
||||||
|
// ("AI-900" -> "AI-901", "gpt-4o" -> "gpt-5.1o" twice, a Java agent DOWNgrade),
|
||||||
|
// because a matching identifier prefix ("AI-", "gpt-") satisfies the context
|
||||||
|
// condition while the digit is part of a name rather than a quantity.
|
||||||
|
const O1_HAND_VERIFIED_TYPES = new Set(['iso_date']);
|
||||||
|
const byType = tally(o1, (i) => i.proposal.type);
|
||||||
|
const recommended = o1.filter((i) => O1_HAND_VERIFIED_TYPES.has(i.proposal.type));
|
||||||
|
const pct = (n) => `${((n / items.length) * 100).toFixed(1)} %`;
|
||||||
|
|
||||||
|
const report = {
|
||||||
|
_meta: {
|
||||||
|
purpose:
|
||||||
|
'R11 pilot measurement (§10): fix-operation classification over the densest not_grounded sample. Read-only — no KB file and no ledger record was written.',
|
||||||
|
contract: 'docs/r11-tiered-fix-design.md §3/§4/§10',
|
||||||
|
classifier: 'scripts/kb-eval/lib/fix-op.mjs (the O1 driver with writes disabled)',
|
||||||
|
ledger: 'scripts/kb-eval/data/judge-pass-manifest.json',
|
||||||
|
ledger_records: ledger.files.length,
|
||||||
|
threshold: `not_grounded >= ${THRESHOLD} (source_silent excluded, per §10)`,
|
||||||
|
generated_from: 'ledger snapshot at run time — counts are re-derived, never read from a plan',
|
||||||
|
disclaimer_two_202s:
|
||||||
|
"This population is 202 flags. §3's '202 flags whose claim and quote contain a numeric token' is a DIFFERENT 202, measured over the full 712-flag population. Do not conflate them.",
|
||||||
|
},
|
||||||
|
population: { files: population.length, flags: items.length },
|
||||||
|
s4_as_written: {
|
||||||
|
O1: s4O1,
|
||||||
|
note:
|
||||||
|
'What §4 exactly as written would admit. NOT a source of proposals: on the >=7 pilot all 6 were hand-verified and 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers) — measured precision 2/6. Runs at other thresholds carry no hand-verification.',
|
||||||
|
},
|
||||||
|
o1_by_type: byType,
|
||||||
|
o1_recommended: {
|
||||||
|
count: recommended.length,
|
||||||
|
types: [...O1_HAND_VERIFIED_TYPES],
|
||||||
|
note:
|
||||||
|
'The only O1 class that survived hand-verification: iso_date, which in this corpus is always an api-version bump inside a URL or code sample. number/version proposals are NOT safe to apply — they mutilate product, model and certification identifiers.',
|
||||||
|
},
|
||||||
|
split: { O1: o1.length, O2: 0, O3: o3.length, O2_note: 'O2 requires operator ratification (§5); until then every non-O1 item is O3 by design.' },
|
||||||
|
abort_codes: byCode,
|
||||||
|
locator_aborts: { count: locatorAborts, note: 'fixable engineering gap — every other abort is intrinsic to the flag' },
|
||||||
|
by_rule: byRule,
|
||||||
|
r8: { total: r8.length, O1: r8.filter((i) => i.op === 'O1').length, codes: tally(r8.filter((i) => i.op === 'O3'), 'code') },
|
||||||
|
items,
|
||||||
|
};
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------- output
|
||||||
|
|
||||||
|
console.log(`R11 pilot — ${population.length} files / ${items.length} not_grounded flags (threshold >= ${THRESHOLD})`);
|
||||||
|
console.log(`ledger: ${ledger.files.length} records\n`);
|
||||||
|
console.log(`O1 (provable value swap): ${o1.length} (${pct(o1.length)})`);
|
||||||
|
console.log(`O3 (human): ${o3.length} (${pct(o3.length)})`);
|
||||||
|
console.log(`O2: 0 (unratified — §5)`);
|
||||||
|
const handNote =
|
||||||
|
THRESHOLD === 7
|
||||||
|
? ' — all 6 hand-verified: 4 are wrong edits (unit crossing, metric crossing, two mutilated identifiers)'
|
||||||
|
: ' (hand-verification was done on the >=7 pilot only)';
|
||||||
|
console.log(`\n§4 as written would admit ${s4O1}${handNote}. Context condition removes ${s4O1 - o1.length}.\n`);
|
||||||
|
console.log('abort codes:');
|
||||||
|
for (const [code, n] of Object.entries(byCode).sort((a, b) => b[1] - a[1])) {
|
||||||
|
console.log(` ${code.padEnd(20)} ${String(n).padStart(4)} ${pct(n)}`);
|
||||||
|
}
|
||||||
|
console.log(`\nO1 by token type: ${JSON.stringify(byType)}`);
|
||||||
|
console.log(`O1 hand-verified-safe class (iso_date / api-version): ${recommended.length} — the rest mutilate identifiers, do NOT apply`);
|
||||||
|
console.log(`\nlocator aborts (fixable): ${locatorAborts} intrinsic aborts: ${o3.length - locatorAborts}`);
|
||||||
|
console.log(`\nrule distribution: ${JSON.stringify(byRule)}`);
|
||||||
|
console.log(`R8: ${r8.length} flags — O1 ${report.r8.O1}, aborts ${JSON.stringify(report.r8.codes)}`);
|
||||||
|
|
||||||
|
if (EXAMPLES > 0 && o1.length > 0) {
|
||||||
|
console.log(`\n--- ${Math.min(EXAMPLES, o1.length)} proven O1 proposals ---`);
|
||||||
|
for (const i of o1.slice(0, EXAMPLES)) {
|
||||||
|
console.log(`\n${i.file}:${i.proposal.line} [${i.rule}] ${i.token || i.proposal.token} -> ${i.proposal.replacement}`);
|
||||||
|
console.log(` - ${i.proposal.before}`);
|
||||||
|
console.log(` + ${i.proposal.after}`);
|
||||||
|
console.log(` quote: ${i.proposal.evidence_quote.slice(0, 160)}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (argv.includes('--write')) {
|
||||||
|
const out = path.join(DATA, 'r11-pilot-classification.json');
|
||||||
|
fs.writeFileSync(out, JSON.stringify(report, null, 2) + '\n');
|
||||||
|
console.log(`\nwrote ${out}`);
|
||||||
|
} else {
|
||||||
|
console.log('\n(dry run — pass --write to persist r11-pilot-classification.json)');
|
||||||
|
}
|
||||||
297
scripts/kb-eval/lib/fix-op.mjs
Normal file
297
scripts/kb-eval/lib/fix-op.mjs
Normal file
|
|
@ -0,0 +1,297 @@
|
||||||
|
// fix-op.mjs — R11 fix-operation classifier over judge-pass flags.
|
||||||
|
//
|
||||||
|
// Contract: docs/r11-tiered-fix-design.md §3 (the O1/O2/O3 partition is by
|
||||||
|
// OPERATION, not by rule code) and §4 (the O1 invariant).
|
||||||
|
//
|
||||||
|
// This module IS the O1 driver with writes disabled. It attempts the value swap
|
||||||
|
// and checks §4's three conditions; an item it cannot prove is O3 with a typed
|
||||||
|
// abort code. That is deliberate: a proxy heuristic would have measured
|
||||||
|
// something other than the mechanism that will later touch a public corpus.
|
||||||
|
//
|
||||||
|
// Two properties the callers depend on:
|
||||||
|
// - PURE. No fs, no network, no mutation of the input flag. The caller reads
|
||||||
|
// the file and passes its text.
|
||||||
|
// - FAILS CLOSED. Every path returns O1-with-proof or O3-with-a-known-code.
|
||||||
|
// A misrouted O3 costs one human review; a misrouted O1 ships a wrong edit
|
||||||
|
// to a publicly distributed file.
|
||||||
|
//
|
||||||
|
// What this module deliberately does NOT do: decide O2. Subtraction candidacy
|
||||||
|
// turns on which sub-assertion the judge's prose `reason` names as failing, and
|
||||||
|
// no regex reads prose. O2 requires operator ratification (§5) before it exists
|
||||||
|
// as a class at all; until then every non-O1 item is O3 by design.
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Abort codes. The taxonomy is part of the contract, not diagnostics: the pilot's
|
||||||
|
* measurement #3 (§10) is the DISTRIBUTION of these, because "abort rate 85 %"
|
||||||
|
* is not actionable while "60 % LOCATOR_MISS" is an engineering gap and "60 %
|
||||||
|
* NOT_VERBATIM" is intrinsic to the corpus.
|
||||||
|
*/
|
||||||
|
export const ABORT_CODES = {
|
||||||
|
MULTI_PART_CLAIM: 'MULTI_PART_CLAIM', // enumeration / several assertions in one claim (§3, the R8 class)
|
||||||
|
NO_VALUE_TOKEN: 'NO_VALUE_TOKEN', // nothing swappable — the claim asserts prose
|
||||||
|
STATUS_SYNONYM: 'STATUS_SYNONYM', // GA/Preview class: file vocabulary != source vocabulary (operator question)
|
||||||
|
MULTI_VALUE_TOKEN: 'MULTI_VALUE_TOKEN', // several distinct values — which one is wrong is a judgement
|
||||||
|
LOCATOR_MISS: 'LOCATOR_MISS', // value not found in the block the flag points at
|
||||||
|
LOCATOR_AMBIGUOUS: 'LOCATOR_AMBIGUOUS', // value occurs more than once in that block
|
||||||
|
NOT_VERBATIM: 'NOT_VERBATIM', // no same-type replacement occurs verbatim in evidence_quote (§4.1)
|
||||||
|
MULTI_REPLACEMENT: 'MULTI_REPLACEMENT', // quote offers several candidate values
|
||||||
|
CONTEXT_MISMATCH: 'CONTEXT_MISMATCH', // §4 held but the tokens do not denote the same quantity (see below)
|
||||||
|
INVARIANT_FAIL: 'INVARIANT_FAIL', // swap constructed but §4 did not hold — must never happen silently
|
||||||
|
};
|
||||||
|
|
||||||
|
/** Verdict code for a proven swap. Kept out of ABORT_CODES so `op === 'O1' <=> code === 'PROVEN'`. */
|
||||||
|
export const PROVEN = 'PROVEN';
|
||||||
|
|
||||||
|
// Value types, most specific first. Matching is non-overlapping and priority
|
||||||
|
// ordered, so `2.3.0` is one version rather than two numbers, and `20 %` is a
|
||||||
|
// percent rather than the number 20. Types never cross in a swap: a percent may
|
||||||
|
// only be replaced by a percent.
|
||||||
|
const TOKEN_PATTERNS = [
|
||||||
|
['iso_date', /\d{4}-\d{2}-\d{2}/g],
|
||||||
|
['percent', /\d+(?:[.,]\d+)?\s?%/g],
|
||||||
|
['version', /v?\d+\.\d+\.\d+/g],
|
||||||
|
['number', /\d+(?:[.,]\d+)?/g],
|
||||||
|
];
|
||||||
|
|
||||||
|
// Lifecycle vocabulary. Present in a claim without any numeric token, this is the
|
||||||
|
// GA/Preview class: the corpus writes `**Preview**` / `**GA**` while the cited
|
||||||
|
// source writes "generally available". A swap would satisfy §4 literally while
|
||||||
|
// pasting English prose into a Norwegian table, so the class aborts and is put to
|
||||||
|
// the operator as a design question (a ratified synonym table, or permanent O3).
|
||||||
|
const STATUS_RE =
|
||||||
|
/\b(?:GA|generally available|allment tilgjengelig|public preview|private preview|preview|deprecated|utfaset|retired|avviklet)\b/i;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract swappable value tokens, non-overlapping and priority ordered.
|
||||||
|
* Status words are NOT value tokens — see STATUS_RE.
|
||||||
|
* @returns {Array<{type: string, value: string, index: number}>} in order of appearance
|
||||||
|
*/
|
||||||
|
export function extractValueTokens(text) {
|
||||||
|
if (!text) return [];
|
||||||
|
const taken = []; // [start, end) ranges already consumed by a higher-priority type
|
||||||
|
const out = [];
|
||||||
|
for (const [type, re] of TOKEN_PATTERNS) {
|
||||||
|
re.lastIndex = 0;
|
||||||
|
let m;
|
||||||
|
while ((m = re.exec(text)) !== null) {
|
||||||
|
const start = m.index;
|
||||||
|
const end = start + m[0].length;
|
||||||
|
if (taken.some(([s, e]) => start < e && end > s)) continue;
|
||||||
|
taken.push([start, end]);
|
||||||
|
out.push({ type, value: m[0], index: start });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return out.sort((a, b) => a.index - b.index);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** True if the text carries lifecycle-status vocabulary. */
|
||||||
|
export function hasStatusWord(text) {
|
||||||
|
return STATUS_RE.test(text || '');
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The contiguous non-blank block containing `line` (1-indexed).
|
||||||
|
*
|
||||||
|
* This is the search window, and it is structural rather than a magic ±N: claims
|
||||||
|
* are LLM-extracted restatements whose `line` often points at a table HEADER
|
||||||
|
* while the asserted value sits in a row below. A block is exactly that table,
|
||||||
|
* list, or paragraph. A blank line degenerates to itself.
|
||||||
|
*/
|
||||||
|
export function blockWindow(lines, line) {
|
||||||
|
if (line < 1 || line > lines.length) return { start: line, end: line };
|
||||||
|
if (lines[line - 1].trim() === '') return { start: line, end: line };
|
||||||
|
let start = line;
|
||||||
|
let end = line;
|
||||||
|
while (start > 1 && lines[start - 2].trim() !== '') start -= 1;
|
||||||
|
while (end < lines.length && lines[end].trim() !== '') end += 1;
|
||||||
|
return { start, end };
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------- context condition
|
||||||
|
//
|
||||||
|
// MEASURED, NOT ASSUMED: §4 alone admits wrong edits. On the pilot it proved six
|
||||||
|
// swaps of which four were false — "30-dagers" -> "24" from a quote saying 24
|
||||||
|
// HOURS (unit crossing), an indexing rate replaced by a query throttle (metric
|
||||||
|
// crossing), and two identifiers mutilated by digits harvested out of "E7" and a
|
||||||
|
// table cell ("Microsoft Agent 365" -> "Agent 7", "text-embedding-ada-002" ->
|
||||||
|
// "ada-2"). §4 constrains where the new value CAME FROM and what the edit LOOKS
|
||||||
|
// LIKE; it constrains nothing about whether the two tokens denote the same
|
||||||
|
// quantity.
|
||||||
|
//
|
||||||
|
// The condition below adds that, and it is deliberately lexical rather than
|
||||||
|
// semantic: the token must sit under the same label, or the same trailing unit,
|
||||||
|
// on both sides. No translation table — "dokumenter" is not taught to equal
|
||||||
|
// "documents", because a synonym/translation table introduces a new fact source
|
||||||
|
// and is an operator decision (§5-class), not an engineering one. The consequence
|
||||||
|
// is measured and reported: a swap is provable essentially only where the context
|
||||||
|
// is language-neutral (a URL, a code sample, a parameter key).
|
||||||
|
|
||||||
|
const WORD = /[A-Za-z0-9_.\-æøåÆØÅ]/;
|
||||||
|
|
||||||
|
/** Normalise a context run for comparison: lowercase, punctuation stripped. */
|
||||||
|
const normContext = (s) => s.toLowerCase().replace(/[^a-z0-9æøå]/g, '');
|
||||||
|
|
||||||
|
/** The word run immediately left of [index], skipping any separator run first. */
|
||||||
|
function leftContext(text, index) {
|
||||||
|
let i = index - 1;
|
||||||
|
// A separator run may be skipped; a word character adjacent to the token may
|
||||||
|
// NOT be — that adjacency is what makes "7" part of the identifier "E7".
|
||||||
|
if (i >= 0 && !WORD.test(text[i])) {
|
||||||
|
while (i >= 0 && !WORD.test(text[i])) i -= 1;
|
||||||
|
}
|
||||||
|
let end = i + 1;
|
||||||
|
while (i >= 0 && WORD.test(text[i])) i -= 1;
|
||||||
|
return normContext(text.slice(i + 1, end));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The word run immediately right of [index], skipping any separator run first. */
|
||||||
|
function rightContext(text, index) {
|
||||||
|
let i = index;
|
||||||
|
if (i < text.length && !WORD.test(text[i])) {
|
||||||
|
while (i < text.length && !WORD.test(text[i])) i += 1;
|
||||||
|
}
|
||||||
|
const start = i;
|
||||||
|
while (i < text.length && WORD.test(text[i])) i += 1;
|
||||||
|
return normContext(text.slice(start, i));
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Do the two occurrences sit in corresponding context? True when a non-empty
|
||||||
|
* label matches on the left, or a non-empty unit matches on the right.
|
||||||
|
*/
|
||||||
|
export function contextCorresponds(fileLine, fileIndex, fileLen, quote, quoteIndex, quoteLen) {
|
||||||
|
const lf = leftContext(fileLine, fileIndex);
|
||||||
|
const lq = leftContext(quote, quoteIndex);
|
||||||
|
if (lf && lf === lq) return true;
|
||||||
|
const rf = rightContext(fileLine, fileIndex + fileLen);
|
||||||
|
const rq = rightContext(quote, quoteIndex + quoteLen);
|
||||||
|
return Boolean(rf) && rf === rq;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Distinct by type+value, preserving order. */
|
||||||
|
function distinct(tokens) {
|
||||||
|
const seen = new Set();
|
||||||
|
return tokens.filter((t) => {
|
||||||
|
const k = `${t.type} ${t.value}`;
|
||||||
|
if (seen.has(k)) return false;
|
||||||
|
seen.add(k);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function abort(code, detail = {}) {
|
||||||
|
return { op: 'O3', code, ...detail };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Classify one flag record into a fix operation.
|
||||||
|
*
|
||||||
|
* @param {object} flag flag record per docs/r11-flag-format-2026-07.md
|
||||||
|
* @param {string} fileText current content of flag.file
|
||||||
|
* @param {{contextCheck?: boolean}} [opts] contextCheck:false reproduces §4 exactly as
|
||||||
|
* written — used to MEASURE what the context condition adds, never to ship edits.
|
||||||
|
* @returns {{op: 'O1'|'O3', code: string, proposal?: object, detail?: object}}
|
||||||
|
*/
|
||||||
|
export function classifyFlag(flag, fileText, opts = {}) {
|
||||||
|
const contextCheck = opts.contextCheck !== false;
|
||||||
|
const claim = flag.claim || '';
|
||||||
|
const lines = (fileText || '').split('\n');
|
||||||
|
|
||||||
|
// §3: an enumeration is not a value swap even when it contains a number. The
|
||||||
|
// structure decides, not the rule code — R8 is a signal, and the run records
|
||||||
|
// it, but it is not the partition.
|
||||||
|
if (claim.includes(' | ') || (claim.match(/,/g) || []).length >= 3) {
|
||||||
|
return abort(ABORT_CODES.MULTI_PART_CLAIM);
|
||||||
|
}
|
||||||
|
|
||||||
|
const tokens = distinct(extractValueTokens(claim));
|
||||||
|
if (tokens.length === 0) {
|
||||||
|
return abort(hasStatusWord(claim) ? ABORT_CODES.STATUS_SYNONYM : ABORT_CODES.NO_VALUE_TOKEN);
|
||||||
|
}
|
||||||
|
if (tokens.length > 1) {
|
||||||
|
return abort(ABORT_CODES.MULTI_VALUE_TOKEN, { detail: { candidates: tokens.map((t) => t.value) } });
|
||||||
|
}
|
||||||
|
|
||||||
|
const wrong = tokens[0];
|
||||||
|
|
||||||
|
// Locate the value in the block the flag points at. Unique hit or nothing:
|
||||||
|
// a locator that guesses is how a wrong edit reaches a public file.
|
||||||
|
if (flag.line < 1 || flag.line > lines.length) {
|
||||||
|
return abort(ABORT_CODES.LOCATOR_MISS, { detail: { reason: 'line out of range', line: flag.line } });
|
||||||
|
}
|
||||||
|
const win = blockWindow(lines, flag.line);
|
||||||
|
const hits = [];
|
||||||
|
for (let n = win.start; n <= win.end; n += 1) {
|
||||||
|
const text = lines[n - 1];
|
||||||
|
let from = 0;
|
||||||
|
for (;;) {
|
||||||
|
const at = text.indexOf(wrong.value, from);
|
||||||
|
if (at === -1) break;
|
||||||
|
hits.push({ line: n, index: at });
|
||||||
|
from = at + wrong.value.length;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (hits.length === 0) return abort(ABORT_CODES.LOCATOR_MISS, { detail: { token: wrong.value, window: win } });
|
||||||
|
if (hits.length > 1) {
|
||||||
|
return abort(ABORT_CODES.LOCATOR_AMBIGUOUS, { detail: { token: wrong.value, hits: hits.length, window: win } });
|
||||||
|
}
|
||||||
|
|
||||||
|
// §4.1: the replacement must occur verbatim in the quote the judge actually
|
||||||
|
// cited. Same type only, and a quote that merely restates the claim's own value
|
||||||
|
// offers no replacement at all.
|
||||||
|
const replacements = distinct(extractValueTokens(flag.evidence_quote || '')).filter(
|
||||||
|
(t) => t.type === wrong.type && t.value !== wrong.value,
|
||||||
|
);
|
||||||
|
if (replacements.length === 0) return abort(ABORT_CODES.NOT_VERBATIM, { detail: { token: wrong.value, type: wrong.type } });
|
||||||
|
if (replacements.length > 1) {
|
||||||
|
return abort(ABORT_CODES.MULTI_REPLACEMENT, { detail: { candidates: replacements.map((t) => t.value) } });
|
||||||
|
}
|
||||||
|
const right = replacements[0];
|
||||||
|
|
||||||
|
const hit = hits[0];
|
||||||
|
const before = lines[hit.line - 1];
|
||||||
|
|
||||||
|
// The condition §4 is missing: same label or same unit on both sides.
|
||||||
|
if (
|
||||||
|
contextCheck &&
|
||||||
|
!contextCorresponds(before, hit.index, wrong.value.length, flag.evidence_quote || '', right.index, right.value.length)
|
||||||
|
) {
|
||||||
|
return abort(ABORT_CODES.CONTEXT_MISMATCH, {
|
||||||
|
detail: {
|
||||||
|
token: wrong.value,
|
||||||
|
replacement: right.value,
|
||||||
|
would_have_been: before.slice(0, hit.index) + right.value + before.slice(hit.index + wrong.value.length),
|
||||||
|
},
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
const after = before.slice(0, hit.index) + right.value + before.slice(hit.index + wrong.value.length);
|
||||||
|
|
||||||
|
// Re-check §4 against the constructed edit rather than trusting construction.
|
||||||
|
const quoteHasValue = (flag.evidence_quote || '').includes(right.value);
|
||||||
|
const restIdentical =
|
||||||
|
before.slice(0, hit.index) === after.slice(0, hit.index) &&
|
||||||
|
before.slice(hit.index + wrong.value.length) === after.slice(hit.index + right.value.length);
|
||||||
|
const rebuilt = lines.slice();
|
||||||
|
rebuilt[hit.line - 1] = after;
|
||||||
|
const changedLines = rebuilt.reduce((n, l, i) => n + (l === lines[i] ? 0 : 1), 0);
|
||||||
|
if (!quoteHasValue || !restIdentical || changedLines !== 1) {
|
||||||
|
return abort(ABORT_CODES.INVARIANT_FAIL, { detail: { quoteHasValue, restIdentical, changedLines } });
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
op: 'O1',
|
||||||
|
code: PROVEN,
|
||||||
|
proposal: {
|
||||||
|
file: flag.file,
|
||||||
|
line: hit.line,
|
||||||
|
token: wrong.value,
|
||||||
|
replacement: right.value,
|
||||||
|
type: wrong.type,
|
||||||
|
before,
|
||||||
|
after,
|
||||||
|
evidence_url: flag.evidence_url,
|
||||||
|
evidence_quote: flag.evidence_quote,
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
BIN
tests/kb-eval/test-fix-op-classify.test.mjs
Normal file
BIN
tests/kb-eval/test-fix-op-classify.test.mjs
Normal file
Binary file not shown.
Loading…
Add table
Add a link
Reference in a new issue