feat(ms-ai-architect): R11 pilot kjørt — §4-invarianten falsifisert, 9 provbare swaps i hele korpuset [skip-docs]
§10-målingen er gjennomført mot live ledger (243 records). Ingen KB-fil er redigert og ingen ledger-record er skrevet — §8s single-writer-state er urørt. Instrumentet ER O1-driveren med writes av (scripts/kb-eval/lib/fix-op.mjs, 30 tester). Måling #1 og #3 kommer dermed ut av mekanismen som senere skal ta på korpuset, ikke ut av en proxy-heuristikk. HOVEDFUNN — §4 som skrevet er utilstrekkelig, målt: Kjørt eksakt som spesifisert slapp den gjennom 6 swaps på piloten, hvorav 4 er GALE editer (presisjon 2/6): - 30-dagers → 24-dagers (enhets-kryssing: kilden sier 24 HOURS) - 3000 req/sek → 50 (metrikk-kryssing: query-throttle vs indexing-rate) - Microsoft Agent 365 → 7 (identifikator lemlestet, «7» høstet fra «E7») - text-embedding-ada-002 → ada-2 (identifikator lemlestet) §4 binder proveniensen til verdien og formen på editen — ingenting om at de to tokenene betegner SAMME STØRRELSE. Påstanden om at invarianten er «deliberately stronger than human review at scale» holder ikke. TILLEGG: contextCorresponds() krever samme label eller samme enhet på begge sider. Bevisst leksikalsk, UTEN oversettelsestabell — «dokumenter» læres ikke å være «documents», fordi en synonymtabell innfører en ny faktakilde og er en operatørbeslutning. Konsekvensen er målt: swap er provbar praktisk talt bare der konteksten er språknøytral (URL, kodeeksempel, parameternøkkel). TALLENE: - Pilot (≥7): 24 filer / 202 flagg → O1 = 2 (1,0 %), O3 = 200 (99,0 %) - Hele korpuset: 218 filer / 776 flagg → 15 sluppet gjennom, 9 korrekte - Kun iso_date (api-version-bump) overlever hånd-verifisering: 9/9. number/version lemlester identifikatorer (AI-900 → AI-901, gpt-4o → gpt-5.1o ×2, Java-agent 3.7.5 → 3.4.0 = nedgradering) og skal IKKE påføres. - Kun 7 av 200 aborter (3,5 %) er en fiksbar engineering-gap. Mer locator- arbeid kan ikke flytte O1-tallet vesentlig. Måling #2 (R8 → O2) er IKKE besvart og kan ikke besvares maskinelt: R8 gir null O1, og hvilke av de 46 enumerasjonene som subtraherer rent avhenger av dommerens PROSA-reason. Måling #4 (review-throughput) er ikke målt — det krever menneskelige review-økter som ikke har skjedd. Begge står som ikke-målt, ikke som antatt. VIDERE FUNN: subtraksjon kan etterlate en misvisende rest (§5 sier den «cannot introduce a new error» — sant om setningen, usant om leserens slutning), og kan ødelegge sann informasjon (prebuilt-check → finnes, heter prebuilt-check.us). `disposition` er `outdated` på 202/202 og bærer null informasjon, i strid med flagg-formatspesifikasjonen. `claim` matcher fillinjen ordrett i 0 av 202. Full oppskrift og åpne operatørbeslutninger: docs/r11-pilot-results.md
This commit is contained in:
parent
4fae3c46b2
commit
b0b5890703
5 changed files with 637 additions and 0 deletions
155
docs/r11-pilot-results.md
Normal file
155
docs/r11-pilot-results.md
Normal file
|
|
@ -0,0 +1,155 @@
|
|||
# R11 pilot results — measured, 2026-08-03
|
||||
|
||||
**The §10 acceptance measurement of `docs/r11-tiered-fix-design.md`, run against
|
||||
the live ledger. No KB file was edited and no ledger record was written.**
|
||||
|
||||
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
|
||||
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
|
||||
the O1 driver with writes disabled — it constructs the swap and checks the §4
|
||||
invariant, so measurements 1 and 3 come out of the mechanism that would later
|
||||
touch the corpus, not out of a proxy heuristic.
|
||||
|
||||
Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
|
||||
regenerable; per-flag records so the run can be re-analysed without re-running).
|
||||
|
||||
> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
|
||||
> claim *and* quote contain a numeric token" is a different 202, measured over
|
||||
> the full 712-flag population. They are unrelated.
|
||||
|
||||
---
|
||||
|
||||
## 1. The headline
|
||||
|
||||
**Nine provable, correct value swaps exist in the entire 776-flag `not_grounded`
|
||||
population — 1.2 %.** The machine half of the R11 tiering buys nine edits. Every
|
||||
other flag needs a human.
|
||||
|
||||
| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|
||||
|---|---|---|---|---|
|
||||
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
|
||||
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
|
||||
|
||||
This is the answer §10 asked for, and it is materially worse than the design
|
||||
assumed: *"If the split is materially worse than assumed, that is known after one
|
||||
session rather than after ten."*
|
||||
|
||||
## 2. §4 as written is not sufficient — measured, not argued
|
||||
|
||||
§4 claims its invariant is *"deliberately stronger than human review at scale."*
|
||||
It is not. Run exactly as specified over the pilot, it admitted **6 swaps, of
|
||||
which 4 were wrong** — precision **2/6**:
|
||||
|
||||
| Proposed swap | Why it is wrong |
|
||||
|---|---|
|
||||
| `30-dagers` → `24-dagers` | **Unit crossing.** The quote says 24 **hours**. |
|
||||
| `3000 requests/sekund` → `50` | **Metric crossing.** The quote is a *query* throttle per index; the claim is an *indexing* rate per replica. |
|
||||
| `Microsoft Agent 365` → `Agent 7` | **Identifier mutilated.** The `7` was harvested out of `E7`. |
|
||||
| `text-embedding-ada-002` → `ada-2` | **Identifier mutilated.** The `2` came from a dimensions column. |
|
||||
|
||||
The defect is structural, not incidental. §4 constrains **where the new value
|
||||
came from** (verbatim in the cited quote) and **what the edit looks like** (one
|
||||
line, rest byte-identical). It constrains nothing about whether the two tokens
|
||||
**denote the same quantity**. Same-type-and-provenance is not same-referent.
|
||||
|
||||
### The added condition
|
||||
|
||||
`contextCorresponds()` requires the token to sit under **the same label or the
|
||||
same trailing unit on both sides**. It is deliberately lexical, with **no
|
||||
translation table**: `dokumenter` is not taught to equal `documents`, because a
|
||||
synonym table introduces a new fact source and is an operator decision, not an
|
||||
engineering one. Consequence, measured: a swap is provable essentially only where
|
||||
the context is language-neutral — a URL, a code sample, a parameter key.
|
||||
|
||||
## 3. The condition is necessary but still not sufficient
|
||||
|
||||
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
|
||||
splits them cleanly by token type:
|
||||
|
||||
| Token type | Proposals | Correct | Failure mode |
|
||||
|---|---|---|---|
|
||||
| `iso_date` | 9 | **9** | — every one is an `api-version=` bump in a URL or code sample |
|
||||
| `number` | 5 | 2 | `AI-900` → `AI-901`, `gpt-4o` → `gpt-5.1o` (×2) |
|
||||
| `version` | 1 | 0 | Java agent `3.7.5` → `3.4.0` — a downgrade |
|
||||
|
||||
A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
|
||||
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
|
||||
hand-verification**, and the report marks it as the sole recommended class
|
||||
(`o1_recommended`). `number` and `version` proposals must not be applied.
|
||||
|
||||
## 4. The four §10 measurements
|
||||
|
||||
1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
|
||||
class only). O2 is **undetermined** — it does not exist as a class until §5 is
|
||||
ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.
|
||||
2. **How much of R8 resolves as O2.** **Not answered, and not answerable by
|
||||
machine.** R8 is 87/202 on the pilot (366/776 corpus-wide) and yields **zero**
|
||||
O1. Of the pilot's 87, **46 are structural enumerations** — the O2 candidate
|
||||
shape. Which of them subtract cleanly turns on the judge's prose `reason`, and
|
||||
no regex reads prose. This needs prose classification (see §6).
|
||||
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
|
||||
actionable:
|
||||
|
||||
| Code | Pilot | Class |
|
||||
|---|---|---|
|
||||
| `MULTI_PART_CLAIM` | 96 (47.5 %) | intrinsic — not a value swap at all |
|
||||
| `MULTI_VALUE_TOKEN` | 29 | intrinsic |
|
||||
| `NO_VALUE_TOKEN` | 28 | intrinsic — the claim asserts prose |
|
||||
| `STATUS_SYNONYM` | 15 | **operator question** (§6.2) |
|
||||
| `NOT_VERBATIM` | 15 | intrinsic |
|
||||
| `LOCATOR_AMBIGUOUS` | 7 | **fixable engineering gap** |
|
||||
| `MULTI_REPLACEMENT` | 6 | intrinsic |
|
||||
| `CONTEXT_MISMATCH` | 4 | intrinsic — these are the 4 wrong edits above |
|
||||
|
||||
**Only 7 of 200 aborts (3.5 %) are a fixable engineering gap.** More locator
|
||||
engineering cannot move the O1 number materially.
|
||||
4. **Review throughput per class. NOT MEASURED.** It requires human review
|
||||
sessions, which have not happened. Recording it as measured would be false.
|
||||
|
||||
## 5. Two further findings
|
||||
|
||||
**F1 — subtraction can leave a misleading remainder.** §5 argues O2 *"cannot
|
||||
introduce a new error, because it asserts strictly less."* True of the sentence,
|
||||
false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
|
||||
+ Bing) er GA (juni 2025)"* where the source says the tool is **deprecated**.
|
||||
Subtracting `er GA (juni 2025)` leaves the tool standing in a list of available
|
||||
tools. Strictly less asserted; still misleading. O2 therefore still requires a
|
||||
human to look at the remainder — cheaper than O3 (no fact-finding) but not
|
||||
mechanical.
|
||||
|
||||
**F2 — subtraction can destroy true information.** Real case: a list of seven
|
||||
prebuilt model IDs where the judge found six correct and `prebuilt-check` wrong —
|
||||
the real ID is `prebuilt-check.us`. Subtraction drops a model that **exists**; the
|
||||
correct fix is a swap. Subtraction is not the safe default everywhere.
|
||||
|
||||
**F3 — `disposition` carries zero information.** It is `outdated` on **202 of
|
||||
202** flags. `docs/r11-flag-format-2026-07.md` specifies `not_grounded →
|
||||
{outdated, wrong}` with *"the human assigns which at R11"*, but the pass
|
||||
hard-assigned `outdated`. Do not use it as a classifier signal. Spec/data
|
||||
divergence, recorded.
|
||||
|
||||
**F4 — claims are not file text.** `claim` is an LLM-extracted, translated
|
||||
restatement: **0 of 202** match their file line verbatim, and 188 share no 40-char
|
||||
run with it. For table claims, `line` points at the **header**, not the value.
|
||||
This is why the locator exists at all, and why it searches the enclosing block
|
||||
rather than the line.
|
||||
|
||||
## 6. Open operator decisions
|
||||
|
||||
1. **Ratify O2 (§5)?** Until then O2 does not exist and everything is O3. F1/F2
|
||||
above argue for ratifying it **with a remainder check** rather than as a blanket
|
||||
rule.
|
||||
2. **Amend §4 with a ratified synonym table?** 15 pilot flags (54 corpus-wide) are
|
||||
`STATUS_SYNONYM`: the corpus writes `**Preview**` / `**GA**`, the source writes
|
||||
*"generally available"*. Either a small ratified equivalence table admits them,
|
||||
or they are permanently O3.
|
||||
3. **Is O1 worth building at all?** Nine edits corpus-wide, all `api-version`
|
||||
bumps. The alternative is to drop the O1 driver and treat those nine as O3.
|
||||
|
||||
## 7. What this does not change
|
||||
|
||||
The design's core reading survives: the expensive half (locating the source,
|
||||
reading it, extracting the deciding passage) was already paid for by the judge
|
||||
pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
|
||||
falsifies is the assumption that a meaningful share of that evidence converts into
|
||||
machine-provable edits. It does not. R11 is a human review programme with a
|
||||
nine-item machine assist, and its leverage lies entirely in the O2 decision.
|
||||
|
|
@ -204,6 +204,16 @@ Measure and record:
|
|||
Scale to the remaining files only on measured numbers. If the split is materially
|
||||
worse than assumed, that is known after one session rather than after ten.
|
||||
|
||||
> **RUN 2026-08-03 — results in `docs/r11-pilot-results.md`.** The split is
|
||||
> materially worse than assumed: **9 provable, correct value swaps in the whole
|
||||
> 776-flag `not_grounded` population (1.2 %)**, all of them `api-version` bumps.
|
||||
> The pilot also falsifies §4 as written — run exactly as specified it admitted 6
|
||||
> swaps of which **4 were wrong edits** (unit crossing, metric crossing, two
|
||||
> mutilated identifiers), so the invariant is *not* "stronger than human review at
|
||||
> scale". A context-correspondence condition was added; read §4 together with the
|
||||
> results doc, not on its own. §5 (O2) and the `STATUS_SYNONYM` class are the open
|
||||
> operator decisions, and they now carry the whole programme's leverage.
|
||||
|
||||
## 11. Out of scope
|
||||
|
||||
- **Rebuild instead of repair.** Regenerating flagged files from source rather
|
||||
|
|
|
|||
175
scripts/kb-eval/classify-fix-ops.mjs
Normal file
175
scripts/kb-eval/classify-fix-ops.mjs
Normal file
|
|
@ -0,0 +1,175 @@
|
|||
#!/usr/bin/env node
|
||||
// classify-fix-ops.mjs — R11 pilot runner (docs/r11-tiered-fix-design.md §10).
|
||||
//
|
||||
// Runs the fix-operation classifier over the pilot population: the files
|
||||
// carrying >= 7 `not_grounded` flags, the densest available sample. Produces the
|
||||
// four §10 measurements — the O1/O3 split, the R8 breakdown, the typed abort
|
||||
// distribution, and the per-flag record needed to re-analyse without re-running.
|
||||
//
|
||||
// READ-ONLY over the corpus and the ledger. It never edits a KB file and never
|
||||
// touches judge-pass-manifest.json — §8's single-writer state is untouched. The
|
||||
// only write is its own report, and only with --write.
|
||||
//
|
||||
// Usage: node scripts/kb-eval/classify-fix-ops.mjs [--write] [--threshold N] [--examples N]
|
||||
|
||||
import fs from 'node:fs';
|
||||
import path from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
import { ABORT_CODES, classifyFlag } from './lib/fix-op.mjs';
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const REPO = path.resolve(__dirname, '..', '..');
|
||||
const DATA = path.join(__dirname, 'data');
|
||||
|
||||
const argv = process.argv.slice(2);
|
||||
const flagArg = (name, fallback) => {
|
||||
const i = argv.indexOf(name);
|
||||
return i === -1 ? fallback : Number(argv[i + 1]);
|
||||
};
|
||||
const THRESHOLD = flagArg('--threshold', 7);
|
||||
const EXAMPLES = flagArg('--examples', 3);
|
||||
|
||||
const ledger = JSON.parse(fs.readFileSync(path.join(DATA, 'judge-pass-manifest.json'), 'utf8'));
|
||||
|
||||
const ng = (rec) => (rec.flags || []).filter((f) => f.judge_verdict === 'not_grounded');
|
||||
const population = ledger.files.filter((rec) => ng(rec).length >= THRESHOLD);
|
||||
|
||||
// Two passes over the same population. The canonical one applies the context
|
||||
// condition; the §4-only pass exists purely to MEASURE what that condition
|
||||
// removes — it is never a source of proposals, because four of the six swaps it
|
||||
// admits on this population are wrong edits (see lib/fix-op.mjs).
|
||||
const items = [];
|
||||
const s4Only = [];
|
||||
for (const rec of population) {
|
||||
const text = fs.readFileSync(path.join(REPO, rec.file), 'utf8');
|
||||
for (const flag of ng(rec)) {
|
||||
const verdict = classifyFlag(flag, text);
|
||||
s4Only.push(classifyFlag(flag, text, { contextCheck: false }));
|
||||
items.push({
|
||||
id: flag.id,
|
||||
file: flag.file,
|
||||
line: flag.line,
|
||||
rule: flag.rule || '(none)',
|
||||
claim: flag.claim,
|
||||
evidence_url: flag.evidence_url,
|
||||
evidence_quote: flag.evidence_quote,
|
||||
reason: flag.reason,
|
||||
op: verdict.op,
|
||||
code: verdict.code,
|
||||
detail: verdict.detail,
|
||||
proposal: verdict.proposal,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------------ measurements
|
||||
|
||||
const tally = (rows, key) =>
|
||||
rows.reduce((acc, r) => {
|
||||
const k = typeof key === 'function' ? key(r) : r[key];
|
||||
acc[k] = (acc[k] || 0) + 1;
|
||||
return acc;
|
||||
}, {});
|
||||
|
||||
const o1 = items.filter((i) => i.op === 'O1');
|
||||
const o3 = items.filter((i) => i.op === 'O3');
|
||||
const byCode = tally(o3, 'code');
|
||||
const byRule = tally(items, 'rule');
|
||||
const r8 = items.filter((i) => i.rule === 'R8');
|
||||
|
||||
// LOCATOR_MISS / LOCATOR_AMBIGUOUS are a FIXABLE engineering gap (the locator did
|
||||
// not find the value the claim asserts). Every other abort is intrinsic to the
|
||||
// flag: no swappable value, no same-type replacement in the cited quote, or a
|
||||
// claim that is not a value swap at all. The distinction is what tells the
|
||||
// operator whether more engineering would move the O1 number.
|
||||
const LOCATOR_CODES = new Set([ABORT_CODES.LOCATOR_MISS, ABORT_CODES.LOCATOR_AMBIGUOUS]);
|
||||
const locatorAborts = o3.filter((i) => LOCATOR_CODES.has(i.code)).length;
|
||||
|
||||
const s4O1 = s4Only.filter((v) => v.op === 'O1').length;
|
||||
|
||||
// O1 precision is NOT uniform across token types, and this split is the pilot's
|
||||
// operational conclusion. Hand-verified over the whole not_grounded population:
|
||||
// every iso_date swap is an `api-version=` bump in a URL or code sample and all
|
||||
// were correct; the number/version swaps mutilated identifiers instead
|
||||
// ("AI-900" -> "AI-901", "gpt-4o" -> "gpt-5.1o" twice, a Java agent DOWNgrade),
|
||||
// because a matching identifier prefix ("AI-", "gpt-") satisfies the context
|
||||
// condition while the digit is part of a name rather than a quantity.
|
||||
const O1_HAND_VERIFIED_TYPES = new Set(['iso_date']);
|
||||
const byType = tally(o1, (i) => i.proposal.type);
|
||||
const recommended = o1.filter((i) => O1_HAND_VERIFIED_TYPES.has(i.proposal.type));
|
||||
const pct = (n) => `${((n / items.length) * 100).toFixed(1)} %`;
|
||||
|
||||
const report = {
|
||||
_meta: {
|
||||
purpose:
|
||||
'R11 pilot measurement (§10): fix-operation classification over the densest not_grounded sample. Read-only — no KB file and no ledger record was written.',
|
||||
contract: 'docs/r11-tiered-fix-design.md §3/§4/§10',
|
||||
classifier: 'scripts/kb-eval/lib/fix-op.mjs (the O1 driver with writes disabled)',
|
||||
ledger: 'scripts/kb-eval/data/judge-pass-manifest.json',
|
||||
ledger_records: ledger.files.length,
|
||||
threshold: `not_grounded >= ${THRESHOLD} (source_silent excluded, per §10)`,
|
||||
generated_from: 'ledger snapshot at run time — counts are re-derived, never read from a plan',
|
||||
disclaimer_two_202s:
|
||||
"This population is 202 flags. §3's '202 flags whose claim and quote contain a numeric token' is a DIFFERENT 202, measured over the full 712-flag population. Do not conflate them.",
|
||||
},
|
||||
population: { files: population.length, flags: items.length },
|
||||
s4_as_written: {
|
||||
O1: s4O1,
|
||||
note:
|
||||
'What §4 exactly as written would admit. NOT a source of proposals: on the >=7 pilot all 6 were hand-verified and 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers) — measured precision 2/6. Runs at other thresholds carry no hand-verification.',
|
||||
},
|
||||
o1_by_type: byType,
|
||||
o1_recommended: {
|
||||
count: recommended.length,
|
||||
types: [...O1_HAND_VERIFIED_TYPES],
|
||||
note:
|
||||
'The only O1 class that survived hand-verification: iso_date, which in this corpus is always an api-version bump inside a URL or code sample. number/version proposals are NOT safe to apply — they mutilate product, model and certification identifiers.',
|
||||
},
|
||||
split: { O1: o1.length, O2: 0, O3: o3.length, O2_note: 'O2 requires operator ratification (§5); until then every non-O1 item is O3 by design.' },
|
||||
abort_codes: byCode,
|
||||
locator_aborts: { count: locatorAborts, note: 'fixable engineering gap — every other abort is intrinsic to the flag' },
|
||||
by_rule: byRule,
|
||||
r8: { total: r8.length, O1: r8.filter((i) => i.op === 'O1').length, codes: tally(r8.filter((i) => i.op === 'O3'), 'code') },
|
||||
items,
|
||||
};
|
||||
|
||||
// ---------------------------------------------------------------------- output
|
||||
|
||||
console.log(`R11 pilot — ${population.length} files / ${items.length} not_grounded flags (threshold >= ${THRESHOLD})`);
|
||||
console.log(`ledger: ${ledger.files.length} records\n`);
|
||||
console.log(`O1 (provable value swap): ${o1.length} (${pct(o1.length)})`);
|
||||
console.log(`O3 (human): ${o3.length} (${pct(o3.length)})`);
|
||||
console.log(`O2: 0 (unratified — §5)`);
|
||||
const handNote =
|
||||
THRESHOLD === 7
|
||||
? ' — all 6 hand-verified: 4 are wrong edits (unit crossing, metric crossing, two mutilated identifiers)'
|
||||
: ' (hand-verification was done on the >=7 pilot only)';
|
||||
console.log(`\n§4 as written would admit ${s4O1}${handNote}. Context condition removes ${s4O1 - o1.length}.\n`);
|
||||
console.log('abort codes:');
|
||||
for (const [code, n] of Object.entries(byCode).sort((a, b) => b[1] - a[1])) {
|
||||
console.log(` ${code.padEnd(20)} ${String(n).padStart(4)} ${pct(n)}`);
|
||||
}
|
||||
console.log(`\nO1 by token type: ${JSON.stringify(byType)}`);
|
||||
console.log(`O1 hand-verified-safe class (iso_date / api-version): ${recommended.length} — the rest mutilate identifiers, do NOT apply`);
|
||||
console.log(`\nlocator aborts (fixable): ${locatorAborts} intrinsic aborts: ${o3.length - locatorAborts}`);
|
||||
console.log(`\nrule distribution: ${JSON.stringify(byRule)}`);
|
||||
console.log(`R8: ${r8.length} flags — O1 ${report.r8.O1}, aborts ${JSON.stringify(report.r8.codes)}`);
|
||||
|
||||
if (EXAMPLES > 0 && o1.length > 0) {
|
||||
console.log(`\n--- ${Math.min(EXAMPLES, o1.length)} proven O1 proposals ---`);
|
||||
for (const i of o1.slice(0, EXAMPLES)) {
|
||||
console.log(`\n${i.file}:${i.proposal.line} [${i.rule}] ${i.token || i.proposal.token} -> ${i.proposal.replacement}`);
|
||||
console.log(` - ${i.proposal.before}`);
|
||||
console.log(` + ${i.proposal.after}`);
|
||||
console.log(` quote: ${i.proposal.evidence_quote.slice(0, 160)}`);
|
||||
}
|
||||
}
|
||||
|
||||
if (argv.includes('--write')) {
|
||||
const out = path.join(DATA, 'r11-pilot-classification.json');
|
||||
fs.writeFileSync(out, JSON.stringify(report, null, 2) + '\n');
|
||||
console.log(`\nwrote ${out}`);
|
||||
} else {
|
||||
console.log('\n(dry run — pass --write to persist r11-pilot-classification.json)');
|
||||
}
|
||||
297
scripts/kb-eval/lib/fix-op.mjs
Normal file
297
scripts/kb-eval/lib/fix-op.mjs
Normal file
|
|
@ -0,0 +1,297 @@
|
|||
// fix-op.mjs — R11 fix-operation classifier over judge-pass flags.
|
||||
//
|
||||
// Contract: docs/r11-tiered-fix-design.md §3 (the O1/O2/O3 partition is by
|
||||
// OPERATION, not by rule code) and §4 (the O1 invariant).
|
||||
//
|
||||
// This module IS the O1 driver with writes disabled. It attempts the value swap
|
||||
// and checks §4's three conditions; an item it cannot prove is O3 with a typed
|
||||
// abort code. That is deliberate: a proxy heuristic would have measured
|
||||
// something other than the mechanism that will later touch a public corpus.
|
||||
//
|
||||
// Two properties the callers depend on:
|
||||
// - PURE. No fs, no network, no mutation of the input flag. The caller reads
|
||||
// the file and passes its text.
|
||||
// - FAILS CLOSED. Every path returns O1-with-proof or O3-with-a-known-code.
|
||||
// A misrouted O3 costs one human review; a misrouted O1 ships a wrong edit
|
||||
// to a publicly distributed file.
|
||||
//
|
||||
// What this module deliberately does NOT do: decide O2. Subtraction candidacy
|
||||
// turns on which sub-assertion the judge's prose `reason` names as failing, and
|
||||
// no regex reads prose. O2 requires operator ratification (§5) before it exists
|
||||
// as a class at all; until then every non-O1 item is O3 by design.
|
||||
|
||||
/**
|
||||
* Abort codes. The taxonomy is part of the contract, not diagnostics: the pilot's
|
||||
* measurement #3 (§10) is the DISTRIBUTION of these, because "abort rate 85 %"
|
||||
* is not actionable while "60 % LOCATOR_MISS" is an engineering gap and "60 %
|
||||
* NOT_VERBATIM" is intrinsic to the corpus.
|
||||
*/
|
||||
export const ABORT_CODES = {
|
||||
MULTI_PART_CLAIM: 'MULTI_PART_CLAIM', // enumeration / several assertions in one claim (§3, the R8 class)
|
||||
NO_VALUE_TOKEN: 'NO_VALUE_TOKEN', // nothing swappable — the claim asserts prose
|
||||
STATUS_SYNONYM: 'STATUS_SYNONYM', // GA/Preview class: file vocabulary != source vocabulary (operator question)
|
||||
MULTI_VALUE_TOKEN: 'MULTI_VALUE_TOKEN', // several distinct values — which one is wrong is a judgement
|
||||
LOCATOR_MISS: 'LOCATOR_MISS', // value not found in the block the flag points at
|
||||
LOCATOR_AMBIGUOUS: 'LOCATOR_AMBIGUOUS', // value occurs more than once in that block
|
||||
NOT_VERBATIM: 'NOT_VERBATIM', // no same-type replacement occurs verbatim in evidence_quote (§4.1)
|
||||
MULTI_REPLACEMENT: 'MULTI_REPLACEMENT', // quote offers several candidate values
|
||||
CONTEXT_MISMATCH: 'CONTEXT_MISMATCH', // §4 held but the tokens do not denote the same quantity (see below)
|
||||
INVARIANT_FAIL: 'INVARIANT_FAIL', // swap constructed but §4 did not hold — must never happen silently
|
||||
};
|
||||
|
||||
/** Verdict code for a proven swap. Kept out of ABORT_CODES so `op === 'O1' <=> code === 'PROVEN'`. */
|
||||
export const PROVEN = 'PROVEN';
|
||||
|
||||
// Value types, most specific first. Matching is non-overlapping and priority
|
||||
// ordered, so `2.3.0` is one version rather than two numbers, and `20 %` is a
|
||||
// percent rather than the number 20. Types never cross in a swap: a percent may
|
||||
// only be replaced by a percent.
|
||||
const TOKEN_PATTERNS = [
|
||||
['iso_date', /\d{4}-\d{2}-\d{2}/g],
|
||||
['percent', /\d+(?:[.,]\d+)?\s?%/g],
|
||||
['version', /v?\d+\.\d+\.\d+/g],
|
||||
['number', /\d+(?:[.,]\d+)?/g],
|
||||
];
|
||||
|
||||
// Lifecycle vocabulary. Present in a claim without any numeric token, this is the
|
||||
// GA/Preview class: the corpus writes `**Preview**` / `**GA**` while the cited
|
||||
// source writes "generally available". A swap would satisfy §4 literally while
|
||||
// pasting English prose into a Norwegian table, so the class aborts and is put to
|
||||
// the operator as a design question (a ratified synonym table, or permanent O3).
|
||||
const STATUS_RE =
|
||||
/\b(?:GA|generally available|allment tilgjengelig|public preview|private preview|preview|deprecated|utfaset|retired|avviklet)\b/i;
|
||||
|
||||
/**
|
||||
* Extract swappable value tokens, non-overlapping and priority ordered.
|
||||
* Status words are NOT value tokens — see STATUS_RE.
|
||||
* @returns {Array<{type: string, value: string, index: number}>} in order of appearance
|
||||
*/
|
||||
export function extractValueTokens(text) {
|
||||
if (!text) return [];
|
||||
const taken = []; // [start, end) ranges already consumed by a higher-priority type
|
||||
const out = [];
|
||||
for (const [type, re] of TOKEN_PATTERNS) {
|
||||
re.lastIndex = 0;
|
||||
let m;
|
||||
while ((m = re.exec(text)) !== null) {
|
||||
const start = m.index;
|
||||
const end = start + m[0].length;
|
||||
if (taken.some(([s, e]) => start < e && end > s)) continue;
|
||||
taken.push([start, end]);
|
||||
out.push({ type, value: m[0], index: start });
|
||||
}
|
||||
}
|
||||
return out.sort((a, b) => a.index - b.index);
|
||||
}
|
||||
|
||||
/** True if the text carries lifecycle-status vocabulary. */
|
||||
export function hasStatusWord(text) {
|
||||
return STATUS_RE.test(text || '');
|
||||
}
|
||||
|
||||
/**
|
||||
* The contiguous non-blank block containing `line` (1-indexed).
|
||||
*
|
||||
* This is the search window, and it is structural rather than a magic ±N: claims
|
||||
* are LLM-extracted restatements whose `line` often points at a table HEADER
|
||||
* while the asserted value sits in a row below. A block is exactly that table,
|
||||
* list, or paragraph. A blank line degenerates to itself.
|
||||
*/
|
||||
export function blockWindow(lines, line) {
|
||||
if (line < 1 || line > lines.length) return { start: line, end: line };
|
||||
if (lines[line - 1].trim() === '') return { start: line, end: line };
|
||||
let start = line;
|
||||
let end = line;
|
||||
while (start > 1 && lines[start - 2].trim() !== '') start -= 1;
|
||||
while (end < lines.length && lines[end].trim() !== '') end += 1;
|
||||
return { start, end };
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------- context condition
|
||||
//
|
||||
// MEASURED, NOT ASSUMED: §4 alone admits wrong edits. On the pilot it proved six
|
||||
// swaps of which four were false — "30-dagers" -> "24" from a quote saying 24
|
||||
// HOURS (unit crossing), an indexing rate replaced by a query throttle (metric
|
||||
// crossing), and two identifiers mutilated by digits harvested out of "E7" and a
|
||||
// table cell ("Microsoft Agent 365" -> "Agent 7", "text-embedding-ada-002" ->
|
||||
// "ada-2"). §4 constrains where the new value CAME FROM and what the edit LOOKS
|
||||
// LIKE; it constrains nothing about whether the two tokens denote the same
|
||||
// quantity.
|
||||
//
|
||||
// The condition below adds that, and it is deliberately lexical rather than
|
||||
// semantic: the token must sit under the same label, or the same trailing unit,
|
||||
// on both sides. No translation table — "dokumenter" is not taught to equal
|
||||
// "documents", because a synonym/translation table introduces a new fact source
|
||||
// and is an operator decision (§5-class), not an engineering one. The consequence
|
||||
// is measured and reported: a swap is provable essentially only where the context
|
||||
// is language-neutral (a URL, a code sample, a parameter key).
|
||||
|
||||
const WORD = /[A-Za-z0-9_.\-æøåÆØÅ]/;
|
||||
|
||||
/** Normalise a context run for comparison: lowercase, punctuation stripped. */
|
||||
const normContext = (s) => s.toLowerCase().replace(/[^a-z0-9æøå]/g, '');
|
||||
|
||||
/** The word run immediately left of [index], skipping any separator run first. */
|
||||
function leftContext(text, index) {
|
||||
let i = index - 1;
|
||||
// A separator run may be skipped; a word character adjacent to the token may
|
||||
// NOT be — that adjacency is what makes "7" part of the identifier "E7".
|
||||
if (i >= 0 && !WORD.test(text[i])) {
|
||||
while (i >= 0 && !WORD.test(text[i])) i -= 1;
|
||||
}
|
||||
let end = i + 1;
|
||||
while (i >= 0 && WORD.test(text[i])) i -= 1;
|
||||
return normContext(text.slice(i + 1, end));
|
||||
}
|
||||
|
||||
/** The word run immediately right of [index], skipping any separator run first. */
|
||||
function rightContext(text, index) {
|
||||
let i = index;
|
||||
if (i < text.length && !WORD.test(text[i])) {
|
||||
while (i < text.length && !WORD.test(text[i])) i += 1;
|
||||
}
|
||||
const start = i;
|
||||
while (i < text.length && WORD.test(text[i])) i += 1;
|
||||
return normContext(text.slice(start, i));
|
||||
}
|
||||
|
||||
/**
|
||||
* Do the two occurrences sit in corresponding context? True when a non-empty
|
||||
* label matches on the left, or a non-empty unit matches on the right.
|
||||
*/
|
||||
export function contextCorresponds(fileLine, fileIndex, fileLen, quote, quoteIndex, quoteLen) {
|
||||
const lf = leftContext(fileLine, fileIndex);
|
||||
const lq = leftContext(quote, quoteIndex);
|
||||
if (lf && lf === lq) return true;
|
||||
const rf = rightContext(fileLine, fileIndex + fileLen);
|
||||
const rq = rightContext(quote, quoteIndex + quoteLen);
|
||||
return Boolean(rf) && rf === rq;
|
||||
}
|
||||
|
||||
/** Distinct by type+value, preserving order. */
|
||||
function distinct(tokens) {
|
||||
const seen = new Set();
|
||||
return tokens.filter((t) => {
|
||||
const k = `${t.type} | ||||