feat(ms-ai-architect): R11 pilot kjørt — §4-invarianten falsifisert, 9 provbare swaps i hele korpuset [skip-docs]

§10-målingen er gjennomført mot live ledger (243 records). Ingen KB-fil er
redigert og ingen ledger-record er skrevet — §8s single-writer-state er urørt.

Instrumentet ER O1-driveren med writes av (scripts/kb-eval/lib/fix-op.mjs, 30
tester). Måling #1 og #3 kommer dermed ut av mekanismen som senere skal ta på
korpuset, ikke ut av en proxy-heuristikk.

HOVEDFUNN — §4 som skrevet er utilstrekkelig, målt:
Kjørt eksakt som spesifisert slapp den gjennom 6 swaps på piloten, hvorav 4 er
GALE editer (presisjon 2/6):
- 30-dagers → 24-dagers   (enhets-kryssing: kilden sier 24 HOURS)
- 3000 req/sek → 50       (metrikk-kryssing: query-throttle vs indexing-rate)
- Microsoft Agent 365 → 7 (identifikator lemlestet, «7» høstet fra «E7»)
- text-embedding-ada-002 → ada-2 (identifikator lemlestet)
§4 binder proveniensen til verdien og formen på editen — ingenting om at de to
tokenene betegner SAMME STØRRELSE. Påstanden om at invarianten er «deliberately
stronger than human review at scale» holder ikke.

TILLEGG: contextCorresponds() krever samme label eller samme enhet på begge
sider. Bevisst leksikalsk, UTEN oversettelsestabell — «dokumenter» læres ikke å
være «documents», fordi en synonymtabell innfører en ny faktakilde og er en
operatørbeslutning. Konsekvensen er målt: swap er provbar praktisk talt bare der
konteksten er språknøytral (URL, kodeeksempel, parameternøkkel).

TALLENE:
- Pilot (≥7): 24 filer / 202 flagg → O1 = 2 (1,0 %), O3 = 200 (99,0 %)
- Hele korpuset: 218 filer / 776 flagg → 15 sluppet gjennom, 9 korrekte
- Kun iso_date (api-version-bump) overlever hånd-verifisering: 9/9.
  number/version lemlester identifikatorer (AI-900 → AI-901, gpt-4o → gpt-5.1o
  ×2, Java-agent 3.7.5 → 3.4.0 = nedgradering) og skal IKKE påføres.
- Kun 7 av 200 aborter (3,5 %) er en fiksbar engineering-gap. Mer locator-
  arbeid kan ikke flytte O1-tallet vesentlig.

Måling #2 (R8 → O2) er IKKE besvart og kan ikke besvares maskinelt: R8 gir null
O1, og hvilke av de 46 enumerasjonene som subtraherer rent avhenger av dommerens
PROSA-reason. Måling #4 (review-throughput) er ikke målt — det krever
menneskelige review-økter som ikke har skjedd. Begge står som ikke-målt, ikke
som antatt.

VIDERE FUNN: subtraksjon kan etterlate en misvisende rest (§5 sier den «cannot
introduce a new error» — sant om setningen, usant om leserens slutning), og kan
ødelegge sann informasjon (prebuilt-check → finnes, heter prebuilt-check.us).
`disposition` er `outdated` på 202/202 og bærer null informasjon, i strid med
flagg-formatspesifikasjonen. `claim` matcher fillinjen ordrett i 0 av 202.

Full oppskrift og åpne operatørbeslutninger: docs/r11-pilot-results.md
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 16:30:31 +02:00
commit b0b5890703
5 changed files with 637 additions and 0 deletions

155
docs/r11-pilot-results.md Normal file
View file

@ -0,0 +1,155 @@
# R11 pilot results — measured, 2026-08-03
**The §10 acceptance measurement of `docs/r11-tiered-fix-design.md`, run against
the live ledger. No KB file was edited and no ledger record was written.**
Instrument: `scripts/kb-eval/lib/fix-op.mjs` (+ `tests/kb-eval/test-fix-op-classify.test.mjs`,
30 tests) driven by `scripts/kb-eval/classify-fix-ops.mjs`. The classifier **is**
the O1 driver with writes disabled — it constructs the swap and checks the §4
invariant, so measurements 1 and 3 come out of the mechanism that would later
touch the corpus, not out of a proxy heuristic.
Artefact: `scripts/kb-eval/data/r11-pilot-classification.json` (untracked,
regenerable; per-flag records so the run can be re-analysed without re-running).
> **Two different 202s.** This population is 202 flags. §3's "202 flags whose
> claim *and* quote contain a numeric token" is a different 202, measured over
> the full 712-flag population. They are unrelated.
---
## 1. The headline
**Nine provable, correct value swaps exist in the entire 776-flag `not_grounded`
population — 1.2 %.** The machine half of the R11 tiering buys nine edits. Every
other flag needs a human.
| Population | Files | Flags | O1 admitted | O1 hand-verified correct |
|---|---|---|---|---|
| Pilot (`not_grounded` ≥ 7) | 24 | 202 | 2 | 2 |
| Whole `not_grounded` corpus | 218 | 776 | 15 | **9** |
This is the answer §10 asked for, and it is materially worse than the design
assumed: *"If the split is materially worse than assumed, that is known after one
session rather than after ten."*
## 2. §4 as written is not sufficient — measured, not argued
§4 claims its invariant is *"deliberately stronger than human review at scale."*
It is not. Run exactly as specified over the pilot, it admitted **6 swaps, of
which 4 were wrong** — precision **2/6**:
| Proposed swap | Why it is wrong |
|---|---|
| `30-dagers``24-dagers` | **Unit crossing.** The quote says 24 **hours**. |
| `3000 requests/sekund``50` | **Metric crossing.** The quote is a *query* throttle per index; the claim is an *indexing* rate per replica. |
| `Microsoft Agent 365``Agent 7` | **Identifier mutilated.** The `7` was harvested out of `E7`. |
| `text-embedding-ada-002``ada-2` | **Identifier mutilated.** The `2` came from a dimensions column. |
The defect is structural, not incidental. §4 constrains **where the new value
came from** (verbatim in the cited quote) and **what the edit looks like** (one
line, rest byte-identical). It constrains nothing about whether the two tokens
**denote the same quantity**. Same-type-and-provenance is not same-referent.
### The added condition
`contextCorresponds()` requires the token to sit under **the same label or the
same trailing unit on both sides**. It is deliberately lexical, with **no
translation table**: `dokumenter` is not taught to equal `documents`, because a
synonym table introduces a new fact source and is an operator decision, not an
engineering one. Consequence, measured: a swap is provable essentially only where
the context is language-neutral — a URL, a code sample, a parameter key.
## 3. The condition is necessary but still not sufficient
Corpus-wide the context condition cut 38 admissions to 15. Hand-verifying all 15
splits them cleanly by token type:
| Token type | Proposals | Correct | Failure mode |
|---|---|---|---|
| `iso_date` | 9 | **9** | — every one is an `api-version=` bump in a URL or code sample |
| `number` | 5 | 2 | `AI-900``AI-901`, `gpt-4o``gpt-5.1o` (×2) |
| `version` | 1 | 0 | Java agent `3.7.5``3.4.0` — a downgrade |
A matching identifier *prefix* (`AI-`, `gpt-`) satisfies the context condition
while the digit is part of a **name**, not a quantity. **Only `iso_date` survives
hand-verification**, and the report marks it as the sole recommended class
(`o1_recommended`). `number` and `version` proposals must not be applied.
## 4. The four §10 measurements
1. **O1 / O2 / O3 split.** O1 = 2/202 on the pilot (9/776 corpus-wide, safe
class only). O2 is **undetermined** — it does not exist as a class until §5 is
ratified, so every non-O1 item is O3 by design. O3 ≥ 200/202.
2. **How much of R8 resolves as O2.** **Not answered, and not answerable by
machine.** R8 is 87/202 on the pilot (366/776 corpus-wide) and yields **zero**
O1. Of the pilot's 87, **46 are structural enumerations** — the O2 candidate
shape. Which of them subtract cleanly turns on the judge's prose `reason`, and
no regex reads prose. This needs prose classification (see §6).
3. **O1 abort rate: 99 % (200/202).** Typed, because "99 %" alone is not
actionable:
| Code | Pilot | Class |
|---|---|---|
| `MULTI_PART_CLAIM` | 96 (47.5 %) | intrinsic — not a value swap at all |
| `MULTI_VALUE_TOKEN` | 29 | intrinsic |
| `NO_VALUE_TOKEN` | 28 | intrinsic — the claim asserts prose |
| `STATUS_SYNONYM` | 15 | **operator question** (§6.2) |
| `NOT_VERBATIM` | 15 | intrinsic |
| `LOCATOR_AMBIGUOUS` | 7 | **fixable engineering gap** |
| `MULTI_REPLACEMENT` | 6 | intrinsic |
| `CONTEXT_MISMATCH` | 4 | intrinsic — these are the 4 wrong edits above |
**Only 7 of 200 aborts (3.5 %) are a fixable engineering gap.** More locator
engineering cannot move the O1 number materially.
4. **Review throughput per class. NOT MEASURED.** It requires human review
sessions, which have not happened. Recording it as measured would be false.
## 5. Two further findings
**F1 — subtraction can leave a misleading remainder.** §5 argues O2 *"cannot
introduce a new error, because it asserts strictly less."* True of the sentence,
false of the reader's inference. Real case: *"Deep Research tool (o3-deep-research
+ Bing) er GA (juni 2025)"* where the source says the tool is **deprecated**.
Subtracting `er GA (juni 2025)` leaves the tool standing in a list of available
tools. Strictly less asserted; still misleading. O2 therefore still requires a
human to look at the remainder — cheaper than O3 (no fact-finding) but not
mechanical.
**F2 — subtraction can destroy true information.** Real case: a list of seven
prebuilt model IDs where the judge found six correct and `prebuilt-check` wrong —
the real ID is `prebuilt-check.us`. Subtraction drops a model that **exists**; the
correct fix is a swap. Subtraction is not the safe default everywhere.
**F3 — `disposition` carries zero information.** It is `outdated` on **202 of
202** flags. `docs/r11-flag-format-2026-07.md` specifies `not_grounded →
{outdated, wrong}` with *"the human assigns which at R11"*, but the pass
hard-assigned `outdated`. Do not use it as a classifier signal. Spec/data
divergence, recorded.
**F4 — claims are not file text.** `claim` is an LLM-extracted, translated
restatement: **0 of 202** match their file line verbatim, and 188 share no 40-char
run with it. For table claims, `line` points at the **header**, not the value.
This is why the locator exists at all, and why it searches the enclosing block
rather than the line.
## 6. Open operator decisions
1. **Ratify O2 (§5)?** Until then O2 does not exist and everything is O3. F1/F2
above argue for ratifying it **with a remainder check** rather than as a blanket
rule.
2. **Amend §4 with a ratified synonym table?** 15 pilot flags (54 corpus-wide) are
`STATUS_SYNONYM`: the corpus writes `**Preview**` / `**GA**`, the source writes
*"generally available"*. Either a small ratified equivalence table admits them,
or they are permanently O3.
3. **Is O1 worth building at all?** Nine edits corpus-wide, all `api-version`
bumps. The alternative is to drop the O1 driver and treat those nine as O3.
## 7. What this does not change
The design's core reading survives: the expensive half (locating the source,
reading it, extracting the deciding passage) was already paid for by the judge
pass, and §7's freshness re-fetch per distinct URL is untouched. What the pilot
falsifies is the assumption that a meaningful share of that evidence converts into
machine-provable edits. It does not. R11 is a human review programme with a
nine-item machine assist, and its leverage lies entirely in the O2 decision.

View file

@ -204,6 +204,16 @@ Measure and record:
Scale to the remaining files only on measured numbers. If the split is materially
worse than assumed, that is known after one session rather than after ten.
> **RUN 2026-08-03 — results in `docs/r11-pilot-results.md`.** The split is
> materially worse than assumed: **9 provable, correct value swaps in the whole
> 776-flag `not_grounded` population (1.2 %)**, all of them `api-version` bumps.
> The pilot also falsifies §4 as written — run exactly as specified it admitted 6
> swaps of which **4 were wrong edits** (unit crossing, metric crossing, two
> mutilated identifiers), so the invariant is *not* "stronger than human review at
> scale". A context-correspondence condition was added; read §4 together with the
> results doc, not on its own. §5 (O2) and the `STATUS_SYNONYM` class are the open
> operator decisions, and they now carry the whole programme's leverage.
## 11. Out of scope
- **Rebuild instead of repair.** Regenerating flagged files from source rather

View file

@ -0,0 +1,175 @@
#!/usr/bin/env node
// classify-fix-ops.mjs — R11 pilot runner (docs/r11-tiered-fix-design.md §10).
//
// Runs the fix-operation classifier over the pilot population: the files
// carrying >= 7 `not_grounded` flags, the densest available sample. Produces the
// four §10 measurements — the O1/O3 split, the R8 breakdown, the typed abort
// distribution, and the per-flag record needed to re-analyse without re-running.
//
// READ-ONLY over the corpus and the ledger. It never edits a KB file and never
// touches judge-pass-manifest.json — §8's single-writer state is untouched. The
// only write is its own report, and only with --write.
//
// Usage: node scripts/kb-eval/classify-fix-ops.mjs [--write] [--threshold N] [--examples N]
import fs from 'node:fs';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
import { ABORT_CODES, classifyFlag } from './lib/fix-op.mjs';
const __dirname = path.dirname(fileURLToPath(import.meta.url));
const REPO = path.resolve(__dirname, '..', '..');
const DATA = path.join(__dirname, 'data');
const argv = process.argv.slice(2);
const flagArg = (name, fallback) => {
const i = argv.indexOf(name);
return i === -1 ? fallback : Number(argv[i + 1]);
};
const THRESHOLD = flagArg('--threshold', 7);
const EXAMPLES = flagArg('--examples', 3);
const ledger = JSON.parse(fs.readFileSync(path.join(DATA, 'judge-pass-manifest.json'), 'utf8'));
const ng = (rec) => (rec.flags || []).filter((f) => f.judge_verdict === 'not_grounded');
const population = ledger.files.filter((rec) => ng(rec).length >= THRESHOLD);
// Two passes over the same population. The canonical one applies the context
// condition; the §4-only pass exists purely to MEASURE what that condition
// removes — it is never a source of proposals, because four of the six swaps it
// admits on this population are wrong edits (see lib/fix-op.mjs).
const items = [];
const s4Only = [];
for (const rec of population) {
const text = fs.readFileSync(path.join(REPO, rec.file), 'utf8');
for (const flag of ng(rec)) {
const verdict = classifyFlag(flag, text);
s4Only.push(classifyFlag(flag, text, { contextCheck: false }));
items.push({
id: flag.id,
file: flag.file,
line: flag.line,
rule: flag.rule || '(none)',
claim: flag.claim,
evidence_url: flag.evidence_url,
evidence_quote: flag.evidence_quote,
reason: flag.reason,
op: verdict.op,
code: verdict.code,
detail: verdict.detail,
proposal: verdict.proposal,
});
}
}
// ------------------------------------------------------------------ measurements
const tally = (rows, key) =>
rows.reduce((acc, r) => {
const k = typeof key === 'function' ? key(r) : r[key];
acc[k] = (acc[k] || 0) + 1;
return acc;
}, {});
const o1 = items.filter((i) => i.op === 'O1');
const o3 = items.filter((i) => i.op === 'O3');
const byCode = tally(o3, 'code');
const byRule = tally(items, 'rule');
const r8 = items.filter((i) => i.rule === 'R8');
// LOCATOR_MISS / LOCATOR_AMBIGUOUS are a FIXABLE engineering gap (the locator did
// not find the value the claim asserts). Every other abort is intrinsic to the
// flag: no swappable value, no same-type replacement in the cited quote, or a
// claim that is not a value swap at all. The distinction is what tells the
// operator whether more engineering would move the O1 number.
const LOCATOR_CODES = new Set([ABORT_CODES.LOCATOR_MISS, ABORT_CODES.LOCATOR_AMBIGUOUS]);
const locatorAborts = o3.filter((i) => LOCATOR_CODES.has(i.code)).length;
const s4O1 = s4Only.filter((v) => v.op === 'O1').length;
// O1 precision is NOT uniform across token types, and this split is the pilot's
// operational conclusion. Hand-verified over the whole not_grounded population:
// every iso_date swap is an `api-version=` bump in a URL or code sample and all
// were correct; the number/version swaps mutilated identifiers instead
// ("AI-900" -> "AI-901", "gpt-4o" -> "gpt-5.1o" twice, a Java agent DOWNgrade),
// because a matching identifier prefix ("AI-", "gpt-") satisfies the context
// condition while the digit is part of a name rather than a quantity.
const O1_HAND_VERIFIED_TYPES = new Set(['iso_date']);
const byType = tally(o1, (i) => i.proposal.type);
const recommended = o1.filter((i) => O1_HAND_VERIFIED_TYPES.has(i.proposal.type));
const pct = (n) => `${((n / items.length) * 100).toFixed(1)} %`;
const report = {
_meta: {
purpose:
'R11 pilot measurement (§10): fix-operation classification over the densest not_grounded sample. Read-only — no KB file and no ledger record was written.',
contract: 'docs/r11-tiered-fix-design.md §3/§4/§10',
classifier: 'scripts/kb-eval/lib/fix-op.mjs (the O1 driver with writes disabled)',
ledger: 'scripts/kb-eval/data/judge-pass-manifest.json',
ledger_records: ledger.files.length,
threshold: `not_grounded >= ${THRESHOLD} (source_silent excluded, per §10)`,
generated_from: 'ledger snapshot at run time — counts are re-derived, never read from a plan',
disclaimer_two_202s:
"This population is 202 flags. §3's '202 flags whose claim and quote contain a numeric token' is a DIFFERENT 202, measured over the full 712-flag population. Do not conflate them.",
},
population: { files: population.length, flags: items.length },
s4_as_written: {
O1: s4O1,
note:
'What §4 exactly as written would admit. NOT a source of proposals: on the >=7 pilot all 6 were hand-verified and 4 were wrong edits (unit crossing, metric crossing, two mutilated identifiers) — measured precision 2/6. Runs at other thresholds carry no hand-verification.',
},
o1_by_type: byType,
o1_recommended: {
count: recommended.length,
types: [...O1_HAND_VERIFIED_TYPES],
note:
'The only O1 class that survived hand-verification: iso_date, which in this corpus is always an api-version bump inside a URL or code sample. number/version proposals are NOT safe to apply — they mutilate product, model and certification identifiers.',
},
split: { O1: o1.length, O2: 0, O3: o3.length, O2_note: 'O2 requires operator ratification (§5); until then every non-O1 item is O3 by design.' },
abort_codes: byCode,
locator_aborts: { count: locatorAborts, note: 'fixable engineering gap — every other abort is intrinsic to the flag' },
by_rule: byRule,
r8: { total: r8.length, O1: r8.filter((i) => i.op === 'O1').length, codes: tally(r8.filter((i) => i.op === 'O3'), 'code') },
items,
};
// ---------------------------------------------------------------------- output
console.log(`R11 pilot — ${population.length} files / ${items.length} not_grounded flags (threshold >= ${THRESHOLD})`);
console.log(`ledger: ${ledger.files.length} records\n`);
console.log(`O1 (provable value swap): ${o1.length} (${pct(o1.length)})`);
console.log(`O3 (human): ${o3.length} (${pct(o3.length)})`);
console.log(`O2: 0 (unratified — §5)`);
const handNote =
THRESHOLD === 7
? ' — all 6 hand-verified: 4 are wrong edits (unit crossing, metric crossing, two mutilated identifiers)'
: ' (hand-verification was done on the >=7 pilot only)';
console.log(`\n§4 as written would admit ${s4O1}${handNote}. Context condition removes ${s4O1 - o1.length}.\n`);
console.log('abort codes:');
for (const [code, n] of Object.entries(byCode).sort((a, b) => b[1] - a[1])) {
console.log(` ${code.padEnd(20)} ${String(n).padStart(4)} ${pct(n)}`);
}
console.log(`\nO1 by token type: ${JSON.stringify(byType)}`);
console.log(`O1 hand-verified-safe class (iso_date / api-version): ${recommended.length} — the rest mutilate identifiers, do NOT apply`);
console.log(`\nlocator aborts (fixable): ${locatorAborts} intrinsic aborts: ${o3.length - locatorAborts}`);
console.log(`\nrule distribution: ${JSON.stringify(byRule)}`);
console.log(`R8: ${r8.length} flags — O1 ${report.r8.O1}, aborts ${JSON.stringify(report.r8.codes)}`);
if (EXAMPLES > 0 && o1.length > 0) {
console.log(`\n--- ${Math.min(EXAMPLES, o1.length)} proven O1 proposals ---`);
for (const i of o1.slice(0, EXAMPLES)) {
console.log(`\n${i.file}:${i.proposal.line} [${i.rule}] ${i.token || i.proposal.token} -> ${i.proposal.replacement}`);
console.log(` - ${i.proposal.before}`);
console.log(` + ${i.proposal.after}`);
console.log(` quote: ${i.proposal.evidence_quote.slice(0, 160)}`);
}
}
if (argv.includes('--write')) {
const out = path.join(DATA, 'r11-pilot-classification.json');
fs.writeFileSync(out, JSON.stringify(report, null, 2) + '\n');
console.log(`\nwrote ${out}`);
} else {
console.log('\n(dry run — pass --write to persist r11-pilot-classification.json)');
}

View file

@ -0,0 +1,297 @@
// fix-op.mjs — R11 fix-operation classifier over judge-pass flags.
//
// Contract: docs/r11-tiered-fix-design.md §3 (the O1/O2/O3 partition is by
// OPERATION, not by rule code) and §4 (the O1 invariant).
//
// This module IS the O1 driver with writes disabled. It attempts the value swap
// and checks §4's three conditions; an item it cannot prove is O3 with a typed
// abort code. That is deliberate: a proxy heuristic would have measured
// something other than the mechanism that will later touch a public corpus.
//
// Two properties the callers depend on:
// - PURE. No fs, no network, no mutation of the input flag. The caller reads
// the file and passes its text.
// - FAILS CLOSED. Every path returns O1-with-proof or O3-with-a-known-code.
// A misrouted O3 costs one human review; a misrouted O1 ships a wrong edit
// to a publicly distributed file.
//
// What this module deliberately does NOT do: decide O2. Subtraction candidacy
// turns on which sub-assertion the judge's prose `reason` names as failing, and
// no regex reads prose. O2 requires operator ratification (§5) before it exists
// as a class at all; until then every non-O1 item is O3 by design.
/**
* Abort codes. The taxonomy is part of the contract, not diagnostics: the pilot's
* measurement #3 (§10) is the DISTRIBUTION of these, because "abort rate 85 %"
* is not actionable while "60 % LOCATOR_MISS" is an engineering gap and "60 %
* NOT_VERBATIM" is intrinsic to the corpus.
*/
export const ABORT_CODES = {
MULTI_PART_CLAIM: 'MULTI_PART_CLAIM', // enumeration / several assertions in one claim (§3, the R8 class)
NO_VALUE_TOKEN: 'NO_VALUE_TOKEN', // nothing swappable — the claim asserts prose
STATUS_SYNONYM: 'STATUS_SYNONYM', // GA/Preview class: file vocabulary != source vocabulary (operator question)
MULTI_VALUE_TOKEN: 'MULTI_VALUE_TOKEN', // several distinct values — which one is wrong is a judgement
LOCATOR_MISS: 'LOCATOR_MISS', // value not found in the block the flag points at
LOCATOR_AMBIGUOUS: 'LOCATOR_AMBIGUOUS', // value occurs more than once in that block
NOT_VERBATIM: 'NOT_VERBATIM', // no same-type replacement occurs verbatim in evidence_quote (§4.1)
MULTI_REPLACEMENT: 'MULTI_REPLACEMENT', // quote offers several candidate values
CONTEXT_MISMATCH: 'CONTEXT_MISMATCH', // §4 held but the tokens do not denote the same quantity (see below)
INVARIANT_FAIL: 'INVARIANT_FAIL', // swap constructed but §4 did not hold — must never happen silently
};
/** Verdict code for a proven swap. Kept out of ABORT_CODES so `op === 'O1' <=> code === 'PROVEN'`. */
export const PROVEN = 'PROVEN';
// Value types, most specific first. Matching is non-overlapping and priority
// ordered, so `2.3.0` is one version rather than two numbers, and `20 %` is a
// percent rather than the number 20. Types never cross in a swap: a percent may
// only be replaced by a percent.
const TOKEN_PATTERNS = [
['iso_date', /\d{4}-\d{2}-\d{2}/g],
['percent', /\d+(?:[.,]\d+)?\s?%/g],
['version', /v?\d+\.\d+\.\d+/g],
['number', /\d+(?:[.,]\d+)?/g],
];
// Lifecycle vocabulary. Present in a claim without any numeric token, this is the
// GA/Preview class: the corpus writes `**Preview**` / `**GA**` while the cited
// source writes "generally available". A swap would satisfy §4 literally while
// pasting English prose into a Norwegian table, so the class aborts and is put to
// the operator as a design question (a ratified synonym table, or permanent O3).
const STATUS_RE =
/\b(?:GA|generally available|allment tilgjengelig|public preview|private preview|preview|deprecated|utfaset|retired|avviklet)\b/i;
/**
* Extract swappable value tokens, non-overlapping and priority ordered.
* Status words are NOT value tokens see STATUS_RE.
* @returns {Array<{type: string, value: string, index: number}>} in order of appearance
*/
export function extractValueTokens(text) {
if (!text) return [];
const taken = []; // [start, end) ranges already consumed by a higher-priority type
const out = [];
for (const [type, re] of TOKEN_PATTERNS) {
re.lastIndex = 0;
let m;
while ((m = re.exec(text)) !== null) {
const start = m.index;
const end = start + m[0].length;
if (taken.some(([s, e]) => start < e && end > s)) continue;
taken.push([start, end]);
out.push({ type, value: m[0], index: start });
}
}
return out.sort((a, b) => a.index - b.index);
}
/** True if the text carries lifecycle-status vocabulary. */
export function hasStatusWord(text) {
return STATUS_RE.test(text || '');
}
/**
* The contiguous non-blank block containing `line` (1-indexed).
*
* This is the search window, and it is structural rather than a magic ±N: claims
* are LLM-extracted restatements whose `line` often points at a table HEADER
* while the asserted value sits in a row below. A block is exactly that table,
* list, or paragraph. A blank line degenerates to itself.
*/
export function blockWindow(lines, line) {
if (line < 1 || line > lines.length) return { start: line, end: line };
if (lines[line - 1].trim() === '') return { start: line, end: line };
let start = line;
let end = line;
while (start > 1 && lines[start - 2].trim() !== '') start -= 1;
while (end < lines.length && lines[end].trim() !== '') end += 1;
return { start, end };
}
// ---------------------------------------------------------- context condition
//
// MEASURED, NOT ASSUMED: §4 alone admits wrong edits. On the pilot it proved six
// swaps of which four were false — "30-dagers" -> "24" from a quote saying 24
// HOURS (unit crossing), an indexing rate replaced by a query throttle (metric
// crossing), and two identifiers mutilated by digits harvested out of "E7" and a
// table cell ("Microsoft Agent 365" -> "Agent 7", "text-embedding-ada-002" ->
// "ada-2"). §4 constrains where the new value CAME FROM and what the edit LOOKS
// LIKE; it constrains nothing about whether the two tokens denote the same
// quantity.
//
// The condition below adds that, and it is deliberately lexical rather than
// semantic: the token must sit under the same label, or the same trailing unit,
// on both sides. No translation table — "dokumenter" is not taught to equal
// "documents", because a synonym/translation table introduces a new fact source
// and is an operator decision (§5-class), not an engineering one. The consequence
// is measured and reported: a swap is provable essentially only where the context
// is language-neutral (a URL, a code sample, a parameter key).
const WORD = /[A-Za-z0-9_.\-æøåÆØÅ]/;
/** Normalise a context run for comparison: lowercase, punctuation stripped. */
const normContext = (s) => s.toLowerCase().replace(/[^a-z0-9æøå]/g, '');
/** The word run immediately left of [index], skipping any separator run first. */
function leftContext(text, index) {
let i = index - 1;
// A separator run may be skipped; a word character adjacent to the token may
// NOT be — that adjacency is what makes "7" part of the identifier "E7".
if (i >= 0 && !WORD.test(text[i])) {
while (i >= 0 && !WORD.test(text[i])) i -= 1;
}
let end = i + 1;
while (i >= 0 && WORD.test(text[i])) i -= 1;
return normContext(text.slice(i + 1, end));
}
/** The word run immediately right of [index], skipping any separator run first. */
function rightContext(text, index) {
let i = index;
if (i < text.length && !WORD.test(text[i])) {
while (i < text.length && !WORD.test(text[i])) i += 1;
}
const start = i;
while (i < text.length && WORD.test(text[i])) i += 1;
return normContext(text.slice(start, i));
}
/**
* Do the two occurrences sit in corresponding context? True when a non-empty
* label matches on the left, or a non-empty unit matches on the right.
*/
export function contextCorresponds(fileLine, fileIndex, fileLen, quote, quoteIndex, quoteLen) {
const lf = leftContext(fileLine, fileIndex);
const lq = leftContext(quote, quoteIndex);
if (lf && lf === lq) return true;
const rf = rightContext(fileLine, fileIndex + fileLen);
const rq = rightContext(quote, quoteIndex + quoteLen);
return Boolean(rf) && rf === rq;
}
/** Distinct by type+value, preserving order. */
function distinct(tokens) {
const seen = new Set();
return tokens.filter((t) => {
const k = `${t.type}${t.value}`;
if (seen.has(k)) return false;
seen.add(k);
return true;
});
}
function abort(code, detail = {}) {
return { op: 'O3', code, ...detail };
}
/**
* Classify one flag record into a fix operation.
*
* @param {object} flag flag record per docs/r11-flag-format-2026-07.md
* @param {string} fileText current content of flag.file
* @param {{contextCheck?: boolean}} [opts] contextCheck:false reproduces §4 exactly as
* written used to MEASURE what the context condition adds, never to ship edits.
* @returns {{op: 'O1'|'O3', code: string, proposal?: object, detail?: object}}
*/
export function classifyFlag(flag, fileText, opts = {}) {
const contextCheck = opts.contextCheck !== false;
const claim = flag.claim || '';
const lines = (fileText || '').split('\n');
// §3: an enumeration is not a value swap even when it contains a number. The
// structure decides, not the rule code — R8 is a signal, and the run records
// it, but it is not the partition.
if (claim.includes(' | ') || (claim.match(/,/g) || []).length >= 3) {
return abort(ABORT_CODES.MULTI_PART_CLAIM);
}
const tokens = distinct(extractValueTokens(claim));
if (tokens.length === 0) {
return abort(hasStatusWord(claim) ? ABORT_CODES.STATUS_SYNONYM : ABORT_CODES.NO_VALUE_TOKEN);
}
if (tokens.length > 1) {
return abort(ABORT_CODES.MULTI_VALUE_TOKEN, { detail: { candidates: tokens.map((t) => t.value) } });
}
const wrong = tokens[0];
// Locate the value in the block the flag points at. Unique hit or nothing:
// a locator that guesses is how a wrong edit reaches a public file.
if (flag.line < 1 || flag.line > lines.length) {
return abort(ABORT_CODES.LOCATOR_MISS, { detail: { reason: 'line out of range', line: flag.line } });
}
const win = blockWindow(lines, flag.line);
const hits = [];
for (let n = win.start; n <= win.end; n += 1) {
const text = lines[n - 1];
let from = 0;
for (;;) {
const at = text.indexOf(wrong.value, from);
if (at === -1) break;
hits.push({ line: n, index: at });
from = at + wrong.value.length;
}
}
if (hits.length === 0) return abort(ABORT_CODES.LOCATOR_MISS, { detail: { token: wrong.value, window: win } });
if (hits.length > 1) {
return abort(ABORT_CODES.LOCATOR_AMBIGUOUS, { detail: { token: wrong.value, hits: hits.length, window: win } });
}
// §4.1: the replacement must occur verbatim in the quote the judge actually
// cited. Same type only, and a quote that merely restates the claim's own value
// offers no replacement at all.
const replacements = distinct(extractValueTokens(flag.evidence_quote || '')).filter(
(t) => t.type === wrong.type && t.value !== wrong.value,
);
if (replacements.length === 0) return abort(ABORT_CODES.NOT_VERBATIM, { detail: { token: wrong.value, type: wrong.type } });
if (replacements.length > 1) {
return abort(ABORT_CODES.MULTI_REPLACEMENT, { detail: { candidates: replacements.map((t) => t.value) } });
}
const right = replacements[0];
const hit = hits[0];
const before = lines[hit.line - 1];
// The condition §4 is missing: same label or same unit on both sides.
if (
contextCheck &&
!contextCorresponds(before, hit.index, wrong.value.length, flag.evidence_quote || '', right.index, right.value.length)
) {
return abort(ABORT_CODES.CONTEXT_MISMATCH, {
detail: {
token: wrong.value,
replacement: right.value,
would_have_been: before.slice(0, hit.index) + right.value + before.slice(hit.index + wrong.value.length),
},
});
}
const after = before.slice(0, hit.index) + right.value + before.slice(hit.index + wrong.value.length);
// Re-check §4 against the constructed edit rather than trusting construction.
const quoteHasValue = (flag.evidence_quote || '').includes(right.value);
const restIdentical =
before.slice(0, hit.index) === after.slice(0, hit.index) &&
before.slice(hit.index + wrong.value.length) === after.slice(hit.index + right.value.length);
const rebuilt = lines.slice();
rebuilt[hit.line - 1] = after;
const changedLines = rebuilt.reduce((n, l, i) => n + (l === lines[i] ? 0 : 1), 0);
if (!quoteHasValue || !restIdentical || changedLines !== 1) {
return abort(ABORT_CODES.INVARIANT_FAIL, { detail: { quoteHasValue, restIdentical, changedLines } });
}
return {
op: 'O1',
code: PROVEN,
proposal: {
file: flag.file,
line: hit.line,
token: wrong.value,
replacement: right.value,
type: wrong.type,
before,
after,
evidence_url: flag.evidence_url,
evidence_quote: flag.evidence_quote,
},
};
}

Binary file not shown.