refactor(llm-security): v8 Phase 5 step 4 - swap codepoint tables to commons

First consumer swap of step 4. ZERO_WIDTH_CHARS (5), the Unicode Tag range,
BIDI_CHARS (9) and HOMOGLYPH_MAP (28) stop being hardcoded constants in
unicode-scanner.mjs and string-utils.mjs and are built from the vendored
commons artifact codepoints/carriers.json by the new lib/codepoints.mjs.

Started here rather than at injection-patterns, which the plan ordered first:
that table is the one table that cannot be loaded verbatim (the
hybrid-xss:script-tag divergence is directional, and loading the lexicon as-is
would reverse the 90f576f recall fix). The codepoint tables were measured
byte-equal to the source constants BEFORE the swap - same members, same
values, same insertion order on HOMOGLYPH_MAP - so they load verbatim.

Proof the swap is content-preserving: the golden dump differs in exactly one
record, the sha256 of string-utils.mjs, which changes by construction when a
table leaves the file. All 83 regex records and the
table:string-utils:HOMOGLYPH_MAP digest are byte-identical, and
reference-run.json is unchanged at 61/61. patterns.json is re-blessed for the
file digest alone.

The gate is proven red-capable against the SUBJECT, both directions:
- dropping U+00AD from the vendored zero_width table fails the new
  codepoints gate by name, twice;
- altering one homoglyph value reddens the golden table digest AND a
  behavioural homoglyph test.
That second direction is a property the swap creates rather than preserves:
the golden gate now transitively pins the vendored commons data, where before
it pinned a source literal and a commons mutation was invisible to it.

NOT ported: commons carries cyrillic_confusables (13), and unicode-scanner.mjs
declares a set by that name - but nothing reads it. The homoglyph-mixing
detector tests isCyrillic(cp), the whole U+0400-U+04FF block. Loading it would
move dead data into the load path, so the dead const stays where it is and is
recorded instead. The recorded v8.x-B i/x drift between that set and the
lexicon class is therefore latent, not live. commons' private_use table has no
constant behind it here at all.

Graceful-empty is kept deliberately: codepoints.mjs is on string-utils'
import path and hooks import string-utils in fresh per-tool-call processes, so
a module-load throw would break the tool call rather than degrade the scan.
The loud half is the test, which asserts exact per-table counts through the
real default commons root - the same shape as the lexicon load-assertion.

Drive-by, unavoidable: the deleted JSDoc carried the "~25 entries" claim for a
28-entry table (v8.x-C). It needed a re-bless of the same file digest this
swap already forces, so it closes here at no extra cost.

Suite 2158 -> 2164, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7XEEFrAJsREqa9N4tpfm8

scanners/lib/codepoints.mjs

@@ -0,0 +1,105 @@
+// codepoints.mjs — Code-point carrier tables, built from vendored commons.
+//
+// v8 Phase 5 step 4, first consumer swap. These four tables were hardcoded
+// constants in two places (unicode-scanner.mjs's charset block and
+// string-utils.mjs's HOMOGLYPH_MAP); they are now built once, here, from
+// `codepoints/carriers.json` in the vendored llm-security-commons subtree.
+// Verified byte-equal to the pre-swap constants before the swap: 5 zero-width,
+// 9 BIDI, 28 homoglyph entries, U+E0001–U+E007F, same values, same order.
+//
+// commons carries two further tables this module does NOT build:
+//   - `cyrillic_confusables` (13): unicode-scanner.mjs declares a set by that
+//     name but never reads it — its homoglyph-mixing detector tests
+//     `isCyrillic(cp)`, the whole U+0400–U+04FF block. Loading a table no
+//     runtime consumes would move dead data into the load path.
+//   - `private_use`: no constant behind it here at all; commons transcribed it
+//     from a source comment and marks it `verified: false`.
+//
+// Graceful-empty, deliberately, matching commons-loader.mjs's contract: this
+// module is on the import path of string-utils.mjs, which hooks import, and
+// hooks run per-tool-call in fresh processes. A module-load throw there would
+// break the tool call rather than just degrade the scan. The cost of that
+// choice is that a lost commons is silent at runtime, so the loud half lives
+// in `tests/lib/codepoints.test.mjs`, which asserts exact per-table counts
+// through the real default root.
+//
+// Zero external dependencies — Node.js builtins only.
+
+import { loadArtifact } from './commons-loader.mjs';
+
+/** `"U+200B"` -> `0x200B`. Returns NaN for anything malformed, filtered by the callers. */
+function parseCodepoint(value) {
+  if (typeof value !== 'string') return NaN;
+  const m = /^U\+([0-9A-Fa-f]{4,6})$/.exec(value.trim());
+  return m ? parseInt(m[1], 16) : NaN;
+}
+
+function codepointSet(entries) {
+  const out = new Set();
+  if (!Array.isArray(entries)) return out;
+  for (const entry of entries) {
+    const cp = parseCodepoint(entry?.codepoint);
+    if (!Number.isNaN(cp)) out.add(cp);
+  }
+  return out;
+}
+
+/**
+ * Build the carrier tables from a commons root.
+ *
+ * @param {object} [opts]
+ * @param {string} [opts.commonsRoot] - explicit commons root (tests, dev checkout).
+ * @returns {{
+ *   ZERO_WIDTH_CHARS: Set<number>,
+ *   BIDI_CHARS: Set<number>,
+ *   UNICODE_TAG_START: number,
+ *   UNICODE_TAG_END: number,
+ *   HOMOGLYPH_MAP: Readonly<Record<string, string>>,
+ * }}
+ */
+export function buildCarrierTables(opts = {}) {
+  const artifact = loadArtifact('codepoints/carriers', { fallback: {}, commonsRoot: opts.commonsRoot });
+  const tables = artifact?.tables ?? {};
+
+  const tagStart = parseCodepoint(tables.unicode_tags?.range?.start);
+  const tagEnd = parseCodepoint(tables.unicode_tags?.range?.end);
+  // An absent range must match nothing. It is a comparison, not a set, so
+  // emptiness is expressed as an inverted range rather than a zero-size table.
+  const rangeUsable = !Number.isNaN(tagStart) && !Number.isNaN(tagEnd);
+
+  const homoglyphs = {};
+  for (const entry of tables.homoglyph_map?.entries ?? []) {
+    if (typeof entry?.from_char === 'string' && typeof entry?.to === 'string') {
+      homoglyphs[entry.from_char] = entry.to;
+    }
+  }
+
+  return {
+    ZERO_WIDTH_CHARS: codepointSet(tables.zero_width?.codepoints),
+    BIDI_CHARS: codepointSet(tables.bidi?.codepoints),
+    UNICODE_TAG_START: rangeUsable ? tagStart : Infinity,
+    UNICODE_TAG_END: rangeUsable ? tagEnd : -Infinity,
+    HOMOGLYPH_MAP: Object.freeze(homoglyphs),
+  };
+}
+
+const _tables = buildCarrierTables();
+
+/** U+200B–U+200D, U+FEFF, U+00AD: visually invisible, used to hide content */
+export const ZERO_WIDTH_CHARS = _tables.ZERO_WIDTH_CHARS;
+
+/** BIDI control characters — Trojan Source attack (CVE-2021-42574 class) */
+export const BIDI_CHARS = _tables.BIDI_CHARS;
+
+/** Unicode Tags block U+E0001–U+E007F: encodes hidden ASCII via codepoint - 0xE0000 */
+export const UNICODE_TAG_START = _tables.UNICODE_TAG_START;
+export const UNICODE_TAG_END = _tables.UNICODE_TAG_END;
+
+/**
+ * Confusable → Latin fold table (Cyrillic + Greek), applied by
+ * `foldHomoglyphs` after NFKC. Deliberately small: Latin Extended letters
+ * (æ, ø, å, é, ñ, ü, ...) are legitimate in non-English source and are
+ * excluded by design, as are non-overlapping Cyrillic/Greek letters and the
+ * U+1D400 mathematical block (NFKC already handles it).
+ */
+export const HOMOGLYPH_MAP = _tables.HOMOGLYPH_MAP;

scanners/lib/string-utils.mjs

@@ -1,5 +1,10 @@
 // string-utils.mjs — Entropy, Levenshtein, base64 detection, redaction, decoding
-// Zero dependencies.
+// Zero external dependencies. One internal import: the homoglyph fold table is
+// built from vendored commons by `codepoints.mjs` (v8 Phase 5 step 4), which
+// reads a local JSON file at module load and degrades to an empty table rather
+// than throwing — hooks import this module in fresh per-tool-call processes.
+
+import { HOMOGLYPH_MAP } from './codepoints.mjs';
 
 /**
  * Shannon entropy of a string (bits per character).
@@ -432,58 +437,22 @@ export function stripBidiOverrides(s) {
  * focused on letters that appear in injection vocabulary
  * (`ignore`, `system`, `you are`, `assistant`, `tool`, `response`).
  *
- * Excluded by design:
- *   - Latin Extended characters (æ, ø, å, é, è, ñ, ü, ö, ä, ç, ß, þ, ð, etc.)
- *     — these are legitimate letters in Norwegian, German, Danish, Spanish,
- *     French, Icelandic, etc., and would generate false positives in
- *     non-English source code or documentation.
- *   - Greek letters that don't visually overlap with Latin (`β`, `γ`, `δ`, ...)
- *   - Cyrillic letters that don't visually overlap (`б`, `г`, `д`, `ж`, ...)
- *   - Mathematical alphanumeric symbols (the U+1D400 block) — covered by
- *     NFKC normalization in `foldHomoglyphs` itself.
+ * The 28 entries live in the vendored commons artifact
+ * `scanners/commons/codepoints/carriers.json` (`tables.homoglyph_map`) as of
+ * the v8 Phase 5 step 4 swap, and are built into this object by
+ * `codepoints.mjs`. The exclusions are recorded with the data there — in
+ * short: Latin Extended letters (æ, ø, å, é, ñ, ü, ...) are legitimate in
+ * non-English source, non-overlapping Cyrillic/Greek letters carry no
+ * confusion, and the U+1D400 mathematical block is already handled by the
+ * NFKC pass in `foldHomoglyphs` itself.
  *
- * The map is deliberately small (~25 entries). Adding more risks
- * false-positive escalation on benign multilingual content.
- *
- * Exported for the v8 golden gate (`tests/golden/patterns.json`), which pins a
- * digest of this table so a Phase 5 extraction into commons is provably
- * behaviour-preserving. `foldHomoglyphs` remains the only intended consumer —
- * the export is an observation point, not an invitation to fold by hand.
+ * Re-exported, not re-declared: this name is published surface and the golden
+ * gate (`tests/golden/patterns.json`) pins a digest of the table through it,
+ * so the swap is provably content-preserving. `foldHomoglyphs` remains the
+ * only intended consumer — the export is an observation point, not an
+ * invitation to fold by hand.
  */
-export const HOMOGLYPH_MAP = Object.freeze({
-  // Cyrillic → Latin (lowercase)
-  'а': 'a',  // U+0430
-  'е': 'e',  // U+0435
-  'о': 'o',  // U+043E
-  'с': 'c',  // U+0441
-  'р': 'p',  // U+0440
-  'х': 'x',  // U+0445
-  'у': 'y',  // U+0443
-  'і': 'i',  // U+0456 (Ukrainian)
-  'ј': 'j',  // U+0458
-  'ѕ': 's',  // U+0455
-  'ӏ': 'l',  // U+04CF (Cyrillic Palochka)
-  // Cyrillic → Latin (uppercase)
-  'А': 'A',  // U+0410
-  'Е': 'E',  // U+0415
-  'О': 'O',  // U+041E
-  'С': 'C',  // U+0421
-  'Р': 'P',  // U+0420
-  'Х': 'X',  // U+0425
-  'У': 'Y',  // U+0423
-  // Greek → Latin (only the unambiguous Latin-look-alikes)
-  'α': 'a',  // U+03B1
-  'ο': 'o',  // U+03BF
-  'ρ': 'p',  // U+03C1
-  'ι': 'i',  // U+03B9
-  'ν': 'v',  // U+03BD
-  'τ': 't',  // U+03C4
-  // Greek uppercase
-  'Α': 'A',  // U+0391
-  'Ο': 'O',  // U+039F
-  'Ρ': 'P',  // U+03A1
-  'Τ': 'T',  // U+03A4
-});
+export { HOMOGLYPH_MAP };
 
 /**
  * Fold visually-confusable characters to their Latin look-alikes. Used by

scanners/unicode-scanner.mjs

@@ -8,37 +8,19 @@
 import { readTextFile } from './lib/file-discovery.mjs';
 import { finding, scannerResult } from './lib/output.mjs';
 import { SEVERITY } from './lib/severity.mjs';
+import { ZERO_WIDTH_CHARS, BIDI_CHARS, UNICODE_TAG_START, UNICODE_TAG_END } from './lib/codepoints.mjs';
 
 // ---------------------------------------------------------------------------
 // Character sets
+//
+// ZERO_WIDTH_CHARS (5), the Unicode Tag range and BIDI_CHARS (9) are built
+// from the vendored commons artifact `codepoints/carriers.json` as of the v8
+// Phase 5 step 4 swap — see `lib/codepoints.mjs`. CYRILLIC_CONFUSABLES below
+// is NOT: commons carries it, but nothing in this file reads it (the
+// homoglyph-mixing detector tests `isCyrillic(cp)`, the whole U+0400–U+04FF
+// block), so loading it would move dead data into the load path.
 // ---------------------------------------------------------------------------
 
-/** U+200B–U+200D, U+FEFF, U+00AD: visually invisible, used to hide content */
-const ZERO_WIDTH_CHARS = new Set([
-  0x200B, // ZERO WIDTH SPACE
-  0x200C, // ZERO WIDTH NON-JOINER
-  0x200D, // ZERO WIDTH JOINER
-  0xFEFF,  // ZERO WIDTH NO-BREAK SPACE / BOM (when not at position 0)
-  0x00AD,  // SOFT HYPHEN
-]);
-
-/** Unicode Tags block U+E0001–U+E007F: encodes hidden ASCII via codepoint - 0xE0000 */
-const UNICODE_TAG_START = 0xE0001;
-const UNICODE_TAG_END   = 0xE007F;
-
-/** BIDI control characters — Trojan Source attack (CVE-2021-42574 class) */
-const BIDI_CHARS = new Set([
-  0x202A, // LEFT-TO-RIGHT EMBEDDING
-  0x202B, // RIGHT-TO-LEFT EMBEDDING
-  0x202C, // POP DIRECTIONAL FORMATTING
-  0x202D, // LEFT-TO-RIGHT OVERRIDE
-  0x202E, // RIGHT-TO-LEFT OVERRIDE
-  0x2066, // LEFT-TO-RIGHT ISOLATE
-  0x2067, // RIGHT-TO-LEFT ISOLATE
-  0x2068, // FIRST STRONG ISOLATE
-  0x2069, // POP DIRECTIONAL ISOLATE
-]);
-
 /** Cyrillic lookalike codepoints that visually match Latin letters */
 const CYRILLIC_CONFUSABLES = new Set([
   0x0430, // а — Cyrillic small letter a  (looks like Latin a)