refactor(llm-security): v8 Phase 5 step 4 - swap codepoint tables to commons

First consumer swap of step 4. ZERO_WIDTH_CHARS (5), the Unicode Tag range,
BIDI_CHARS (9) and HOMOGLYPH_MAP (28) stop being hardcoded constants in
unicode-scanner.mjs and string-utils.mjs and are built from the vendored
commons artifact codepoints/carriers.json by the new lib/codepoints.mjs.

Started here rather than at injection-patterns, which the plan ordered first:
that table is the one table that cannot be loaded verbatim (the
hybrid-xss:script-tag divergence is directional, and loading the lexicon as-is
would reverse the 90f576f recall fix). The codepoint tables were measured
byte-equal to the source constants BEFORE the swap - same members, same
values, same insertion order on HOMOGLYPH_MAP - so they load verbatim.

Proof the swap is content-preserving: the golden dump differs in exactly one
record, the sha256 of string-utils.mjs, which changes by construction when a
table leaves the file. All 83 regex records and the
table:string-utils:HOMOGLYPH_MAP digest are byte-identical, and
reference-run.json is unchanged at 61/61. patterns.json is re-blessed for the
file digest alone.

The gate is proven red-capable against the SUBJECT, both directions:
- dropping U+00AD from the vendored zero_width table fails the new
  codepoints gate by name, twice;
- altering one homoglyph value reddens the golden table digest AND a
  behavioural homoglyph test.
That second direction is a property the swap creates rather than preserves:
the golden gate now transitively pins the vendored commons data, where before
it pinned a source literal and a commons mutation was invisible to it.

NOT ported: commons carries cyrillic_confusables (13), and unicode-scanner.mjs
declares a set by that name - but nothing reads it. The homoglyph-mixing
detector tests isCyrillic(cp), the whole U+0400-U+04FF block. Loading it would
move dead data into the load path, so the dead const stays where it is and is
recorded instead. The recorded v8.x-B i/x drift between that set and the
lexicon class is therefore latent, not live. commons' private_use table has no
constant behind it here at all.

Graceful-empty is kept deliberately: codepoints.mjs is on string-utils'
import path and hooks import string-utils in fresh per-tool-call processes, so
a module-load throw would break the tool call rather than degrade the scan.
The loud half is the test, which asserts exact per-table counts through the
real default commons root - the same shape as the lexicon load-assertion.

Drive-by, unavoidable: the deleted JSDoc carried the "~25 entries" claim for a
28-entry table (v8.x-C). It needed a re-bless of the same file digest this
swap already forces, so it closes here at no extra cost.

Suite 2158 -> 2164, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7XEEFrAJsREqa9N4tpfm8

scanners/lib/codepoints.mjs

@@ -0,0 +1,105 @@
+// codepoints.mjs — Code-point carrier tables, built from vendored commons.
+//
+// v8 Phase 5 step 4, first consumer swap. These four tables were hardcoded
+// constants in two places (unicode-scanner.mjs's charset block and
+// string-utils.mjs's HOMOGLYPH_MAP); they are now built once, here, from
+// `codepoints/carriers.json` in the vendored llm-security-commons subtree.
+// Verified byte-equal to the pre-swap constants before the swap: 5 zero-width,
+// 9 BIDI, 28 homoglyph entries, U+E0001–U+E007F, same values, same order.
+//
+// commons carries two further tables this module does NOT build:
+//   - `cyrillic_confusables` (13): unicode-scanner.mjs declares a set by that
+//     name but never reads it — its homoglyph-mixing detector tests
+//     `isCyrillic(cp)`, the whole U+0400–U+04FF block. Loading a table no
+//     runtime consumes would move dead data into the load path.
+//   - `private_use`: no constant behind it here at all; commons transcribed it
+//     from a source comment and marks it `verified: false`.
+//
+// Graceful-empty, deliberately, matching commons-loader.mjs's contract: this
+// module is on the import path of string-utils.mjs, which hooks import, and
+// hooks run per-tool-call in fresh processes. A module-load throw there would
+// break the tool call rather than just degrade the scan. The cost of that
+// choice is that a lost commons is silent at runtime, so the loud half lives
+// in `tests/lib/codepoints.test.mjs`, which asserts exact per-table counts
+// through the real default root.
+//
+// Zero external dependencies — Node.js builtins only.
+
+import { loadArtifact } from './commons-loader.mjs';
+
+/** `"U+200B"` -> `0x200B`. Returns NaN for anything malformed, filtered by the callers. */
+function parseCodepoint(value) {
+  if (typeof value !== 'string') return NaN;
+  const m = /^U\+([0-9A-Fa-f]{4,6})$/.exec(value.trim());
+  return m ? parseInt(m[1], 16) : NaN;
+}
+
+function codepointSet(entries) {
+  const out = new Set();
+  if (!Array.isArray(entries)) return out;
+  for (const entry of entries) {
+    const cp = parseCodepoint(entry?.codepoint);
+    if (!Number.isNaN(cp)) out.add(cp);
+  }
+  return out;
+}
+
+/**
+ * Build the carrier tables from a commons root.
+ *
+ * @param {object} [opts]
+ * @param {string} [opts.commonsRoot] - explicit commons root (tests, dev checkout).
+ * @returns {{
+ *   ZERO_WIDTH_CHARS: Set<number>,
+ *   BIDI_CHARS: Set<number>,
+ *   UNICODE_TAG_START: number,
+ *   UNICODE_TAG_END: number,
+ *   HOMOGLYPH_MAP: Readonly<Record<string, string>>,
+ * }}
+ */
+export function buildCarrierTables(opts = {}) {
+  const artifact = loadArtifact('codepoints/carriers', { fallback: {}, commonsRoot: opts.commonsRoot });
+  const tables = artifact?.tables ?? {};
+
+  const tagStart = parseCodepoint(tables.unicode_tags?.range?.start);
+  const tagEnd = parseCodepoint(tables.unicode_tags?.range?.end);
+  // An absent range must match nothing. It is a comparison, not a set, so
+  // emptiness is expressed as an inverted range rather than a zero-size table.
+  const rangeUsable = !Number.isNaN(tagStart) && !Number.isNaN(tagEnd);
+
+  const homoglyphs = {};
+  for (const entry of tables.homoglyph_map?.entries ?? []) {
+    if (typeof entry?.from_char === 'string' && typeof entry?.to === 'string') {
+      homoglyphs[entry.from_char] = entry.to;
+    }
+  }
+
+  return {
+    ZERO_WIDTH_CHARS: codepointSet(tables.zero_width?.codepoints),
+    BIDI_CHARS: codepointSet(tables.bidi?.codepoints),
+    UNICODE_TAG_START: rangeUsable ? tagStart : Infinity,
+    UNICODE_TAG_END: rangeUsable ? tagEnd : -Infinity,
+    HOMOGLYPH_MAP: Object.freeze(homoglyphs),
+  };
+}
+
+const _tables = buildCarrierTables();
+
+/** U+200B–U+200D, U+FEFF, U+00AD: visually invisible, used to hide content */
+export const ZERO_WIDTH_CHARS = _tables.ZERO_WIDTH_CHARS;
+
+/** BIDI control characters — Trojan Source attack (CVE-2021-42574 class) */
+export const BIDI_CHARS = _tables.BIDI_CHARS;
+
+/** Unicode Tags block U+E0001–U+E007F: encodes hidden ASCII via codepoint - 0xE0000 */
+export const UNICODE_TAG_START = _tables.UNICODE_TAG_START;
+export const UNICODE_TAG_END = _tables.UNICODE_TAG_END;
+
+/**
+ * Confusable → Latin fold table (Cyrillic + Greek), applied by
+ * `foldHomoglyphs` after NFKC. Deliberately small: Latin Extended letters
+ * (æ, ø, å, é, ñ, ü, ...) are legitimate in non-English source and are
+ * excluded by design, as are non-overlapping Cyrillic/Greek letters and the
+ * U+1D400 mathematical block (NFKC already handles it).
+ */
+export const HOMOGLYPH_MAP = _tables.HOMOGLYPH_MAP;