Kjell Tore Guttormsen
b1ba1fbdc6
First consumer swap of step 4. ZERO_WIDTH_CHARS (5), the Unicode Tag range,
BIDI_CHARS (9) and HOMOGLYPH_MAP (28) stop being hardcoded constants in
unicode-scanner.mjs and string-utils.mjs and are built from the vendored
commons artifact codepoints/carriers.json by the new lib/codepoints.mjs.
Started here rather than at injection-patterns, which the plan ordered first:
that table is the one table that cannot be loaded verbatim (the
hybrid-xss:script-tag divergence is directional, and loading the lexicon as-is
would reverse the 90f576f recall fix). The codepoint tables were measured
byte-equal to the source constants BEFORE the swap - same members, same
values, same insertion order on HOMOGLYPH_MAP - so they load verbatim.
Proof the swap is content-preserving: the golden dump differs in exactly one
record, the sha256 of string-utils.mjs, which changes by construction when a
table leaves the file. All 83 regex records and the
table:string-utils:HOMOGLYPH_MAP digest are byte-identical, and
reference-run.json is unchanged at 61/61. patterns.json is re-blessed for the
file digest alone.
The gate is proven red-capable against the SUBJECT, both directions:
- dropping U+00AD from the vendored zero_width table fails the new
codepoints gate by name, twice;
- altering one homoglyph value reddens the golden table digest AND a
behavioural homoglyph test.
That second direction is a property the swap creates rather than preserves:
the golden gate now transitively pins the vendored commons data, where before
it pinned a source literal and a commons mutation was invisible to it.
NOT ported: commons carries cyrillic_confusables (13), and unicode-scanner.mjs
declares a set by that name - but nothing reads it. The homoglyph-mixing
detector tests isCyrillic(cp), the whole U+0400-U+04FF block. Loading it would
move dead data into the load path, so the dead const stays where it is and is
recorded instead. The recorded v8.x-B i/x drift between that set and the
lexicon class is therefore latent, not live. commons' private_use table has no
constant behind it here at all.
Graceful-empty is kept deliberately: codepoints.mjs is on string-utils'
import path and hooks import string-utils in fresh per-tool-call processes, so
a module-load throw would break the tool call rather than degrade the scan.
The loud half is the test, which asserts exact per-table counts through the
real default commons root - the same shape as the lexicon load-assertion.
Drive-by, unavoidable: the deleted JSDoc carried the "~25 entries" claim for a
28-entry table (v8.x-C). It needed a re-bless of the same file digest this
swap already forces, so it closes here at no extra cost.
Suite 2158 -> 2164, all green.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01X7XEEFrAJsREqa9N4tpfm8
105 lines
4.4 KiB
JavaScript
105 lines
4.4 KiB
JavaScript
// codepoints.mjs — Code-point carrier tables, built from vendored commons.
|
||
//
|
||
// v8 Phase 5 step 4, first consumer swap. These four tables were hardcoded
|
||
// constants in two places (unicode-scanner.mjs's charset block and
|
||
// string-utils.mjs's HOMOGLYPH_MAP); they are now built once, here, from
|
||
// `codepoints/carriers.json` in the vendored llm-security-commons subtree.
|
||
// Verified byte-equal to the pre-swap constants before the swap: 5 zero-width,
|
||
// 9 BIDI, 28 homoglyph entries, U+E0001–U+E007F, same values, same order.
|
||
//
|
||
// commons carries two further tables this module does NOT build:
|
||
// - `cyrillic_confusables` (13): unicode-scanner.mjs declares a set by that
|
||
// name but never reads it — its homoglyph-mixing detector tests
|
||
// `isCyrillic(cp)`, the whole U+0400–U+04FF block. Loading a table no
|
||
// runtime consumes would move dead data into the load path.
|
||
// - `private_use`: no constant behind it here at all; commons transcribed it
|
||
// from a source comment and marks it `verified: false`.
|
||
//
|
||
// Graceful-empty, deliberately, matching commons-loader.mjs's contract: this
|
||
// module is on the import path of string-utils.mjs, which hooks import, and
|
||
// hooks run per-tool-call in fresh processes. A module-load throw there would
|
||
// break the tool call rather than just degrade the scan. The cost of that
|
||
// choice is that a lost commons is silent at runtime, so the loud half lives
|
||
// in `tests/lib/codepoints.test.mjs`, which asserts exact per-table counts
|
||
// through the real default root.
|
||
//
|
||
// Zero external dependencies — Node.js builtins only.
|
||
|
||
import { loadArtifact } from './commons-loader.mjs';
|
||
|
||
/** `"U+200B"` -> `0x200B`. Returns NaN for anything malformed, filtered by the callers. */
|
||
function parseCodepoint(value) {
|
||
if (typeof value !== 'string') return NaN;
|
||
const m = /^U\+([0-9A-Fa-f]{4,6})$/.exec(value.trim());
|
||
return m ? parseInt(m[1], 16) : NaN;
|
||
}
|
||
|
||
function codepointSet(entries) {
|
||
const out = new Set();
|
||
if (!Array.isArray(entries)) return out;
|
||
for (const entry of entries) {
|
||
const cp = parseCodepoint(entry?.codepoint);
|
||
if (!Number.isNaN(cp)) out.add(cp);
|
||
}
|
||
return out;
|
||
}
|
||
|
||
/**
|
||
* Build the carrier tables from a commons root.
|
||
*
|
||
* @param {object} [opts]
|
||
* @param {string} [opts.commonsRoot] - explicit commons root (tests, dev checkout).
|
||
* @returns {{
|
||
* ZERO_WIDTH_CHARS: Set<number>,
|
||
* BIDI_CHARS: Set<number>,
|
||
* UNICODE_TAG_START: number,
|
||
* UNICODE_TAG_END: number,
|
||
* HOMOGLYPH_MAP: Readonly<Record<string, string>>,
|
||
* }}
|
||
*/
|
||
export function buildCarrierTables(opts = {}) {
|
||
const artifact = loadArtifact('codepoints/carriers', { fallback: {}, commonsRoot: opts.commonsRoot });
|
||
const tables = artifact?.tables ?? {};
|
||
|
||
const tagStart = parseCodepoint(tables.unicode_tags?.range?.start);
|
||
const tagEnd = parseCodepoint(tables.unicode_tags?.range?.end);
|
||
// An absent range must match nothing. It is a comparison, not a set, so
|
||
// emptiness is expressed as an inverted range rather than a zero-size table.
|
||
const rangeUsable = !Number.isNaN(tagStart) && !Number.isNaN(tagEnd);
|
||
|
||
const homoglyphs = {};
|
||
for (const entry of tables.homoglyph_map?.entries ?? []) {
|
||
if (typeof entry?.from_char === 'string' && typeof entry?.to === 'string') {
|
||
homoglyphs[entry.from_char] = entry.to;
|
||
}
|
||
}
|
||
|
||
return {
|
||
ZERO_WIDTH_CHARS: codepointSet(tables.zero_width?.codepoints),
|
||
BIDI_CHARS: codepointSet(tables.bidi?.codepoints),
|
||
UNICODE_TAG_START: rangeUsable ? tagStart : Infinity,
|
||
UNICODE_TAG_END: rangeUsable ? tagEnd : -Infinity,
|
||
HOMOGLYPH_MAP: Object.freeze(homoglyphs),
|
||
};
|
||
}
|
||
|
||
const _tables = buildCarrierTables();
|
||
|
||
/** U+200B–U+200D, U+FEFF, U+00AD: visually invisible, used to hide content */
|
||
export const ZERO_WIDTH_CHARS = _tables.ZERO_WIDTH_CHARS;
|
||
|
||
/** BIDI control characters — Trojan Source attack (CVE-2021-42574 class) */
|
||
export const BIDI_CHARS = _tables.BIDI_CHARS;
|
||
|
||
/** Unicode Tags block U+E0001–U+E007F: encodes hidden ASCII via codepoint - 0xE0000 */
|
||
export const UNICODE_TAG_START = _tables.UNICODE_TAG_START;
|
||
export const UNICODE_TAG_END = _tables.UNICODE_TAG_END;
|
||
|
||
/**
|
||
* Confusable → Latin fold table (Cyrillic + Greek), applied by
|
||
* `foldHomoglyphs` after NFKC. Deliberately small: Latin Extended letters
|
||
* (æ, ø, å, é, ñ, ü, ...) are legitimate in non-English source and are
|
||
* excluded by design, as are non-overlapping Cyrillic/Greek letters and the
|
||
* U+1D400 mathematical block (NFKC already handles it).
|
||
*/
|
||
export const HOMOGLYPH_MAP = _tables.HOMOGLYPH_MAP;
|