1
0
Fork 0

fix(sanitize): catch every invisible codepoint, exempt only on context

Variation-selector smuggling (A01) and Sneaky Bits (A02) gave 0 findings
on 1.4.1 and stayed in the sanitized text. The denominator is now the
frozen Unicode 18.0 set (DICP | Cf | Co, 141 674 codepoints); each one is
stripped and flagged unless a known legitimate use exempts it on context:
one VS15/16 after an emoji base, one ideographic selector after a CJK
ideograph, ZWNJ/ZWJ beside Arabic-script or Indic letters, LRM/RLM/ALM
in a document with RTL text, an Arabic number sign before a digit. A run
is always a carrier; private use is caught in full (operator decisions
2026-10-28). One predicate serves sanitize, the output gate and the
lexicon's word-split row, whose regex stays byte-identical with commons.

A01 and A02 are green; the FP populations and the pinned OKF corpus are
identical before and after.
This commit is contained in:
Kjell Tore Guttormsen 2026-10-29 08:31:43 +01:00
commit c72e648a94
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 734 additions and 108 deletions

View file

@ -272,13 +272,13 @@ def test_redos_sweep_floor_and_flag_match_the_published_numbers():
assert "2.6 flag threshold" in text
def test_redos_sweep_collector_covers_152_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 152 compiled patterns across
def test_redos_sweep_collector_covers_154_patterns_across_11_tables():
# The count docs/LIMITATIONS.md carries as "all 154 compiled patterns across
# all eleven regex-bearing modules". A pattern added or removed in `src/`
# without re-measuring would drift the doc's claim silently otherwise.
assert len(redos_sweep.TABLES) == 11
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
assert total == 152
assert total == 154
# --- both scripts: the argument-less contract --------------------------------