fix(sanitize): catch every invisible codepoint, exempt only on context
Variation-selector smuggling (A01) and Sneaky Bits (A02) gave 0 findings on 1.4.1 and stayed in the sanitized text. The denominator is now the frozen Unicode 18.0 set (DICP | Cf | Co, 141 674 codepoints); each one is stripped and flagged unless a known legitimate use exempts it on context: one VS15/16 after an emoji base, one ideographic selector after a CJK ideograph, ZWNJ/ZWJ beside Arabic-script or Indic letters, LRM/RLM/ALM in a document with RTL text, an Arabic number sign before a digit. A run is always a carrier; private use is caught in full (operator decisions 2026-10-28). One predicate serves sanitize, the output gate and the lexicon's word-split row, whose regex stays byte-identical with commons. A01 and A02 are green; the FP populations and the pinned OKF corpus are identical before and after.
This commit is contained in:
parent
08ca8f61bc
commit
c72e648a94
11 changed files with 734 additions and 108 deletions
|
|
@ -272,13 +272,13 @@ def test_redos_sweep_floor_and_flag_match_the_published_numbers():
|
|||
assert "2.6 flag threshold" in text
|
||||
|
||||
|
||||
def test_redos_sweep_collector_covers_152_patterns_across_11_tables():
|
||||
# The count docs/LIMITATIONS.md carries as "all 152 compiled patterns across
|
||||
def test_redos_sweep_collector_covers_154_patterns_across_11_tables():
|
||||
# The count docs/LIMITATIONS.md carries as "all 154 compiled patterns across
|
||||
# all eleven regex-bearing modules". A pattern added or removed in `src/`
|
||||
# without re-measuring would drift the doc's claim silently otherwise.
|
||||
assert len(redos_sweep.TABLES) == 11
|
||||
total = sum(len(collect()) for collect in redos_sweep.TABLES.values())
|
||||
assert total == 152
|
||||
assert total == 154
|
||||
|
||||
|
||||
# --- both scripts: the argument-less contract --------------------------------
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue