fix(zwj): the zero-width check tested identity, so every emoji-composed document was hard-blocked
`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
`disposition._CARRIER_LABELS` grades both as any-tier FAIL_SECURE with no
appeal, so any first-party document containing a ZWJ-composed emoji --
professions, families, skin tones, flag variants -- was hard-blocked forever.
Reported by ms-ai-architect; confirmed here against the code.
The strip was the worse half and was not in the report: sanitize *removed* the
joiner, silently decomposing the emoji into two unrelated ones. A module whose
contract is "only ever removes carriers" was corrupting content.
The fix is the one our own lexicon row `unicode:zero-width-in-word` (`\w[ZW]\w`)
already used: judge the joiner by CONTEXT, not identity. A ZWJ is exempt only
when BOTH neighbours are emoji-context codepoints. Half-context is not context,
so `a<ZWJ>{emoji}` stays a carrier and an attacker cannot buy exemption with a
single emoji.
Blocks, not an emoji table. Measured against Unicode 17.0's
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
adjacent to a ZWJ, and the five ranges cover 122/122. The measurement earned
its keep -- the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
Shipping the RGI list itself would be exact on the day it landed and stale at
the next Unicode release, reopening this same false positive for every new
emoji; whole blocks carry the unassigned headroom (458 Cn codepoints) that
future emoji are allocated into, so the table does not age.
The predicate is defined once in sanitize and imported by output. A second copy
is how the input side stops flagging while the output side keeps blocking; the
cross-surface test asserts the two agree on six inputs.
Two residuals, both in LIMITATIONS (33 -> 34, README bumped): a ZWJ between two
emoji is now exempt and could carry a covert channel (one emoji per bit, cannot
split a word); and U+200C (ZWNJ) still has no context test, so Persian, Arabic
and Devanagari documents -- where it is orthographically required -- stay
blocked. That needs a script-based criterion and no corpus is here to verify it
against, so it is parked as a known FP class rather than guessed at.
736 green (was 727), coverage 128/128, 6/6 documented gaps hold.
This commit is contained in:
parent
1d49f83a61
commit
a59184bb7f
6 changed files with 193 additions and 4 deletions
|
|
@ -480,6 +480,24 @@ fails the test, forcing this doc to be updated:
|
|||
consumer-notification promise in `docs/PLAN-v1.md`. Deferred deliberately, not
|
||||
overlooked.
|
||||
|
||||
- **A ZWJ hidden between two emoji is exempt, and ZWNJ's own false-positive
|
||||
class is untouched.** U+200D composes emoji (👩💻 is WOMAN + ZWJ + PERSONAL
|
||||
COMPUTER), so testing it on codepoint membership alone flagged *and stripped*
|
||||
the joiner in any document containing a ZWJ-composed emoji — an any-tier
|
||||
`FAIL_SECURE` carrier, plus silent decomposition of the emoji, on ordinary
|
||||
first-party content. The joiner is now judged by context instead: exempt only
|
||||
when **both** neighbours are emoji-context codepoints, measured to cover
|
||||
122/122 of the codepoints adjacent to a ZWJ across Unicode 17.0's 1614 RGI
|
||||
sequences. Two residuals follow. First, a joiner placed *between two emoji*
|
||||
is now invisible to the carrier check and could carry a covert channel — it
|
||||
costs an emoji per bit and cannot split a word, which is the shape the
|
||||
word-splitting attack needs, so the narrowing is deliberate rather than
|
||||
complete. Second, **U+200C (ZWNJ) still has no context test**, and it is
|
||||
orthographically *required* in Persian, Arabic and Devanagari — those
|
||||
documents remain hard-blocked. That is a separate criterion (script-based,
|
||||
not pictographic) and no corpus is available here to verify it against, so it
|
||||
is parked as a known false-positive class rather than guessed at.
|
||||
|
||||
## Out-of-scope (documented boundary)
|
||||
|
||||
Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue