fix(zwj): the zero-width check tested identity, so every emoji-composed document was hard-blocked

`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
`disposition._CARRIER_LABELS` grades both as any-tier FAIL_SECURE with no
appeal, so any first-party document containing a ZWJ-composed emoji --
professions, families, skin tones, flag variants -- was hard-blocked forever.
Reported by ms-ai-architect; confirmed here against the code.

The strip was the worse half and was not in the report: sanitize *removed* the
joiner, silently decomposing the emoji into two unrelated ones. A module whose
contract is "only ever removes carriers" was corrupting content.

The fix is the one our own lexicon row `unicode:zero-width-in-word` (`\w[ZW]\w`)
already used: judge the joiner by CONTEXT, not identity. A ZWJ is exempt only
when BOTH neighbours are emoji-context codepoints. Half-context is not context,
so `a<ZWJ>{emoji}` stays a carrier and an attacker cannot buy exemption with a
single emoji.

Blocks, not an emoji table. Measured against Unicode 17.0's
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
adjacent to a ZWJ, and the five ranges cover 122/122. The measurement earned
its keep -- the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
Shipping the RGI list itself would be exact on the day it landed and stale at
the next Unicode release, reopening this same false positive for every new
emoji; whole blocks carry the unassigned headroom (458 Cn codepoints) that
future emoji are allocated into, so the table does not age.

The predicate is defined once in sanitize and imported by output. A second copy
is how the input side stops flagging while the output side keeps blocking; the
cross-surface test asserts the two agree on six inputs.

Two residuals, both in LIMITATIONS (33 -> 34, README bumped): a ZWJ between two
emoji is now exempt and could carry a covert channel (one emoji per bit, cannot
split a word); and U+200C (ZWNJ) still has no context test, so Persian, Arabic
and Devanagari documents -- where it is orthographically required -- stay
blocked. That needs a script-based criterion and no corpus is here to verify it
against, so it is parked as a known FP class rather than guessed at.

736 green (was 727), coverage 128/128, 6/6 documented gaps hold.

README.md

@@ -246,7 +246,7 @@ a green scan means safe content. The highest-impact items:
   two of the three corpora are living, so the cells are not rewritten in place.
   Method and before/after: [`docs/rawhtml-census.py`](docs/rawhtml-census.py).
 
-**Full list — 33 items, each with the mechanism, plus the out-of-scope boundary:**
+**Full list — 34 items, each with the mechanism, plus the out-of-scope boundary:**
 [`docs/LIMITATIONS.md`](docs/LIMITATIONS.md). Several carry field measurements from
 consumer corpora, including the false positives the URL-shape rule actually produces.
 

docs/LIMITATIONS.md

@@ -480,6 +480,24 @@ fails the test, forcing this doc to be updated:
   consumer-notification promise in `docs/PLAN-v1.md`. Deferred deliberately, not
   overlooked.
 
+- **A ZWJ hidden between two emoji is exempt, and ZWNJ's own false-positive
+  class is untouched.** U+200D composes emoji (👩‍💻 is WOMAN + ZWJ + PERSONAL
+  COMPUTER), so testing it on codepoint membership alone flagged *and stripped*
+  the joiner in any document containing a ZWJ-composed emoji — an any-tier
+  `FAIL_SECURE` carrier, plus silent decomposition of the emoji, on ordinary
+  first-party content. The joiner is now judged by context instead: exempt only
+  when **both** neighbours are emoji-context codepoints, measured to cover
+  122/122 of the codepoints adjacent to a ZWJ across Unicode 17.0's 1614 RGI
+  sequences. Two residuals follow. First, a joiner placed *between two emoji*
+  is now invisible to the carrier check and could carry a covert channel — it
+  costs an emoji per bit and cannot split a word, which is the shape the
+  word-splitting attack needs, so the narrowing is deliberate rather than
+  complete. Second, **U+200C (ZWNJ) still has no context test**, and it is
+  orthographically *required* in Persian, Arabic and Devanagari — those
+  documents remain hard-blocked. That is a separate criterion (script-based,
+  not pictographic) and no corpus is available here to verify it against, so it
+  is parked as a known false-positive class rather than guessed at.
+
 ## Out-of-scope (documented boundary)
 
 Embedding/vector-layer defenses (OWASP LLM08, downstream of persist); multimodal

src/llm_ingestion_guard/output.py

@@ -71,6 +71,7 @@ from .calibration import MAX_CONNSTR_VALUE
 from .entropy import scan_entropy
 from .lexicon import MAX_SCAN_CHARS, scan_lexicon
 from .report import Finding, Report, Severity, Source
+from .sanitize import _is_joiner_in_emoji_sequence
 
 # --- secret / credential egress patterns (OWASP LLM02) ----------------------
 # Ported from knowledge/secrets-patterns.md. ``value_group`` names the capturing
@@ -253,7 +254,14 @@ _BIDI_CPS = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0
 def _scan_invisible_carriers(text: str, source: Source) -> Report:
     """Flag invisible zero-width / BIDI carriers present in ``text`` (report-only)."""
     report = Report()
-    zero_width = sum(1 for ch in text if ord(ch) in _ZERO_WIDTH_CPS)
+    # The ZWJ exemption is imported from `sanitize`, never re-stated here: the
+    # two surfaces are one decision, and a second copy of the rule is how the
+    # input side stops flagging while the output side keeps hard-blocking.
+    zero_width = sum(
+        1 for i, ch in enumerate(text)
+        if ord(ch) in _ZERO_WIDTH_CPS
+        and not (ord(ch) == 0x200D and _is_joiner_in_emoji_sequence(text, i))
+    )
     bidi = sum(1 for ch in text if ord(ch) in _BIDI_CPS)
     if zero_width:
         report.add(Finding(

src/llm_ingestion_guard/sanitize.py

@@ -24,6 +24,57 @@ _ZERO_WIDTH = frozenset({0x200B, 0x200C, 0x200D, 0xFEFF, 0x00AD})
 _BIDI = frozenset({0x202A, 0x202B, 0x202C, 0x202D, 0x202E, 0x2066, 0x2067, 0x2068, 0x2069})
 _TAG_LO, _TAG_HI = 0xE0000, 0xE007F  # Unicode Tags block (U+E0000–U+E007F)
 
+# ZWJ (U+200D) is the one zero-width codepoint with a legitimate, extremely
+# common use: it composes emoji. 👩‍💻 is WOMAN + ZWJ + PERSONAL COMPUTER, and
+# families, skin-tone professions and flag variants are all built the same way.
+# Testing membership alone therefore flagged — and *stripped* — the joiner in any
+# first-party document containing such an emoji, which `disposition` grades as an
+# any-tier carrier (FAIL_SECURE, no appeal). The strip was the worse half: it
+# silently decomposed 👩‍💻 into two unrelated emoji, so `sanitize` corrupted
+# content it was only ever supposed to remove carriers from.
+#
+# The fix is the one our own lexicon row `unicode:zero-width-in-word`
+# (`\w[ZW]\w`) already used: judge the joiner by CONTEXT, not identity. A ZWJ is
+# exempt only when BOTH neighbours are emoji-context codepoints — half-context is
+# not context, so `a<ZWJ>👩` stays a carrier and an attacker cannot buy exemption
+# with a single emoji.
+#
+# The ranges are blocks, not an emoji table. Measured against Unicode 17.0's
+# `emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
+# adjacent to a ZWJ, and these five ranges cover 122/122. Blocks are the point —
+# shipping the RGI sequence list itself would be precise the day it landed and
+# stale at the next Unicode release, reintroducing this exact false positive for
+# every new emoji until someone bumped the file. Whole blocks include the
+# unassigned headroom (458 Cn codepoints here) that future emoji are allocated
+# into, so the table does not age.
+#
+# Residual, documented in LIMITATIONS: a ZWJ hidden *between two emoji* is
+# exempt and could carry a covert channel. It costs an emoji per bit and cannot
+# split a word, which is the shape the word-splitting attack actually needs.
+_EMOJI_CTX_RANGES = (
+    (0x2190, 0x21FF),    # Arrows — ↔ ↕ (U+2194/2195)
+    (0x2600, 0x27BF),    # Misc Symbols + Dingbats — ❤ ☠ ⚕ ⚧ ✈ ❄ ♀ ♂ ➡
+    (0x2B00, 0x2BFF),    # Misc Symbols & Arrows — ⬛ (U+2B1B)
+    (0xFE0F, 0xFE0F),    # VARIATION SELECTOR-16, the emoji presentation selector
+    (0x1F000, 0x1FAFF),  # Emoji planes, incl. skin-tone modifiers U+1F3FB–FF
+)
+
+
+def _is_emoji_context(cp: int) -> bool:
+    """True when ``cp`` may legitimately sit adjacent to an emoji-composing ZWJ."""
+    return any(lo <= cp <= hi for lo, hi in _EMOJI_CTX_RANGES)
+
+
+def _is_joiner_in_emoji_sequence(text: str, i: int) -> bool:
+    """True when ``text[i]`` is a ZWJ composing an emoji rather than a carrier.
+
+    Requires an emoji-context codepoint on BOTH sides. A joiner at either edge of
+    the document has no neighbour and so is never exempt.
+    """
+    if i == 0 or i + 1 >= len(text):
+        return False
+    return _is_emoji_context(ord(text[i - 1])) and _is_emoji_context(ord(text[i + 1]))
+
 # Span carriers.
 #
 # ReDoS note (OWASP LLM10). The comment stripper used to be `<!--.*?-->` with a
@@ -118,9 +169,11 @@ def sanitize(
     bidi = 0
     tag_cps: list[int] = []
     kept: list[str] = []
-    for ch in text:
+    for i, ch in enumerate(text):
         cp = ord(ch)
-        if cp in _ZERO_WIDTH:
+        if cp == 0x200D and _is_joiner_in_emoji_sequence(text, i):
+            kept.append(ch)  # composing an emoji, not carrying a payload
+        elif cp in _ZERO_WIDTH:
             zero_width += 1
         elif cp in _BIDI:
             bidi += 1

tests/test_output.py

@@ -462,3 +462,39 @@ def test_gate_is_bounded_on_the_long_attribute_arm():
     start = time.monotonic()
     scan_output(payload)
     assert time.monotonic() - start < 2.0
+
+
+# --- ZWJ inside emoji sequences on the output gate ---------------------------
+#
+# Same defect class as `sanitize`, second surface: `_ZERO_WIDTH_CPS` tested
+# U+200D on membership alone, so a model that legitimately reproduced a
+# ZWJ-composed emoji into an artifact raised `output:zero-width-present` — an
+# any-tier FAIL_SECURE carrier. Both surfaces must apply the same context test,
+# or the input side stops flagging and the output side keeps blocking.
+
+def test_zwj_inside_emoji_sequence_is_not_flagged_on_the_output_gate():
+    for emoji in ("\U0001F469‍\U0001F4BB",
+                  "\U0001F468‍\U0001F469‍\U0001F467",
+                  "\U0001F469\U0001F3FD‍\U0001F4BB",
+                  "❤️‍\U0001F525"):
+        labels = {f.label for f in scan_output(f"Shipped {emoji} today.").findings}
+        assert "output:zero-width-present" not in labels, emoji
+
+
+def test_freestanding_zwj_is_still_flagged_on_the_output_gate():
+    labels = {f.label for f in scan_output("impor‍tant instruction").findings}
+    assert "output:zero-width-present" in labels
+
+
+def test_output_zwj_narrowing_matches_the_sanitize_side():
+    # The two surfaces must agree: anything sanitize strips, the output gate
+    # flags. A split here is how a carrier reaches a persisted artifact after
+    # passing the input side.
+    from llm_ingestion_guard.sanitize import sanitize
+    for text in ("a‍\U0001F469", "\U0001F469‍a", "‍\U0001F469", "\U0001F469‍",
+                 "\U0001F469‍\U0001F4BB", "impor‍tant"):
+        stripped = "sanitize:zero-width" in {
+            f.label for f in sanitize(text).report.findings}
+        flagged = "output:zero-width-present" in {
+            f.label for f in scan_output(text).findings}
+        assert stripped == flagged, f"{text!r}: sanitize={stripped} output={flagged}"

tests/test_sanitize.py

@@ -116,3 +116,77 @@ def test_comment_stripping_survives_the_redos_fix():
     assert sanitize("a <!-- x --> b <!-- y --> c").text == "a  b  c"
     assert sanitize("a <!-- never closed").text == "a <!-- never closed"
     assert sanitize("a --> b").text == "a --> b"
+
+
+# --- ZWJ inside emoji sequences is not a carrier ------------------------------
+#
+# `_ZERO_WIDTH` used to test U+200D on codepoint membership alone, so every
+# first-party document containing a ZWJ-composed emoji (professions, families,
+# skin tones) raised `sanitize:zero-width` — an any-tier FAIL_SECURE carrier in
+# `disposition._CARRIER_LABELS`, i.e. hard-blocked with no appeal. Worse, the
+# stripper also *removed* the joiner, silently decomposing 👩‍💻 into two
+# unrelated emoji: a false positive AND content corruption on the same char.
+#
+# The fix mirrors what our own lexicon row `unicode:zero-width-in-word`
+# (`\w[ZW]\w`) already did: judge the ZWJ by its CONTEXT, not its identity. A
+# joiner is exempt only when BOTH neighbours are emoji-context codepoints.
+
+_EMOJI_ZWJ_CASES = [
+    ("woman technologist", "\U0001F469‍\U0001F4BB"),
+    ("family",             "\U0001F468‍\U0001F469‍\U0001F467"),
+    ("skin tone + job",    "\U0001F469\U0001F3FD‍\U0001F4BB"),
+    ("heart on fire",      "❤️‍\U0001F525"),  # VS16 before the joiner
+    ("rainbow flag",       "\U0001F3F3️‍\U0001F308"),
+]
+
+
+def test_zwj_inside_emoji_sequence_is_neither_flagged_nor_stripped():
+    for name, emoji in _EMOJI_ZWJ_CASES:
+        text = f"Release notes: {emoji} shipped."
+        result = sanitize(text)
+        labels = {f.label for f in result.report.findings}
+        assert "sanitize:zero-width" not in labels, f"{name}: false positive"
+        assert result.text == text, f"{name}: joiner stripped, emoji decomposed"
+
+
+def test_emoji_only_document_stays_byte_identical():
+    # The byte-identity invariant (§9) must survive the exemption: a document
+    # whose ONLY zero-width char is an in-emoji joiner has nothing to strip.
+    text = "\U0001F469‍\U0001F4BB"
+    result = sanitize(text)
+    assert result.text is text or result.text == text
+    assert result.report.findings == []
+
+
+def test_freestanding_zwj_is_still_a_carrier():
+    # The whole point of the narrowing: real invisible-text stego must survive
+    # it. A joiner splitting a word has no emoji on either side.
+    result = sanitize("impor‍tant instruction")
+    assert "sanitize:zero-width" in {f.label for f in result.report.findings}
+    assert "‍" not in result.text
+
+
+def test_zwj_with_only_one_emoji_neighbour_is_still_a_carrier():
+    # Half-context is not context. `a<ZWJ>👩` and `👩<ZWJ>a` are not RGI
+    # sequences, so an attacker cannot buy exemption with a single emoji.
+    for text in ("a‍\U0001F469", "\U0001F469‍a"):
+        result = sanitize(text)
+        assert "sanitize:zero-width" in {f.label for f in result.report.findings}, text
+        assert "‍" not in result.text
+
+
+def test_zwj_at_document_edge_is_still_a_carrier():
+    # A joiner with no neighbour at all cannot be inside a sequence.
+    for text in ("‍\U0001F469", "\U0001F469‍", "‍"):
+        result = sanitize(text)
+        assert "sanitize:zero-width" in {f.label for f in result.report.findings}, repr(text)
+
+
+def test_other_zero_width_classes_are_untouched_by_the_zwj_narrowing():
+    # Only U+200D got a context test. ZWSP/ZWNJ/BOM/soft-hyphen between emoji
+    # stay carriers — ZWNJ's own false-positive class (Persian/Devanagari
+    # orthography) is a separate, deliberately unaddressed decision.
+    for cp in (0x200B, 0x200C, 0xFEFF, 0x00AD):
+        text = f"\U0001F469{chr(cp)}\U0001F4BB"
+        result = sanitize(text)
+        assert "sanitize:zero-width" in {f.label for f in result.report.findings}, hex(cp)