open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Releases 4 Activity
4 releases 15 tags
  • v0.6.1 0903785187

    v0.6.1 Stable

    ktg released this 2026-08-11 20:30:51 +00:00 | 30 commits to main since this release

    Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked

    _ZERO_WIDTH (sanitize, input) and _ZERO_WIDTH_CPS (output,
    _scan_invisible_carriers) tested U+200D on codepoint membership alone.
    disposition._CARRIER_LABELS grades a carrier as any-tier fail_secure with no
    appeal, so any first-party document containing a ZWJ-composed emoji —
    professions, families, skin tones, flag variants — was blocked permanently, with
    no preset able to release it. Reported by ms-ai-architect, confirmed here
    against the code.

    The worse half was not in the report: sanitize removed the joiner, silently
    decomposing one emoji into two unrelated ones. A module whose published contract
    is "only ever removes carriers" was corrupting content.

    The fix is the shape our own lexicon row unicode:zero-width-in-word
    (\w[ZW]\w) already used — judge the joiner by context, not identity. A ZWJ
    is exempt only when both neighbours are emoji-context codepoints. Half-context
    is not context, so a<ZWJ>😀 stays a carrier and an attacker cannot buy
    exemption with a single trailing emoji.

    Blocks, not an emoji table. Measured against Unicode 17.0's
    emoji-zwj-sequences.txt: 1614 RGI sequences use 122 distinct codepoints
    adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
    its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
    Shipping the RGI list itself would be exact the day it landed and stale at the
    next Unicode release, reopening this false positive for every new emoji; whole
    blocks carry the unassigned headroom (458 Cn codepoints) that future emoji are
    allocated into, so the table does not age. The predicate is defined once in
    sanitize and imported by output; a cross-surface test asserts the two halves
    agree on six inputs.

    Known behaviour change

    Documents whose only finding was an emoji-context ZWJ now persist unattended.
    On PRESET_USER_UPLOAD they move from fail_secure to WARN. Measured across
    the three false-positive populations at one corpus state: 1 document of 1126
    (reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
    loosening of the upload door, not of detection — recall is unchanged at 128/128
    demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
    including between a word character and an emoji, is graded exactly as before.

    Patch, not minor. 0.6.0 called itself minor for loosening the same door, but
    that was a policy choice — <base> left the active name set by decision. This
    one restores a contract the module already published, against a class that was
    never meant to be blocked. A fix whose observable effect is the point of the fix
    is what the patch level is for.

    Residuals — both documented (docs/LIMITATIONS.md, 33 → 34 items)

    • A ZWJ between two emoji is now exempt, so it can carry a narrowband covert
      channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
      stated rather than hidden.
    • U+200C (ZWNJ) still has no context test. It is orthographically required
      in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
      criterion has to be script-based rather than pictographic, and no corpus is
      here to verify one against — parked as a known false-positive class rather than
      guessed at.

    736 tests pass (was 727).

    Downloads