1
0
Fork 0

release(0.6.1): the ZWJ fix gets a tag a consumer can pin

a59184b fixed a hard-block that reached every emoji-composed document, and
llm-ingestion-okf was told so -- while the fix sat untagged on main. A consumer
cannot pin what has no tag, so the notice was a promise the repo had not kept.

SURFACES MOVED (the eight the 0.5.0 sweep established, plus the ninth verified)

  pyproject.toml           0.6.0 -> 0.6.1
  __init__.py              0.6.0 -> 0.6.1
  CHANGELOG.md             [0.6.1] entry, [Unreleased] reset to "Nothing yet."
  README.md                badge, install tag @v0.6.1
  docs/ADOPTION-BRIEF.md   `v0.6.0` x2, and 727 -> 736 passing
  CLAUDE.md                one clause: ZWJ judged by context, not identity
  SECURITY.md              `0.6.x` -- still true at 0.6.1, verified, not moved
  docs/BRIEF.md            `v0.6 (alpha)` -- likewise
  README.md:36             `v0.6`, alpha -- likewise
  forge description        re-verified THIS release: 178 codepoints, no version
                           claim. Not inherited from 0.6.0's check.

PATCH, NOT MINOR -- and it was asked, not assumed. 6bcb898 made "loosens the
upload door" the criterion for minor, and this loosens it too, so the reading
was put to the operator with the count rather than settled quietly. 0.6.0 was a
policy choice (`<base>` left the active name set by decision); this restores a
contract the module already published, against a class never meant to be
blocked. Measured loosening: 1 document of 1126 across the three FP populations
(reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550), against
0.6.0's 25 + 2 + 2. Operator chose patch.

VERIFIED ON THE BUMPED TREE, NOT THE PRE-BUMP ONE

  736 passed; coverage 128/128 recall, 6/6 documented gaps hold
  `git grep '0\.6\.0'` returns provenance only -- LIMITATIONS history, the
    census script, source docstrings, tests, README's pre-0.6.0 numbers
  docs/LIMITATIONS.md: 34 items, README says 34
  __version__ and pyproject agree at 0.6.1
  no tracked file carries a stale current-state version or test count

The sweep tool itself was wrong first: `git grep -E '\b727\b'` returns nothing,
because POSIX ERE has no `\b`. An empty result read as "clean" when the claim
was there. Every sweep above is plain `git grep`.

Still to prove before this is announced: a scratch-venv install at the tag with
the resolved version asserted -- the README line above is only true once v0.6.1
resolves.
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 22:30:32 +02:00
commit 0903785187
6 changed files with 72 additions and 7 deletions

View file

@ -10,6 +10,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
Nothing yet.
## [0.6.1] — 2026-08-11
### Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked
`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
`disposition._CARRIER_LABELS` grades a carrier as any-tier `fail_secure` with no
appeal, so **any** first-party document containing a ZWJ-composed emoji —
professions, families, skin tones, flag variants — was blocked permanently, with
no preset able to release it. Reported by `ms-ai-architect`, confirmed here
against the code.
The worse half was not in the report: sanitize *removed* the joiner, silently
decomposing one emoji into two unrelated ones. A module whose published contract
is "only ever removes carriers" was corrupting content.
The fix is the shape our own lexicon row `unicode:zero-width-in-word`
(`\w[ZW]\w`) already used — **judge the joiner by context, not identity**. A ZWJ
is exempt only when *both* neighbours are emoji-context codepoints. Half-context
is not context, so `a<ZWJ>😀` stays a carrier and an attacker cannot buy
exemption with a single trailing emoji.
**Blocks, not an emoji table.** Measured against Unicode 17.0's
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
Shipping the RGI list itself would be exact the day it landed and stale at the
next Unicode release, reopening this false positive for every new emoji; whole
blocks carry the unassigned headroom (458 `Cn` codepoints) that future emoji are
allocated into, so the table does not age. The predicate is defined once in
`sanitize` and imported by `output`; a cross-surface test asserts the two halves
agree on six inputs.
### Known behaviour change
**Documents whose only finding was an emoji-context ZWJ now persist unattended.**
On `PRESET_USER_UPLOAD` they move from `fail_secure` to `WARN`. Measured across
the three false-positive populations at one corpus state: **1 document of 1126**
(reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
loosening of the *upload door*, not of detection — recall is unchanged at 128/128
demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
including between a word character and an emoji, is graded exactly as before.
**Patch, not minor.** 0.6.0 called itself minor for loosening the same door, but
that was a policy choice — `<base>` left the active name set by decision. This
one restores a contract the module already published, against a class that was
never meant to be blocked. A fix whose observable effect is the point of the fix
is what the patch level is for.
### Residuals — both documented (`docs/LIMITATIONS.md`, 33 → 34 items)
- **A ZWJ between two emoji is now exempt**, so it can carry a narrowband covert
channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
stated rather than hidden.
- **U+200C (ZWNJ) still has no context test.** It is orthographically *required*
in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
criterion has to be script-based rather than pictographic, and no corpus is
here to verify one against — parked as a known false-positive class rather than
guessed at.
736 tests pass (was 727).
## [0.6.0] — 2026-08-11
### Changed — `active:raw-html` stops firing on two things that carry no affordance

View file

@ -22,6 +22,8 @@ og secret-egress (LLM02), så en base64-innpakket credential fanges som
restgap (entropy eksponerer kun base64-klartekst). `active:raw-html` krever nå et
EKSTERNT mål på URL-attributt-grenen, og `<base>` er ute av det aktive navnesettet;
scanner og mutator har hver sin predikat (`is_active_tag` / `is_defangable_tag`).
ZWJ (U+200D) dømmes på KONTEKST, ikke identitet — unntas kun mellom to emoji, på
begge flater (`sanitize` eier predikatet, `output` importerer det).
Start med `docs/BRIEF.md` for design, `README.md` for bruk, `docs/PLAN.md` for
byggerekkefølgen.

View file

@ -2,7 +2,7 @@
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
![Version](https://img.shields.io/badge/version-0.6.0-blue)
![Version](https://img.shields.io/badge/version-0.6.1-blue)
![Status](https://img.shields.io/badge/status-alpha-orange)
![Python](https://img.shields.io/badge/python-3.10%2B-purple)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
@ -43,7 +43,7 @@ may still change. There are real limitations, stated plainly below; read them.
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
```bash
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.6.0"
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.6.1"
```
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or

View file

@ -4,7 +4,7 @@
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
to decide **when** and **where** to add a write-time ingestion guard.
**Status of the guard:** `v0.6.0` (alpha). Stdlib-only core, framework-agnostic.
**Status of the guard:** `v0.6.1` (alpha). Stdlib-only core, framework-agnostic.
Public API may still change. Read the known-limitations section before you rely
on it.
@ -140,9 +140,9 @@ live payload:
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
```
As of `v0.6.0`: **128 / 128 defended classes demonstrated (recall 100%)** and **6 /
As of `v0.6.1`: **128 / 128 defended classes demonstrated (recall 100%)** and **6 /
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
update). The matrix is the single source of truth for the test suite (**727
update). The matrix is the single source of truth for the test suite (**736
passing**), which also asserts total recall, that every lexicon pattern has a
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,

View file

@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "llm-ingestion-guard"
version = "0.6.0"
version = "0.6.1"
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
readme = "README.md"
requires-python = ">=3.10"

View file

@ -63,7 +63,7 @@ from .grounding import (
)
from . import okf
__version__ = "0.6.0"
__version__ = "0.6.1"
# --- §6 bookends: the two library-side halves around the transform ---------