release(0.6.1): the ZWJ fix gets a tag a consumer can pin
a59184bfixed a hard-block that reached every emoji-composed document, and llm-ingestion-okf was told so -- while the fix sat untagged on main. A consumer cannot pin what has no tag, so the notice was a promise the repo had not kept. SURFACES MOVED (the eight the 0.5.0 sweep established, plus the ninth verified) pyproject.toml 0.6.0 -> 0.6.1 __init__.py 0.6.0 -> 0.6.1 CHANGELOG.md [0.6.1] entry, [Unreleased] reset to "Nothing yet." README.md badge, install tag @v0.6.1 docs/ADOPTION-BRIEF.md `v0.6.0` x2, and 727 -> 736 passing CLAUDE.md one clause: ZWJ judged by context, not identity SECURITY.md `0.6.x` -- still true at 0.6.1, verified, not moved docs/BRIEF.md `v0.6 (alpha)` -- likewise README.md:36 `v0.6`, alpha -- likewise forge description re-verified THIS release: 178 codepoints, no version claim. Not inherited from 0.6.0's check. PATCH, NOT MINOR -- and it was asked, not assumed.6bcb898made "loosens the upload door" the criterion for minor, and this loosens it too, so the reading was put to the operator with the count rather than settled quietly. 0.6.0 was a policy choice (`<base>` left the active name set by decision); this restores a contract the module already published, against a class never meant to be blocked. Measured loosening: 1 document of 1126 across the three FP populations (reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550), against 0.6.0's 25 + 2 + 2. Operator chose patch. VERIFIED ON THE BUMPED TREE, NOT THE PRE-BUMP ONE 736 passed; coverage 128/128 recall, 6/6 documented gaps hold `git grep '0\.6\.0'` returns provenance only -- LIMITATIONS history, the census script, source docstrings, tests, README's pre-0.6.0 numbers docs/LIMITATIONS.md: 34 items, README says 34 __version__ and pyproject agree at 0.6.1 no tracked file carries a stale current-state version or test count The sweep tool itself was wrong first: `git grep -E '\b727\b'` returns nothing, because POSIX ERE has no `\b`. An empty result read as "clean" when the claim was there. Every sweep above is plain `git grep`. Still to prove before this is announced: a scratch-venv install at the tag with the resolved version asserted -- the README line above is only true once v0.6.1 resolves.
This commit is contained in:
parent
a59184bb7f
commit
0903785187
6 changed files with 72 additions and 7 deletions
63
CHANGELOG.md
63
CHANGELOG.md
|
|
@ -10,6 +10,69 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
|||
Nothing yet.
|
||||
|
||||
|
||||
## [0.6.1] — 2026-08-11
|
||||
|
||||
### Fixed — the zero-width check tested identity, so emoji-composed documents were hard-blocked
|
||||
|
||||
`_ZERO_WIDTH` (sanitize, input) and `_ZERO_WIDTH_CPS` (output,
|
||||
`_scan_invisible_carriers`) tested U+200D on codepoint membership alone.
|
||||
`disposition._CARRIER_LABELS` grades a carrier as any-tier `fail_secure` with no
|
||||
appeal, so **any** first-party document containing a ZWJ-composed emoji —
|
||||
professions, families, skin tones, flag variants — was blocked permanently, with
|
||||
no preset able to release it. Reported by `ms-ai-architect`, confirmed here
|
||||
against the code.
|
||||
|
||||
The worse half was not in the report: sanitize *removed* the joiner, silently
|
||||
decomposing one emoji into two unrelated ones. A module whose published contract
|
||||
is "only ever removes carriers" was corrupting content.
|
||||
|
||||
The fix is the shape our own lexicon row `unicode:zero-width-in-word`
|
||||
(`\w[ZW]\w`) already used — **judge the joiner by context, not identity**. A ZWJ
|
||||
is exempt only when *both* neighbours are emoji-context codepoints. Half-context
|
||||
is not context, so `a<ZWJ>😀` stays a carrier and an attacker cannot buy
|
||||
exemption with a single trailing emoji.
|
||||
|
||||
**Blocks, not an emoji table.** Measured against Unicode 17.0's
|
||||
`emoji-zwj-sequences.txt`: 1614 RGI sequences use 122 distinct codepoints
|
||||
adjacent to a ZWJ, and five block ranges cover 122/122. The measurement earned
|
||||
its keep — the hand-reasoned candidate table missed U+2194, U+2195 and U+2B1B.
|
||||
Shipping the RGI list itself would be exact the day it landed and stale at the
|
||||
next Unicode release, reopening this false positive for every new emoji; whole
|
||||
blocks carry the unassigned headroom (458 `Cn` codepoints) that future emoji are
|
||||
allocated into, so the table does not age. The predicate is defined once in
|
||||
`sanitize` and imported by `output`; a cross-surface test asserts the two halves
|
||||
agree on six inputs.
|
||||
|
||||
### Known behaviour change
|
||||
|
||||
**Documents whose only finding was an emoji-context ZWJ now persist unattended.**
|
||||
On `PRESET_USER_UPLOAD` they move from `fail_secure` to `WARN`. Measured across
|
||||
the three false-positive populations at one corpus state: **1 document of 1126**
|
||||
(reference-corpus 1/389, vendor-harvest 0/187, generated-notes 0/550). This is a
|
||||
loosening of the *upload door*, not of detection — recall is unchanged at 128/128
|
||||
demonstrated classes with 6/6 documented gaps holding, and a ZWJ anywhere else,
|
||||
including between a word character and an emoji, is graded exactly as before.
|
||||
|
||||
**Patch, not minor.** 0.6.0 called itself minor for loosening the same door, but
|
||||
that was a policy choice — `<base>` left the active name set by decision. This
|
||||
one restores a contract the module already published, against a class that was
|
||||
never meant to be blocked. A fix whose observable effect is the point of the fix
|
||||
is what the patch level is for.
|
||||
|
||||
### Residuals — both documented (`docs/LIMITATIONS.md`, 33 → 34 items)
|
||||
|
||||
- **A ZWJ between two emoji is now exempt**, so it can carry a narrowband covert
|
||||
channel: one emoji per bit, and it cannot split a word. A deliberate narrowing,
|
||||
stated rather than hidden.
|
||||
- **U+200C (ZWNJ) still has no context test.** It is orthographically *required*
|
||||
in Persian, Arabic and Devanagari, so those documents stay hard-blocked. The
|
||||
criterion has to be script-based rather than pictographic, and no corpus is
|
||||
here to verify one against — parked as a known false-positive class rather than
|
||||
guessed at.
|
||||
|
||||
736 tests pass (was 727).
|
||||
|
||||
|
||||
## [0.6.0] — 2026-08-11
|
||||
|
||||
### Changed — `active:raw-html` stops firing on two things that carry no affordance
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ og secret-egress (LLM02), så en base64-innpakket credential fanges som
|
|||
restgap (entropy eksponerer kun base64-klartekst). `active:raw-html` krever nå et
|
||||
EKSTERNT mål på URL-attributt-grenen, og `<base>` er ute av det aktive navnesettet;
|
||||
scanner og mutator har hver sin predikat (`is_active_tag` / `is_defangable_tag`).
|
||||
ZWJ (U+200D) dømmes på KONTEKST, ikke identitet — unntas kun mellom to emoji, på
|
||||
begge flater (`sanitize` eier predikatet, `output` importerer det).
|
||||
Start med `docs/BRIEF.md` for design, `README.md` for bruk, `docs/PLAN.md` for
|
||||
byggerekkefølgen.
|
||||
|
||||
|
|
|
|||
|
|
@ -2,7 +2,7 @@
|
|||
|
||||
Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure.
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||

|
||||
|
|
@ -43,7 +43,7 @@ may still change. There are real limitations, stated plainly below; read them.
|
|||
Not on PyPI. The guard is distributed from its Forgejo origin — pin a release tag:
|
||||
|
||||
```bash
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.6.0"
|
||||
pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.6.1"
|
||||
```
|
||||
|
||||
The `open/` mirror is anonymously readable, so CI needs no deploy key, token, or
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@
|
|||
especially one converging on Google's Open Knowledge Format (OKF v0.1) — and needs
|
||||
to decide **when** and **where** to add a write-time ingestion guard.
|
||||
|
||||
**Status of the guard:** `v0.6.0` (alpha). Stdlib-only core, framework-agnostic.
|
||||
**Status of the guard:** `v0.6.1` (alpha). Stdlib-only core, framework-agnostic.
|
||||
Public API may still change. Read the known-limitations section before you rely
|
||||
on it.
|
||||
|
||||
|
|
@ -140,9 +140,9 @@ live payload:
|
|||
python -m llm_ingestion_guard.coverage # exit 0 = all as documented
|
||||
```
|
||||
|
||||
As of `v0.6.0`: **128 / 128 defended classes demonstrated (recall 100%)** and **6 /
|
||||
As of `v0.6.1`: **128 / 128 defended classes demonstrated (recall 100%)** and **6 /
|
||||
6 documented gaps still hold** (a *closed* gap fails the test, forcing a doc
|
||||
update). The matrix is the single source of truth for the test suite (**727
|
||||
update). The matrix is the single source of truth for the test suite (**736
|
||||
passing**), which also asserts total recall, that every lexicon pattern has a
|
||||
case (so the matrix cannot fall behind the lexicon), the full LLM02 secret-egress
|
||||
set, and the container-layer front-end (CSV formula-injection, zip-slip/bomb,
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ build-backend = "hatchling.build"
|
|||
|
||||
[project]
|
||||
name = "llm-ingestion-guard"
|
||||
version = "0.6.0"
|
||||
version = "0.6.1"
|
||||
description = "Write-time defensive layer for Python pipelines that persist LLM output: sanitize, fence, tool-less quarantined transform, capability isolation, scan before persist, fail-secure."
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.10"
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ from .grounding import (
|
|||
)
|
||||
from . import okf
|
||||
|
||||
__version__ = "0.6.0"
|
||||
__version__ = "0.6.1"
|
||||
|
||||
|
||||
# --- §6 bookends: the two library-side halves around the transform ---------
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue