fix(conformance): the tag carrier has no output: label, and our blocker claimed it did

The carriers blocker described the guard as emitting two stage-coupled labels
per carrier, "the same split for bidi and unicode-tag". Measured at guard
`a59184b`: the artifact-side label for tags is `lexicon:unicode-tags-present`,
emitted from lexicon.py, and `output.py` says in a comment that it deliberately
does not repeat it. Checked at `e671edb` too — coverage.py asserted that label
there as well, so the sentence was wrong when written, not stale.

The correction moves the blocker rather than shrinking it. The guard's `Finding`
carries a `detector` field, and the label prefix is that field's value, so the
prefix names the detector and there was never a stage to be neutral about. Six
labels exist to adopt verbatim. What blocks adoption is measured and named
instead: `sanitize:` asserts a strip that llm-security does not perform, three
of the six name a persist gate it does not have, and the entry point pinned for
it in `measurement.runtimes` does not reach carriers at all.

manifest.json 0.3.2 -> 0.3.3. No case, no id and no expected.json moved.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01F1HPHHP1zDNk1tMKDPuLCC
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 22:25:27 +02:00
commit 302625ead5

View file

@ -1,5 +1,5 @@
{
"version": "0.3.2",
"version": "0.3.3",
"id": "llm-security-commons/conformance",
"description": "Enumeration and measurement header for the conformance corpus. Every case directory holds input.txt (the exact bytes to scan) and expected.json (the findings a conforming runtime must produce). The normative reading of those files is spec/conformance-corpus.md; this file records where the cases came from and what was measured.",
"$comment": "Fixture files carry no individual version field. The corpus is versioned as a whole, here — a case is added, removed or corrected by bumping this version, and a case-id change is a MAJOR bump because consumers name cases.",
@ -28,7 +28,7 @@
"codepoints/carriers.json": 3,
"signatures/secret-egress.json": 1,
"blockers": {
"codepoints/carriers.json": "No adoptable id space, and a second problem underneath it. The guard emits TWO stage-coupled labels for the same carrier depending on pipeline position — `sanitize:zero-width` (input) versus `output:zero-width-present` (output), and the same split for bidi and unicode-tag; llm-security emits prose titles (unicode-scanner.mjs:191,236). A commons id would therefore have to be invented stage-neutral, which no other id space here required. And because `exact-within-scope` compares a finding SET, a commons id aliasing both guard labels would make the verdict depend on which entry point the runtime was measured through — an entry-point dependence the lexicon cases do not have, since this manifest pins entry point as a measurement fact rather than as contract.",
"codepoints/carriers.json": "No adoptable id space, and until 0.3.3 the shape of the problem was mis-recorded here. CORRECTED IN 0.3.3: through 0.3.2 this text opened 'The guard emits TWO stage-coupled labels for the same carrier depending on pipeline position — `sanitize:zero-width` (input) versus `output:zero-width-present` (output), and the same split for bidi and unicode-tag'. The split is NOT the same for unicode-tag, and it was not the same on the day that sentence was written: the artifact-side label for tags is `lexicon:unicode-tags-present`, emitted from `lexicon.py`, not an `output:`-prefixed one. Checked at guard commit `e671edb` — the commit this blocker's sibling was measured against — where `coverage.py` already asserted that label, and re-measured at `a59184b`. So the earlier text was wrong when written, not stale. `output.py` states the delegation in a comment: Unicode-tag / PUA stego 'is already surfaced by scan_lexicon (`lexicon:unicode-tags-present`), so it is not repeated'. The labels are therefore not stage-coupled at all. The guard's `Finding` carries a `detector` field alongside `label`, and the prefix is that field's value: `lexicon:unicode-tags-present` carries `detector=\"lexicon\"`, `output:zero-width-present` carries `detector=\"output\"`, `sanitize:*` is emitted by `sanitize.py`. THE PREFIX NAMES THE DETECTOR, not the pipeline stage — and for tags one detector serves both entry points, which is why there is no sixth `output:` label to find. Measured 2026-08-11 at `a59184b`. What remains open is smaller than 'invent a stage-neutral id', and none of it is a naming question. Six labels exist to adopt verbatim, the way the lexicon's 83 ids were adopted from this same runtime: `sanitize:zero-width` / `output:zero-width-present`, `sanitize:bidi-override` / `output:bidi-present`, `sanitize:unicode-tag` / `lexicon:unicode-tags-present`. Three objections stand against adopting them, all measured at llm-security `47905da`: (a) `sanitize:` asserts an ACTION — the carrier was stripped — and llm-security strips nothing. `scanners/unicode-scanner.mjs` exports a single entry point, `scan(targetPath, discovery)`, which reports presence with `scanner: 'UNI'` and a prose `title` and no id at all. Adopting verbatim would hand one runtime an id whose name asserts something it does not do. (b) Three of the six name a persist gate llm-security does not have, so a case expecting one is UNMEASURABLE there rather than failed — a third status `exact-within-scope` has no room for, and one the lexicon's `aliases` block never needed because every lexicon id had a name on both sides. (c) The entry point pinned for llm-security in `measurement.runtimes` (`scanForInjection`, the injection scanner) does not reach carriers at all; carrier findings there come from the path-based unicode scanner instead. A carrier case therefore needs a per-scope entry point per runtime, which this manifest expresses nowhere — it pins one entry point per runtime for the whole corpus. Note that both runtimes already consume this file's tables: llm-security builds its zero-width, tag-range and BIDI sets from `codepoints/carriers.json` via `scanners/lib/codepoints.mjs`. The divergence is in what the finding is CALLED and where it can be observed, never in which code points are carriers.",
"signatures/secret-egress.json": "Not an id-naming question at all. The two runtimes carry DIFFERENT TABLES, not two namings of one: this file holds 19 entries from llm-security, the guard's `_SECRET_PATTERNS` (output.py) holds 25 at different cut points — this file's single `GitHub Token` is four ids there, `Private Key PEM Block` is three, `Database connection string` is four. `aws-access-key-id` is the one clean one-to-one, which is why exactly one egress case was ever offered. A shared id space presupposes a table reconciliation that has not happened. Membership diverges both ways, and `Slack/Discord Webhook URL` (order 13) and `Azure AI Services Key` (order 4) are here and absent from the guard. CORRECTED IN 0.3.1: through 0.3.0 this text ended '(the guard has `gcp-service-account-json` and `openai-api-key-legacy`, which are absent here; ...)'. Two entries were named as one kind of fact, and they are two different kinds. Measured here on 2026-08-11 by running this file's own patterns, in `order`, over a service-account document, against the guard at commit `e671edb` — 18 patterns then, 19 since `secret-egress.json` 0.3.0. (1) `gcp-service-account-json` is NOT a coverage hole here. A complete service-account key file matches this table at order 11, `Private Key PEM Block`: that pattern's prefix group `(?:RSA |EC |DSA |OPENSSH )?` is optional, so the bare PKCS#8 header `-----BEGIN PRIVATE KEY-----` such a file carries is matched. What differs is the CUT POINT: the same document with its `private_key` field removed matches nothing here, while the guard's `\"type\"\\s*:\\s*\"service_account\"` still fires — the guard detects the document MARKER, this table detects the KEY MATERIAL. Both fire on a real key file; only the guard fires on a stripped one. That is exactly the cut-point divergence this blocker is about, and it is not a missing entry. (2) `openai-api-key-legacy` WAS a real hole here and is CLOSED IN 0.3.2. Through `secret-egress.json` 0.2.0 no pattern in this file matched a legacy `sk-…T3BlbkFJ…` key, measured the same way, and llm-security's 18 -> 19 report stood recorded as their unreproduced claim because the commit carrying it was not yet on their public remote. It is now: `refs/heads/main` reads `47905da`, `088e458` is an ancestor of it (checked with `git merge-base --is-ancestor`, not read off their log), and `secret-egress.json` 0.3.0 carries the entry re-extracted from the module text at that commit, with all 19 positions compared for name, source, flags and order. So the two tables now agree on this one shape. What has NOT changed is the reason this blocker exists: 19 against 25 at different cut points is still a table reconciliation nobody has performed, and one closed hole is not that reconciliation."
}
},