docs(carriers): the third verdict exists, and publishing an alias is what takes it away
The carrier decision request went to both runtimes today. Then re-reading our own spec turned up an error in it: the request said the corpus has no verdict for a case a runtime cannot reach. It has one, and this repository wrote it — §1.1 `not-applicable`, attaching to a declared table. A correction went to both. What the correction found is sharper than the mistake. llm-security declares the lexicon table alone (`DECLARED_TABLES`, measured at 47905da), so carrier cases are not-applicable there today. Their suite derives the registered set by walking each vendored file for any node carrying `aliases.llm_security`, then asserts every registered table is declared — the §1.1 anti-narrowing floor. Granularity is the file. So one aliased carrier id would force `codepoints/carriers.json` into their declared set, oblige them to run all six carrier cases, and convert the three artifact-side ones into failures. Publishing the alias is the irreversible act, not minting the case. manifest.json 0.3.3 -> 0.3.4; README's carrier paragraph carried the same superseded "labels differ by pipeline stage" claim and now carries the measured one. No case, no id and no expected.json moved. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01F1HPHHP1zDNk1tMKDPuLCC
This commit is contained in:
parent
302625ead5
commit
7ce0ba706c
3 changed files with 66 additions and 9 deletions
55
CHANGELOG.md
55
CHANGELOG.md
|
|
@ -9,6 +9,61 @@ Versioning note: the repository tag versions **the contract** (file set, key nam
|
||||||
case ids, disposition semantics). Each JSON file additionally carries its own
|
case ids, disposition semantics). Each JSON file additionally carries its own
|
||||||
`"version"` field, bumped when that file changes.
|
`"version"` field, bumped when that file changes.
|
||||||
|
|
||||||
|
## [0.4.3] — 2026-08-11
|
||||||
|
|
||||||
|
**No fixture, id or `expected.json` moved — a runtime that passes `0.4.2` passes `0.4.3`
|
||||||
|
unchanged. What changed is a claim this repository made about a runtime it does not own, and the
|
||||||
|
claim was wrong on the day it was written.**
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- `conformance/manifest.json` `0.3.2` → `0.3.4` — the `scope_planned.blockers` entry for
|
||||||
|
`codepoints/carriers.json` described the guard as emitting two stage-coupled labels per carrier,
|
||||||
|
"the same split for bidi and unicode-tag". The artifact-side label for tags is
|
||||||
|
`lexicon:unicode-tags-present`, emitted from `lexicon.py`, and `output.py` carries a comment
|
||||||
|
saying it deliberately does not repeat it there. **Wrong when written, not stale:** checked at
|
||||||
|
`e671edb` — the commit the sibling secret-egress blocker was measured against — where
|
||||||
|
`coverage.py` already asserted that label, and re-measured at `a59184b`.
|
||||||
|
|
||||||
|
The correction moves the blocker rather than removing it. The guard's `Finding` carries a
|
||||||
|
`detector` field beside `label`, and the prefix is that field's value: `detector="lexicon"` on
|
||||||
|
`lexicon:unicode-tags-present`, `"output"` on `output:zero-width-present`. **The prefix names
|
||||||
|
the detector, not the pipeline stage** — and for tags a single detector serves both entry
|
||||||
|
points, which is why there is no sixth `output:` label to find. "A commons id would have to be
|
||||||
|
invented stage-neutral" was never the problem. Six labels exist to adopt verbatim, the way the
|
||||||
|
83 lexicon ids were adopted from this same runtime's port.
|
||||||
|
|
||||||
|
What blocks adoption is measured and named instead, at llm-security `47905da`: `sanitize:`
|
||||||
|
asserts a strip that runtime does not perform (`scanners/unicode-scanner.mjs` exports one entry
|
||||||
|
point, `scan(targetPath, discovery)`, reporting presence with `scanner: 'UNI'`, a prose title
|
||||||
|
and no id); three of the six name a persist gate it does not have, which the corpus already has
|
||||||
|
a verdict for — §1.1 `not-applicable`, attaching to a declared **table** — but which
|
||||||
|
**publishing the alias is what takes away**: that runtime's suite walks each vendored file for
|
||||||
|
any node carrying `aliases.llm_security` and asserts every registered table is declared, so one
|
||||||
|
aliased carrier id forces `codepoints/carriers.json` into a declared set of what is today the
|
||||||
|
lexicon alone, obliging it to run all six cases and converting the three artifact-side ones into
|
||||||
|
failures; and the entry point pinned for it in `measurement.runtimes` (`scanForInjection`) does not
|
||||||
|
reach carriers at all, so carrier cases need a per-scope entry point this manifest expresses
|
||||||
|
nowhere. Both runtimes already build their carrier sets from `codepoints/carriers.json`, so the
|
||||||
|
divergence is in what a finding is *called* and where it can be *observed*, never in which code
|
||||||
|
points are carriers.
|
||||||
|
|
||||||
|
### Asked, not decided
|
||||||
|
|
||||||
|
- The three objections went to both runtimes over coord on 2026-08-11 as a decision request, each
|
||||||
|
asked the question only it can answer. **Nothing was minted.** A case id is contract surface
|
||||||
|
consumers pin against, and publishing a single carrier alias is itself irreversible — it widens
|
||||||
|
another runtime's declared table set by force of that runtime's own test suite. Minting first
|
||||||
|
would have made a proposal into a fait accompli. The manifest records the request, so a later
|
||||||
|
reader can tell "asked, unanswered" from "nobody asked".
|
||||||
|
|
||||||
|
- **A correction followed the request the same day, on our own error.** The request asserted that
|
||||||
|
the corpus had no third verdict for a case a runtime cannot reach. It has one — §1.1
|
||||||
|
`not-applicable` — and this repository wrote that section. The question was put before its own
|
||||||
|
normative spec was re-read; the follow-up says so to both runtimes and restates the choice as
|
||||||
|
mint-input-side-only, accept three standing failures, or publish a guard-only id space with no
|
||||||
|
`llm_security` alias at all.
|
||||||
|
|
||||||
## [0.4.2] — 2026-08-11
|
## [0.4.2] — 2026-08-11
|
||||||
|
|
||||||
**No data file changed and no pattern moved.** A runtime that passes `0.4.1` passes `0.4.2`
|
**No data file changed and no pattern moved.** A runtime that passes `0.4.1` passes `0.4.2`
|
||||||
|
|
|
||||||
16
README.md
16
README.md
|
|
@ -24,7 +24,7 @@ As a `git subtree` (recommended: history is preserved and upgrades are a single
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git subtree add --prefix vendor/commons \
|
git subtree add --prefix vendor/commons \
|
||||||
https://git.fromaitochitta.com/open/llm-security-commons.git v0.4.2 --squash
|
https://git.fromaitochitta.com/open/llm-security-commons.git v0.4.3 --squash
|
||||||
|
|
||||||
# later, to move to a newer tag
|
# later, to move to a newer tag
|
||||||
git subtree pull --prefix vendor/commons \
|
git subtree pull --prefix vendor/commons \
|
||||||
|
|
@ -34,7 +34,7 @@ git subtree pull --prefix vendor/commons \
|
||||||
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
|
Or pin a tag and copy — `fork-and-own` is an explicitly supported path:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
git clone --depth 1 --branch v0.4.2 \
|
git clone --depth 1 --branch v0.4.3 \
|
||||||
https://git.fromaitochitta.com/open/llm-security-commons.git
|
https://git.fromaitochitta.com/open/llm-security-commons.git
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -61,14 +61,14 @@ and keeping it that small is the point.
|
||||||
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
|
| [`schema/conformance-declaration.schema.json`](schema/conformance-declaration.schema.json) | **Normative.** The shape a runtime publishes alongside a conformance result: which commons tables it implements, the commons commit it measured, and the four verdict counts. Required by the corpus spec §1.1; not validated by anything here, because nothing here runs. |
|
||||||
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
|
| [`spec/conformance-corpus.md`](spec/conformance-corpus.md) | **Normative.** How to read the corpus: what a case is, why `input.txt` is bytes rather than text, what `exact-within-scope` requires of a runtime, and how a runtime declares its table set so a case scoped outside it reads as `not-applicable` rather than as a failure. |
|
||||||
| [`conformance/`](conformance/) | 90 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
| [`conformance/`](conformance/) | 90 cases. One directory per case: `input.txt` in, `expected.json` out. Ground truth. 84 cover the injection lexicon — 83 one per pattern, both seeding runtimes measured producing the same verdict on all 83, plus one variant case gating a pattern form against its predecessor. Six cover active content and are measured against the one runtime that implements that table — `not-applicable` for the other, not failing. See [`conformance/manifest.json`](conformance/manifest.json). |
|
||||||
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.4.2.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
| `spec/decode-pipeline.md` | **Planned, still not shipped as of v0.4.3.** The decode order, in RFC 2119 language. Two runtimes that decode in different orders will disagree on identical input. Writing it needs the decode implementation, which is engine code and has not been supplied — and a normative spec guessed from a data dump would be worse than an absent one. |
|
||||||
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
| [`docs/extraction-plan.md`](docs/extraction-plan.md) | Informative: where each file was seeded from, and what v0.1.0 promised. |
|
||||||
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions. Most of it is still open, and the two rows that closed in v0.4.0 closed because the runtime that owns the value decided, not because this document found them wrong. |
|
| [`docs/lexicon-port-divergence.md`](docs/lexicon-port-divergence.md) | Informative: a measured disagreement between two ports of the injection lexicon — 13 patterns that behave differently, in both directions. Most of it is still open, and the two rows that closed in v0.4.0 closed because the runtime that owns the value decided, not because this document found them wrong. |
|
||||||
|
|
||||||
Every JSON file carries a top-level `version`. Every normative specification carries a
|
Every JSON file carries a top-level `version`. Every normative specification carries a
|
||||||
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
|
`Status: normative` marker. Rows marked **Planned** are named here because the layout is
|
||||||
part of the contract, but the file does not exist yet — they are not links, and nothing in
|
part of the contract, but the file does not exist yet — they are not links, and nothing in
|
||||||
v0.4.2 depends on them.
|
v0.4.3 depends on them.
|
||||||
|
|
||||||
Each data file records its own provenance and, in `verified`, how strongly it is backed.
|
Each data file records its own provenance and, in `verified`, how strongly it is backed.
|
||||||
`calibration/calibration.json` is currently the one file that says `false`: it was
|
`calibration/calibration.json` is currently the one file that says `false`: it was
|
||||||
|
|
@ -102,9 +102,11 @@ number would misreport both:
|
||||||
`not-applicable` says the corpus did not ask, never that the runtime is blind.
|
`not-applicable` says the corpus did not ask, never that the runtime is blind.
|
||||||
|
|
||||||
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
|
Four cases remain unshipped, for the carrier and secret-egress tables, and neither is
|
||||||
blocked on effort. Carriers has no adoptable id space: one runtime labels the same carrier
|
blocked on effort. Carriers has no *ratified* id space: six labels exist to adopt verbatim
|
||||||
differently depending on pipeline stage, which would make a case's verdict depend on the
|
from one runtime, but adopting them would hand the other an id asserting a strip it does not
|
||||||
entry point it was measured through. Secret egress is not an id question at all — the two
|
perform, and publishing a single alias would force that runtime's declared table set to widen
|
||||||
|
— turning three cases it cannot reach from `not-applicable` into failures. Put to both
|
||||||
|
runtimes as a decision request on 2026-08-11; unanswered. Secret egress is not an id question at all — the two
|
||||||
runtimes carry *different tables*, 19 entries against 25, cut at different granularities.
|
runtimes carry *different tables*, 19 entries against 25, cut at different granularities.
|
||||||
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
|
`conformance/manifest.json` records both blockers under `scope_planned.blockers`, measured,
|
||||||
so the gap is visible rather than inferred.
|
so the gap is visible rather than inferred.
|
||||||
|
|
|
||||||
File diff suppressed because one or more lines are too long
Loading…
Add table
Add a link
Reference in a new issue