feat(lexicon): set hybrid severity to high, record the id space as ratified
Two things the file was waiting on arrived, and both are now data rather than open questions. families[hybrid].severity was null by design - the seed dump supplied the array but not the severity. It is "high". The value is not stored anywhere as a field: the engine assigns it by pushing HYBRID_PATTERNS matches straight into the high bucket at injection-patterns.mjs:274-281, under the comment "Hybrid patterns are HIGH severity (v5.0 S6)". severity_provenance names the file it is NOT in. Both this repository and the Python guard first looked in severity.mjs, which contains no injection- family severity at all - grep it for any of the four pattern constants and you get nothing. The guard's port carries the right value while citing that file, so a consumer following its citation finds no answer. A wrong citation to a right value survives review in a way a wrong value does not, so it is recorded rather than quietly fixed. pattern_id_space.not_yet_confirmed is replaced by a ratification block. Both seeding runtimes agreed on 2026-08-09: llm-security ratified the 0.2.0 proposal as-is and treats an id change as breaking on the same terms, and the guard confirmed the space its own port supplied. The id is a cross-runtime contract now, which is what conformance/ needed before a fixture could name a finding at all. alias_evidence.llm_security is sharpened, not upgraded. All 83 alias strings were confirmed equal to the module's label field in array order, so the alias is certainly the pattern's name in the table. It is still not established that a FINDING carries it: output.mjs:finding() emits title and has no label key. Verified at table level, one level short of where it would matter. No detection data moved: all 83 patterns still byte-identical to the module at b0de0ca, and no pattern entry differs from HEAD. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SNMcqrfNyoLRQ7qXUFZnb9
This commit is contained in:
parent
344bc52adf
commit
be84121604
2 changed files with 46 additions and 7 deletions
24
CHANGELOG.md
24
CHANGELOG.md
|
|
@ -53,6 +53,30 @@ checks that were not run instead of attaching a caveat to a pass.
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|
||||||
|
- `lexicon/injection-lexicon.json` **0.4.0 → 0.5.0** — the last null in the file is filled and
|
||||||
|
the id space is ratified. Two blockers close, no detection data moves.
|
||||||
|
|
||||||
|
`families[hybrid].severity` was `null`, deliberately, because the seed dump did not supply
|
||||||
|
it. It is **`high`** — and the interesting part is where that is written. The hybrid family
|
||||||
|
has no severity field anywhere; the engine assigns one by pushing `HYBRID_PATTERNS` matches
|
||||||
|
straight into the `high` bucket at `injection-patterns.mjs:274-281`. Both this repository
|
||||||
|
and the Python guard had first looked in `severity.mjs`, which contains no injection-family
|
||||||
|
severity at all. The guard's port holds the right value behind that wrong citation, so
|
||||||
|
`severity_provenance.not_from` records the miss explicitly: a wrong citation to a right
|
||||||
|
value is the harder defect to catch later.
|
||||||
|
|
||||||
|
`pattern_id_space.not_yet_confirmed` is replaced by `ratification`. Both seeding runtimes
|
||||||
|
agreed on 2026-08-09 — `llm-security` ratified the 0.2.0 proposal as-is and treats an id
|
||||||
|
change as breaking on the same terms, and the guard confirmed the space its own port
|
||||||
|
supplied. `id` is now a cross-runtime contract, which is what `conformance/` was waiting
|
||||||
|
on to be able to name a finding.
|
||||||
|
|
||||||
|
`alias_evidence.llm_security` is sharpened rather than upgraded. All 83 alias strings were
|
||||||
|
confirmed equal to the module's `label` field, in order — so the alias is certainly the
|
||||||
|
pattern's name **in the table**. It is still not established that a finding carries it: the
|
||||||
|
producer is `output.mjs:finding()`, which emits `title` and has no `label` key at all.
|
||||||
|
Verified at table level, one level short of where it would matter. Match on `id`.
|
||||||
|
|
||||||
- `lexicon/injection-lexicon.json` **0.3.0 → 0.4.0** — verified against the source module
|
- `lexicon/injection-lexicon.json` **0.3.0 → 0.4.0** — verified against the source module
|
||||||
instead of against the dump it was transcribed from, and **two false provenance claims
|
instead of against the dump it was transcribed from, and **two false provenance claims
|
||||||
retracted**. The source is now pinned: `b0de0ca` on the public remote, imported in Node
|
retracted**. The source is now pinned: `b0de0ca` on the public remote, imported in Node
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
{
|
{
|
||||||
"version": "0.4.0",
|
"version": "0.5.0",
|
||||||
"id": "injection-lexicon",
|
"id": "injection-lexicon",
|
||||||
"description": "Prompt-injection pattern lexicon: the four pattern families a detector matches against untrusted text, and the severity family each belongs to. Data only. The variant building that feeds these patterns (normalisation, homoglyph folding, rot13, unicode-tag escalation) and the buried-payload heuristic are engine behaviour and are deliberately NOT described here.",
|
"description": "Prompt-injection pattern lexicon: the four pattern families a detector matches against untrusted text, and the severity family each belongs to. Data only. The variant building that feeds these patterns (normalisation, homoglyph folding, rot13, unicode-tag escalation) and the buried-payload heuristic are engine behaviour and are deliberately NOT described here.",
|
||||||
"owasp": "LLM01",
|
"owasp": "LLM01",
|
||||||
|
|
@ -11,15 +11,23 @@
|
||||||
"matching_method": "Each commons pattern was matched to a guard pattern by comparing this file's `label` to the guard's `desc`, with em-dash normalised to hyphen (the two differ only by that character, in the eight hitl-trap entries). 83/83 matched, one-to-one.",
|
"matching_method": "Each commons pattern was matched to a guard pattern by comparing this file's `label` to the guard's `desc`, with em-dash normalised to hyphen (the two differ only by that character, in the eight hitl-trap entries). 83/83 matched, one-to-one.",
|
||||||
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
|
"stability": "An id is a stable identifier. Changing one is a BREAKING change for any consumer matching on it, and requires a major bump of this file.",
|
||||||
"aliases": "Each pattern carries an `aliases` object naming what each seeding runtime calls it today. A runtime renames its own label by changing its alias here; the `id` does not move. The two aliases do NOT have the same evidentiary status — see alias_evidence.",
|
"aliases": "Each pattern carries an `aliases` object naming what each seeding runtime calls it today. A runtime renames its own label by changing its alias here; the `id` does not move. The two aliases do NOT have the same evidentiary status — see alias_evidence.",
|
||||||
"not_yet_confirmed": "Neither llm-security nor the guard has yet agreed to this id space — both were asked via coord on 2026-08-09. Until they answer, `id` is this repository's proposal, not a ratified cross-runtime contract.",
|
"ratification": {
|
||||||
|
"status": "ratified by both seeding runtimes",
|
||||||
|
"date": "2026-08-09",
|
||||||
|
"llm_security": "Ratified as-is, including the 0.2.0 proposal in commit 7b70f5b, by coord reply on 2026-08-09. They treat an id change as breaking on the same terms this file does.",
|
||||||
|
"llm_ingestion_pipeline_security": "Ratified by coord reply on 2026-08-09. The id space was adopted verbatim from their port, so ratification confirms rather than adopts.",
|
||||||
|
"$comment": "This replaces a `not_yet_confirmed` field carried through version 0.4.0, which said neither runtime had agreed. Both have. `id` is now a ratified cross-runtime contract rather than this repository's proposal, and conformance fixtures may name findings by it."
|
||||||
|
},
|
||||||
"alias_evidence": {
|
"alias_evidence": {
|
||||||
"llm_ingestion_guard": {
|
"llm_ingestion_guard": {
|
||||||
"verified": true,
|
"verified": true,
|
||||||
"method": "The guard's coverage matrix asserts on this exact string: coverage.py builds one case per lexicon pattern passing `pattern.id` as the expected value, and its probe compares that against the set of `label` fields on the findings the guard actually returns. The id is therefore demonstrably what a guard finding carries."
|
"method": "The guard's coverage matrix asserts on this exact string: coverage.py builds one case per lexicon pattern passing `pattern.id` as the expected value, and its probe compares that against the set of `label` fields on the findings the guard actually returns. The id is therefore demonstrably what a guard finding carries."
|
||||||
},
|
},
|
||||||
"llm_security": {
|
"llm_security": {
|
||||||
"verified": false,
|
"verified": "at table level only",
|
||||||
"method": "Not established. This value is the `label` field of the pattern table, which is what the source module names the pattern — not necessarily what a llm-security FINDING carries. The finding producer was never supplied (see finding.schema.json), and the known Node finding shape uses `title`, not `label`, with the SARIF profile deriving rule ids by slugging that title. Treat this alias as the pattern's name in the source table, and confirm the finding-level string before matching on it."
|
"method": "Every one of the 83 alias strings was compared to the `label` field of the corresponding entry in injection-patterns.mjs at b0de0ca: 83/83 exact, in array order. So this alias is certainly the pattern's name in the source table.",
|
||||||
|
"still_not_verified": "That a llm-security FINDING carries this string. The finding producer is now known - scanners/lib/output.mjs:finding(), line 32 - and it emits `title`, not `label`; there is no `label` key in the finding shape at all. Nothing connects a pattern's table label to the title a scanner passes in. The SARIF profile derives rule ids by slugging the title, so it inherits the same gap. Match on `id`, not on this alias.",
|
||||||
|
"asymmetry": "This is why the two aliases are recorded separately rather than averaged: the guard's alias is verified at the level that matters (its coverage matrix asserts the finding carries that exact string), and this one is verified one level short of it."
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
|
@ -40,7 +48,7 @@
|
||||||
"source_remote": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
|
"source_remote": "ssh://git@git.fromaitochitta.com/open/llm-security.git",
|
||||||
"verified": "differentially, against the module file at b0de0ca - all 83 patterns compared on source, flags and label, in family and array order, 2026-08-09. See source_fidelity.",
|
"verified": "differentially, against the module file at b0de0ca - all 83 patterns compared on source, flags and label, in family and array order, 2026-08-09. See source_fidelity.",
|
||||||
"evidence_limits": [
|
"evidence_limits": [
|
||||||
"The severity the engine assigns to the hybrid family was not supplied. See families[].severity_note.",
|
"Family severities are not a field in the source. Three of the four are named by the constant they live in; the hybrid family's is set by the engine at the point of match. This file records the value and where it is set, but a consumer that changes how it buckets matches can still diverge without contradicting any value here.",
|
||||||
"Whether a consumer applies these patterns to raw or to normalised text is engine behaviour and is not fixed by this file. Two runtimes matching identical patterns against differently normalised text will still disagree on identical input."
|
"Whether a consumer applies these patterns to raw or to normalised text is engine behaviour and is not fixed by this file. Two runtimes matching identical patterns against differently normalised text will still disagree on identical input."
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|
@ -879,8 +887,15 @@
|
||||||
{
|
{
|
||||||
"id": "hybrid",
|
"id": "hybrid",
|
||||||
"source_export": "HYBRID_PATTERNS",
|
"source_export": "HYBRID_PATTERNS",
|
||||||
"severity": null,
|
"severity": "high",
|
||||||
"severity_note": "The seed dump supplies this array but NOT the severity the engine assigns to it. Left null rather than inferred from the three sibling arrays. A consumer MUST NOT assume a severity for this family until the producing module is published.",
|
"severity_note": "Resolved 2026-08-09. This family has no severity field of its own in the source: the engine assigns it by pushing HYBRID_PATTERNS matches straight into the `high` bucket, at scanners/lib/injection-patterns.mjs:274-281 under the comment 'Hybrid patterns are HIGH severity (v5.0 S6)'. Note that this family therefore shares a severity with the `high` family without sharing an array - a consumer that reconstructs the engine must keep the arrays separate even though both produce `high`.",
|
||||||
|
"severity_provenance": {
|
||||||
|
"source_file": "scanners/lib/injection-patterns.mjs",
|
||||||
|
"source_lines": "274-281",
|
||||||
|
"source_commit": "b0de0ca6d86ce697f39669d177c2c2654c280128",
|
||||||
|
"verified": "directly, by reading the module at the pinned commit",
|
||||||
|
"not_from": "scanners/lib/severity.mjs, which was the first file asked about and which contains no injection-family severity at all - grep for CRITICAL_PATTERNS, HIGH_PATTERNS, MEDIUM_PATTERNS or HYBRID_PATTERNS there returns nothing. The Python guard's port carries the correct value but cites that file, so a consumer following its citation would find no answer. Recorded because a wrong citation to a right value is the harder defect to notice."
|
||||||
|
},
|
||||||
"count": 8,
|
"count": 8,
|
||||||
"patterns": [
|
"patterns": [
|
||||||
{
|
{
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue