Kjell Tore Guttormsen
22d3a78591
The guard's coverage.py turned out to be readable from a session here — the
read boundary covers llm-security only. Counting it instead of estimating
corrected two published numbers and surfaced the real conformance blocker.
Counted, not estimated: CORE_CASES holds 134 cases (128 caught, 6 gap), not
the "126 classes + 4 gaps-must-hold" the extraction plan claimed. 105 are
convertible to static input/expected; 29 assert a runtime API surface this
repository does not own, and are named as out of scope rather than faked.
Measured while counting, and the reason the corpus is still blocked: the
commons lexicon and the guard's are two ports of one source file, and they
disagree. 83/83 patterns correspond, 64 byte-identical, 6 differ only by
escaping and are proven equivalent by string identity, and 13 behave
differently — with a witness input for each and misses in both directions.
Cause is two different ReDoS mitigations of the same table: the Node side
bounds the run ({0,256}), the guard excludes the anchor character ([^><]).
Each has a recall cost the other does not.
No data file changed. Behaviour preservation is a v0.1.0 invariant, so the
divergence is reported to the owning repositories, not fixed here. The 8
hybrid severities the guard's port assigns are deliberately NOT copied in —
a second-hand port is not the producing module.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3
67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
# Changelog
|
|
|
|
All notable changes to this project will be documented in this file.
|
|
|
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
|
|
project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
|
|
Versioning note: the repository tag versions **the contract** (file set, key names,
|
|
case ids, disposition semantics). Each JSON file additionally carries its own
|
|
`"version"` field, bumped when that file changes.
|
|
|
|
## [Unreleased]
|
|
|
|
Initial extraction, in progress. Runtime-neutral detection data and the finding contract,
|
|
extracted from the `llm-security` Node implementation and a Python guard without behaviour
|
|
change. **Not yet tagged** — see *Not included* below.
|
|
|
|
### Added
|
|
|
|
- `schema/finding.schema.json` — the finding contract plus the SARIF output profile.
|
|
Normative. The JSONL profile is deliberately left `unspecified`.
|
|
- `signatures/active-content.json` — the EchoLeak class (CVE-2025-32711): 17 patterns,
|
|
severities, opacity floors and pass order, from the Python guard.
|
|
- `lexicon/injection-lexicon.json` — 83 prompt-injection patterns in four families
|
|
(21 critical, 32 high, 22 medium, 8 hybrid).
|
|
- `codepoints/carriers.json` — six carrier tables: zero-width characters, the Unicode Tags
|
|
block, the Supplementary Private Use Areas, BIDI controls, the Cyrillic presence set and
|
|
the 28-entry fold-to-Latin homoglyph map.
|
|
- `signatures/secret-egress.json` — the 18 fixed credential and token shapes. Array order
|
|
is normative.
|
|
- `mapping/owasp-map.json` — four taxonomy maps (LLM, ASI, AST, MCP) over one shared
|
|
16-prefix key set.
|
|
- `calibration/calibration.json` — risk-score tier constants, verdict thresholds, risk-band
|
|
cutoffs, posture grade thresholds.
|
|
|
|
### Verification
|
|
|
|
Every file above except `calibration.json` was proven rather than transcribed: the data was
|
|
rebuilt **from the commons JSON alone** and diffed against the source implementation. Each
|
|
file records its own result and its own limits.
|
|
|
|
`calibration/calibration.json` carries `verified: false`. Its source arrived as a prose
|
|
summary rather than as code, so no differential check was possible, and the file names the
|
|
checks that were not run instead of attaching a caveat to a pass.
|
|
|
|
- `docs/lexicon-port-divergence.md` — informative. A differential comparison of the two
|
|
ports of `injection-patterns.mjs` (this repository's and the Python guard's): 83/83
|
|
patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven
|
|
equivalent, and **13 behave differently**, with a witness input for each and misses on
|
|
both sides. The cause is two different ReDoS mitigations of one table. **No data file was
|
|
changed** — behaviour preservation holds and the finding is reported to the owning
|
|
repositories.
|
|
|
|
### Not included
|
|
|
|
- `signatures/malware-signatures.json` — seed data not yet delivered.
|
|
- `spec/decode-pipeline.md` — needs the decode implementation. A normative spec inferred
|
|
from a data dump would be worse than an absent one.
|
|
- `conformance/` — blocked, and the blocker is now identified. 105 of the guard's 134
|
|
coverage cases are convertible to static `input.txt`/`expected.json`; the other 29 assert
|
|
a runtime's API surface, which this repository does not own. But no fixture can be written
|
|
yet at all: an `expected.json` names findings, and the two runtimes have no shared name for
|
|
the same pattern (`override:ignore-previous` vs `override: ignore previous instructions`).
|
|
A commons-owned pattern id is a prerequisite for the corpus, not a parallel task.
|
|
|
|
These are named in the README as planned rather than linked, so nothing in the repository
|
|
points at a file that does not exist.
|