Kjell Tore Guttormsen 22d3a78591 docs: measure the lexicon port divergence, correct the case count

The guard's coverage.py turned out to be readable from a session here — the
read boundary covers llm-security only. Counting it instead of estimating
corrected two published numbers and surfaced the real conformance blocker.

Counted, not estimated: CORE_CASES holds 134 cases (128 caught, 6 gap), not
the "126 classes + 4 gaps-must-hold" the extraction plan claimed. 105 are
convertible to static input/expected; 29 assert a runtime API surface this
repository does not own, and are named as out of scope rather than faked.

Measured while counting, and the reason the corpus is still blocked: the
commons lexicon and the guard's are two ports of one source file, and they
disagree. 83/83 patterns correspond, 64 byte-identical, 6 differ only by
escaping and are proven equivalent by string identity, and 13 behave
differently — with a witness input for each and misses in both directions.
Cause is two different ReDoS mitigations of the same table: the Node side
bounds the run ({0,256}), the guard excludes the anchor character ([^><]).
Each has a recall cost the other does not.

No data file changed. Behaviour preservation is a v0.1.0 invariant, so the
divergence is reported to the owning repositories, not fixed here. The 8
hybrid severities the guard's port assigns are deliberately NOT copied in —
a second-hand port is not the producing module.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FaYqid3mejFmd9ZHsiHgp3

2026-08-09 21:40:26 +02:00

3.6 KiB

Raw Blame History

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

Versioning note: the repository tag versions the contract (file set, key names, case ids, disposition semantics). Each JSON file additionally carries its own "version" field, bumped when that file changes.

[Unreleased]

Initial extraction, in progress. Runtime-neutral detection data and the finding contract, extracted from the llm-security Node implementation and a Python guard without behaviour change. Not yet tagged — see Not included below.

Added

schema/finding.schema.json — the finding contract plus the SARIF output profile. Normative. The JSONL profile is deliberately left unspecified.
signatures/active-content.json — the EchoLeak class (CVE-2025-32711): 17 patterns, severities, opacity floors and pass order, from the Python guard.
lexicon/injection-lexicon.json — 83 prompt-injection patterns in four families (21 critical, 32 high, 22 medium, 8 hybrid).
codepoints/carriers.json — six carrier tables: zero-width characters, the Unicode Tags block, the Supplementary Private Use Areas, BIDI controls, the Cyrillic presence set and the 28-entry fold-to-Latin homoglyph map.
signatures/secret-egress.json — the 18 fixed credential and token shapes. Array order is normative.
mapping/owasp-map.json — four taxonomy maps (LLM, ASI, AST, MCP) over one shared 16-prefix key set.
calibration/calibration.json — risk-score tier constants, verdict thresholds, risk-band cutoffs, posture grade thresholds.

Verification

Every file above except calibration.json was proven rather than transcribed: the data was rebuilt from the commons JSON alone and diffed against the source implementation. Each file records its own result and its own limits.

calibration/calibration.json carries verified: false. Its source arrived as a prose summary rather than as code, so no differential check was possible, and the file names the checks that were not run instead of attaching a caveat to a pass.

docs/lexicon-port-divergence.md — informative. A differential comparison of the two ports of injection-patterns.mjs (this repository's and the Python guard's): 83/83 patterns correspond, 64 are byte-identical, 6 differ only by escaping and are proven equivalent, and 13 behave differently, with a witness input for each and misses on both sides. The cause is two different ReDoS mitigations of one table. No data file was changed — behaviour preservation holds and the finding is reported to the owning repositories.

Not included

signatures/malware-signatures.json — seed data not yet delivered.
spec/decode-pipeline.md — needs the decode implementation. A normative spec inferred from a data dump would be worse than an absent one.
conformance/ — blocked, and the blocker is now identified. 105 of the guard's 134 coverage cases are convertible to static input.txt/expected.json; the other 29 assert a runtime's API surface, which this repository does not own. But no fixture can be written yet at all: an expected.json names findings, and the two runtimes have no shared name for the same pattern (override:ignore-previous vs override: ignore previous instructions). A commons-owned pattern id is a prerequisite for the corpus, not a parallel task.

These are named in the README as planned rather than linked, so nothing in the repository points at a file that does not exist.