open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

feat(llm-security): v8 Phase 5 - implement conformance spec 1.1 (not-applicable)

Browse source 
Commons v0.2.0 (532d70d) grew a MUST: a runtime claiming conformance must
declare the set of commons data files it implements and publish that set with
its result. The corpus went 83 -> 89; the six new cases are scoped to
signatures/active-content.json, which this runtime does not implement. Under
1 they would be six permanent failures; under 1.1 they are not-applicable, a
third verdict distinct from 1's error.

Published shape: 83/83 passed, 6 not-applicable. The six are enumerated by
name, reported as skipped rather than passed, and stay in the denominator.

The declaration is DERIVED from the same constant the runner uses to accept a
scope (DECLARED_TABLES), so it cannot drift from what the suite actually runs
- this is what was promised to commons in reply 20260811T104628Z.

Anti-narrowing was NOT enforced by construction, contrary to the claim in the
reply. Measured: setting DECLARED_TABLES to the empty set turns all 89 cases
not-applicable and leaves the suite GREEN with zero cases run - exactly the
exit 1.1 forbids. "Visible as a code change" describes a reviewer, not a gate.

Closed with a derived floor rather than a second hand-maintained table list
(which would be the parallel declaration we promised not to keep): a commons
table whose aliases name llm_security has registered this runtime as a
consumer per 3.1, and a registered consumer that stops declaring the table is
withdrawing a published claim. The universe of tables comes from the manifest;
membership comes from each table's own aliases. Measured: only the injection
lexicon names us, so the floor is one table and the other three carry no
obligation.

Mutation-proven, all six firing:
  declaration -> []                  green, 0 cases -> 1 fail   (the defect above)
  declaration -> wrong table         7 fail
  over-declare an unimplemented one  6 fail
  count_by_scope 83 -> 82            1 fail
  drop a case from manifest.cases    2 fail
  strip our alias registration       85 fail

Suite 2173 -> 2181 (+2 gates, +6 not-applicable), 2174 pass, 0 real failures.
The one red under parallel load was pre-compact-scan size-cap, a known timing
flake: 358 ms alone against a 1000 ms cap, 1886 ms under load. Budget untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Uj6iB8uUUUFHLgAya1hZt7
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 13:10:57 +02:00
commit e1511f91aa
1 changed files with 186 additions and 14 deletions
Download patch file Download diff file Expand all files Collapse all files

200
tests/lib/conformance-corpus.test.mjs
View file

@ -8,16 +8,30 @@
// scanners/commons/spec/conformance-corpus.md; where this file and that
// document disagree, the document wins.
//
// Two spec rules shape the structure below and are easy to violate by accident:
// Three spec rules shape the structure below and are easy to violate by accident:
//
// §1 A runtime MUST run every case, and a case that cannot be run MUST be
// reported as an error, not as a pass. So case discovery is driven by
// the manifest's declared list, cross-checked against the directories on
// disk — a dropped case dir must fail loudly, not shrink the gate.
//
// §4 A runtime MUST reject a `match` or `scope` it does not implement
// rather than degrade to a weaker comparison. Hence the explicit throws
// instead of a skip.
// §1.1 A runtime MUST declare the set of commons data files it implements and
// publish that set with its result. A case scoped to a file outside the
// declared set is `not-applicable` — a third verdict, distinct from §1's
// error: §1 means *we tried and could not*, not-applicable means *this
// question was never addressed to us*. Such a case MUST still be
// enumerated, MUST NOT count as a pass, and MUST NOT leave the
// denominator. `83/83 passed, 6 not-applicable` is the shape.
//
// §4 A runtime MUST reject a `match` it does not implement rather than
// degrade to a weaker comparison. Hence the explicit throw.
//
// §1.1 has a gate of its own: a runtime MUST NOT narrow its declared set to
// turn failures into not-applicable. That is enforced here by construction
// rather than remembered — DECLARED_TABLES is the *same* constant the runner
// uses to accept a scope, so narrowing the declaration narrows what this suite
// will run at all, and shows up as a code change. A hand-maintained
// declaration would be a claim; a derived one is a measurement.
//
// Findings are named by commons `pattern_id`, which scanForInjection() does not
// carry: it returns our own labels. The lexicon publishes the mapping per
@ -28,7 +42,7 @@
// lookup, not a private translation table: nothing here restates a pattern's
// id, severity or label, so nothing here can drift from the lexicon.
import { describe, it, before } from 'node:test';
import { describe, it, before, after } from 'node:test';
import assert from 'node:assert';
import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs';
import { join, dirname } from 'node:path';
@ -41,10 +55,99 @@ const COMMONS_ROOT = join(__dirname, '..', '..', 'scanners', 'commons');
const CORPUS_ROOT = join(COMMONS_ROOT, 'conformance');
const SUPPORTED_MATCH = 'exact-within-scope';
const SUPPORTED_SCOPE = 'lexicon/injection-lexicon.json';
/**
* Spec §1.1: the set of commons data files this runtime implements.
*
* This is the ONLY scope authority in the file. It is what gets published as
* our declaration, and it is what decides whether a case is run — one constant,
* two uses, so the two cannot disagree. `signatures/active-content.json` is
* absent because we do not implement that table, not because its cases fail;
* see the note on the not-applicable verdict below.
*/
const DECLARED_TABLES = Object.freeze(['lexicon/injection-lexicon.json']);
const manifest = JSON.parse(readFileSync(join(CORPUS_ROOT, 'manifest.json'), 'utf8'));
/**
* case_id -> parsed expected.json, read once at load.
*
* Eager on purpose: a case the manifest declares but which has no directory is
* a §1 error, and erroring here makes the whole file fail loudly rather than
* letting the suite shrink to the cases that happen to still exist.
*/
const caseSpecs = new Map(manifest.cases.map((caseId) => {
const specPath = join(CORPUS_ROOT, caseId, 'expected.json');
try {
return [caseId, JSON.parse(readFileSync(specPath, 'utf8'))];
} catch (err) {
throw new Error(`case ${caseId}: declared by the manifest but unreadable at ${specPath} — spec §1 requires an error, not a shrunken gate (${err.message})`);
}
}));
/**
* §1.1 verdict for one case, decided by its scope alone.
*
* Note what this deliberately cannot express: there is no per-case opt-out.
* The verdict attaches to the TABLE, so a case scoped to a table we declare is
* run even when it fails. That is the point — per-case opt-out is exactly the
* silent skip §1 forbids.
*/
function isApplicable(caseId) {
const scope = caseSpecs.get(caseId).scope || [];
if (scope.length === 0) {
throw new Error(`case ${caseId}: empty scope — spec §3 requires a scope, and an unscoped case cannot be compared`);
}
return scope.every((table) => DECLARED_TABLES.includes(table));
}
const applicableCases = manifest.cases.filter(isApplicable);
const notApplicableCases = manifest.cases.filter((c) => !isApplicable(c));
/**
* §1.1's anti-narrowing constraint, as a gate rather than an intention.
*
* Measured, not assumed: narrowing DECLARED_TABLES to `[]` turns all 89 cases
* not-applicable and leaves this suite GREEN with zero cases run — precisely
* the exit §1.1 forbids. "Narrowing is visible as a code change" describes a
* reviewer, not a gate, so the floor below is derived and asserted instead.
*
* The floor is deliberately NOT a second hand-maintained list of tables; that
* would be the parallel declaration this runtime promised commons it would not
* keep, and it would rot against DECLARED_TABLES the first time one moved. It
* is read out of the vendored tables themselves: a table whose `aliases` name
* `llm_security` has registered this runtime as a consumer (spec §3.1), and a
* registered consumer that stops declaring the table is withdrawing a
* published claim rather than describing what it implements.
*/
/** Every data file the corpus can scope a case to — from the manifest alone. */
function corpusTables() {
return [...new Set([
...(manifest.scope_covered || []),
...Object.keys(manifest.scope_planned || {}),
])].filter((key) => key.includes('/') && key.endsWith('.json'));
}
/** Does this vendored table register `llm_security` as a consumer? */
function registersUs(table) {
const tablePath = join(COMMONS_ROOT, table);
if (!existsSync(tablePath)) return false;
let found = false;
(function walk(node) {
if (found || node === null || typeof node !== 'object') return;
if (!Array.isArray(node) && node.aliases !== null && typeof node.aliases === 'object'
&& Object.hasOwn(node.aliases, 'llm_security')) {
found = true;
return;
}
for (const child of Array.isArray(node) ? node : Object.values(node)) walk(child);
})(JSON.parse(readFileSync(tablePath, 'utf8')));
return found;
}
const REGISTERED_TABLES = corpusTables().filter(registersUs);
/** label -> pattern_id, read from the lexicon's published aliases. */
function buildAliasMap() {
const lexicon = JSON.parse(readFileSync(join(COMMONS_ROOT, 'lexicon', 'injection-lexicon.json'), 'utf8'));
@ -67,16 +170,16 @@ const aliasMap = buildAliasMap();
*/
function runCase(caseId) {
const caseDir = join(CORPUS_ROOT, caseId);
const expected = JSON.parse(readFileSync(join(caseDir, 'expected.json'), 'utf8'));
const expected = caseSpecs.get(caseId);
assert.equal(expected.case_id, caseId, 'expected.json case_id must equal its directory name');
if (expected.match !== SUPPORTED_MATCH) {
throw new Error(`case ${caseId}: unimplemented match "${expected.match}" — spec §4 forbids degrading to a weaker comparison`);
}
const scope = expected.scope || [];
if (scope.length !== 1 || scope[0] !== SUPPORTED_SCOPE) {
throw new Error(`case ${caseId}: unimplemented scope ${JSON.stringify(scope)}`);
}
// Scope is not re-checked here: §1.1 decided it before the case was routed to
// this function, and re-deciding it would be the second authority the single
// DECLARED_TABLES constant exists to prevent.
assert.ok(isApplicable(caseId), `case ${caseId}: routed to the runner despite an undeclared scope`);
// §2: bytes, not text. Both integrity fields are checked because two cases
// carry characters that are invisible on screen — a zero-width space and a
@ -115,7 +218,7 @@ describe('commons conformance corpus — exact-within-scope through scanForInjec
assert.deepEqual(onDisk, [...manifest.cases].sort(),
'case directories on disk disagree with manifest.cases');
assert.equal(onDisk.length, manifest.count, 'manifest.count disagrees with its own case list');
assert.equal(manifest.count, 83, 'corpus size changed — re-verify before adjusting this number');
assert.equal(manifest.count, 89, 'corpus size changed — re-verify before adjusting this number');
});
it('maps every one of our injection labels onto a lexicon pattern_id', () => {
@ -124,14 +227,83 @@ describe('commons conformance corpus — exact-within-scope through scanForInjec
// into a pass, so the bijection is asserted rather than assumed.
assert.equal(aliasMap.size, 83, 'lexicon no longer publishes 83 llm_security aliases');
assert.equal(manifest.match_semantics, SUPPORTED_MATCH);
assert.deepEqual(manifest.scope_covered, [SUPPORTED_SCOPE]);
});
for (const caseId of manifest.cases) {
it('cannot withdraw a table it is registered against (spec §1.1 anti-narrowing)', () => {
// Without this, DECLARED_TABLES = [] is a GREEN suite that runs zero cases.
// Measured — it was, before this test existed.
assert.ok(REGISTERED_TABLES.length > 0,
'no vendored table names llm_security in its aliases — the floor has nothing to stand on, so narrowing the declaration would go undetected again');
for (const table of REGISTERED_TABLES) {
assert.ok(DECLARED_TABLES.includes(table),
`${table} registers llm_security as a consumer but is not in the declared set — §1.1 forbids withdrawing a table to convert failures into not-applicable; if this runtime genuinely dropped the table, the registration in commons must go first`);
}
});
it('declares its table set, and the declaration partitions the corpus (spec §1.1)', () => {
// 1. The declaration is meaningful only against tables the corpus actually
// covers. Declaring a table the corpus does not scope any case to would
// be an unfalsifiable claim.
for (const table of DECLARED_TABLES) {
assert.ok(manifest.scope_covered.includes(table),
`declared table ${table} is not in the corpus's scope_covered — the declaration claims something the corpus cannot check`);
}
// 2. The partition is total and disjoint: every declared case gets exactly
// one verdict, so nothing leaves the denominator.
assert.deepEqual(
[...applicableCases, ...notApplicableCases].sort(),
[...manifest.cases].sort(),
'applicable + not-applicable is not the whole corpus — a case has escaped the denominator');
assert.equal(applicableCases.length + notApplicableCases.length, manifest.count);
// 3. The counts are DERIVED, not restated. manifest.count_by_scope is the
// corpus's own tally per table; our applicable count must equal the sum
// over the tables we declare. This is what makes narrowing the
// declaration visible: drop a table and this number drops with it.
const expectedApplicable = DECLARED_TABLES
.reduce((sum, table) => sum + (manifest.count_by_scope[table] ?? 0), 0);
assert.equal(applicableCases.length, expectedApplicable,
'applicable count disagrees with manifest.count_by_scope for the declared tables');
// 4. Every not-applicable case is scoped ONLY to undeclared tables. Without
// this, a case scoped to a table we declare could slip into the
// not-applicable bucket — the per-case opt-out §1.1 forbids.
for (const caseId of notApplicableCases) {
const scope = caseSpecs.get(caseId).scope;
assert.ok(scope.some((t) => !DECLARED_TABLES.includes(t)),
`case ${caseId} is marked not-applicable but every table it scopes is declared`);
}
});
for (const caseId of applicableCases) {
it(`${caseId}`, () => {
const { missing, extra } = runCase(caseId);
assert.deepEqual(missing, [], `case ${caseId}: expected findings the runtime did not raise`);
assert.deepEqual(extra, [], `case ${caseId}: lexicon findings the runtime raised but the case does not list`);
});
}
// §1.1: enumerated by name, reported as skipped rather than passed. The
// runner's own `# skipped` tally is the third verdict — these cases are
// neither counted as passes nor dropped from the corpus total.
for (const caseId of notApplicableCases) {
const scope = caseSpecs.get(caseId).scope.join(', ');
it.skip(`${caseId} — not-applicable: scoped to ${scope}, which this runtime does not declare`, () => {});
}
after(() => {
// The declaration §1.1 requires us to publish alongside the result. Pass
// and fail counts are the runner's to report; what belongs here is the
// declared set and the enumerated not-applicable cases.
const lines = [
'',
` commons conformance — ${manifest.id} v${manifest.version} (${manifest.count} cases)`,
` declared tables (spec §1.1): ${DECLARED_TABLES.join(', ')}`,
` applicable: ${applicableCases.length} not-applicable: ${notApplicableCases.length}`,
...notApplicableCases.map((c) => ` not-applicable ${c} [${caseSpecs.get(c).scope.join(', ')}]`),
'',
];
console.log(lines.join('\n'));
});
});