open/llm-security
1
0
Fork 0
Code Issues Releases 1 Activity

feat(llm-security): publish the spec 1.1 conformance declaration as an artifact

Browse source 
Closes the gap STATE has been carrying since e1511f9. Section 1.1's runtime
behaviour has been correct since then -- the third verdict is real, the
declared set is one constant with two uses -- but what we PUBLISHED was a
console.log summary: the right facts in a format only its author could
parse, which is most of what section 1.1 exists to prevent. Commons
shipped a shape for it in v0.3.0
(schema/conformance-declaration.schema.json 0.1.0), so the artifact can
now exist.

Every field is counted from the run rather than restated. The cases record
their own verdict as they execute, keeping `failed` (ran and disagreed)
apart from `error` (could not run) on exactly the distinction section 1
turns on. `commons_commit` is read out of the subtree-pull subject in our
own history rather than transcribed into a constant that would drift at
the next pull, and it refuses to publish a coordinate it cannot determine
-- a fabricated commit is worse than no declaration. The artifact is
gitignored: a committed declaration keeps asserting what was true once,
and nothing makes it wrong out loud when it stops being.

Validated once against the vendored schema with a real 2020-12
implementation: VALID, and the validator proven discriminating by six
negative controls it rejected (dropped zero-count, unknown key,
out-of-enum source, non-integer count, missing enumeration, malformed
case id). Continuous validation would mean a Python dependency in a suite
that has none, so what stays is the cheap half that actually drifts -- the
two key sets, asserted exactly.

TWO DEFECTS FOUND BY MUTATING THIS GATE, both in its own first draft:

1. It lived in `after()`. Measured on Node 25.8.2: an assertion that fails
   in an after hook prints under "failing tests" and marks the suite red,
   but leaves `fail 0` and exit code ZERO. `npm test` and CI would have
   read a falsified declaration as green. The gate against "reports
   success without running" was itself reporting success without running.
   It is now a test, declared last, and the verdict-count assertion is
   what guards the ordering that makes "last" meaningful.

2. Nothing tied the PUBLISHED `declared_tables` to the runner's constant.
   Substituting a literal list left every other assertion green, because
   they all read the constant rather than what was published -- so
   `declaration_source: derived-from-runner` could be a lie with no code
   change to point at. Now asserted identical.

Seven mutations, all exiting non-zero: dropped zero-count, falsified
not_applicable, unpublished field, hand-maintained tables, lied-about
source, and a hardcoded `passed` combined with a genuine case failure.

Published this run: 90 total, 84 passed, 0 failed, 6 not-applicable,
0 error, at commons 4641a7b (v0.3.0). Full suite 2192 pass / 0 fail /
6 skipped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017XDdiKC9ZXmcSUQ2m84s6y
This commit is contained in:
Kjell Tore Guttormsen 2026-08-11 14:21:09 +02:00
commit 47905dacae
2 changed files with 177 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

185
tests/lib/conformance-corpus.test.mjs
View file

@ -46,18 +46,31 @@
// lookup, not a private translation table: nothing here restates a pattern's
// id, severity or label, so nothing here can drift from the lexicon.
import { describe, it, before, after } from 'node:test';
import { describe, it, before } from 'node:test';
import assert from 'node:assert';
import { readFileSync, readdirSync, statSync, existsSync } from 'node:fs';
import { readFileSync, readdirSync, statSync, existsSync, mkdirSync, writeFileSync } from 'node:fs';
import { execFileSync } from 'node:child_process';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { createHash } from 'node:crypto';
import { scanForInjection } from '../../scanners/lib/injection-patterns.mjs';
const __dirname = dirname(fileURLToPath(import.meta.url));
const COMMONS_ROOT = join(__dirname, '..', '..', 'scanners', 'commons');
const REPO_ROOT = join(__dirname, '..', '..');
const COMMONS_ROOT = join(REPO_ROOT, 'scanners', 'commons');
const CORPUS_ROOT = join(COMMONS_ROOT, 'conformance');
/**
* Where the §1.1 declaration is published, per run.
*
* Gitignored on purpose. A committed declaration is a claim frozen at commit
* time: its `measured_date` and both commit fields keep saying what was true
* once, and nothing makes them wrong out loud when they stop being true. A
* declaration regenerated by the run that measured it cannot rot, because it
* does not outlive the measurement.
*/
const DECLARATION_PATH = join(REPO_ROOT, 'reports', 'conformance-declaration.json');
const SUPPORTED_MATCH = 'exact-within-scope';
/**
@ -167,6 +180,80 @@ function buildAliasMap() {
const aliasMap = buildAliasMap();
/** case_id -> 'passed' | 'failed' | 'error', filled in by the cases as they run. */
const verdicts = new Map();
/**
* The commons commit this corpus came from, read out of our own history.
*
* The schema requires a COMMIT and notes that a declaration citing a tag
* should give the commit the tag resolved to, "because a tag can be moved and
* a commit cannot". `git subtree pull --squash` records the upstream range in
* its own commit subject, so the coordinate is already in this repository and
* does not have to be transcribed by hand into a constant that would drift
* from the vendored tree at the next pull.
*
* Deliberately not fallback-tolerant: a declaration naming a commons commit it
* could not determine would be a fabricated coordinate, which is worse than no
* declaration. Tests do not ship (`package.json` `files` covers bin/, scanners/,
* knowledge/), so this only ever runs inside a checkout.
*/
function commonsCommit() {
const subject = execFileSync(
'git', ['log', '-1', '--format=%s', "--grep=Squashed 'scanners/commons/'"],
{ cwd: REPO_ROOT, encoding: 'utf8' },
).trim();
const m = /\.\.([0-9a-f]{7,40})$/.exec(subject);
if (m === null) {
throw new Error(`cannot determine the vendored commons commit from git history (last subtree subject: "${subject}") — refusing to publish a declaration with a coordinate nobody can check`);
}
return m[1];
}
/**
* Build the spec §1.1 declaration from what the run actually produced.
*
* Every number here is counted from `verdicts`, and `declared_tables` is the
* same constant the runner accepts scopes against — which is what lets this
* declaration claim `declaration_source: derived-from-runner` honestly. The
* five counts are emitted including the zeros: commons' schema is explicit
* that an absent count is indistinguishable from one the runtime never
* tracked, so `0` is information and a missing key is not.
*/
function buildDeclaration() {
const tally = (name) => [...verdicts.values()].filter((v) => v === name).length;
const idsWith = (name) => [...verdicts.entries()].filter(([, v]) => v === name).map(([id]) => id).sort();
const declaration = {
runtime: 'llm_security',
runtime_commit: execFileSync('git', ['rev-parse', 'HEAD'], { cwd: REPO_ROOT, encoding: 'utf8' }).trim(),
commons_commit: commonsCommit(),
commons_version: `v${manifest.version}`,
declared_tables: [...DECLARED_TABLES],
declaration_source: 'derived-from-runner',
measured_date: new Date().toISOString().slice(0, 10),
result: {
total: applicableCases.length + notApplicableCases.length,
passed: tally('passed'),
failed: tally('failed'),
not_applicable: notApplicableCases.length,
error: tally('error'),
match: manifest.match_semantics,
},
not_applicable_cases: [...notApplicableCases].sort(),
};
// The schema requires these arrays whenever their count is non-zero: an
// unnamed failure is one nobody can reproduce. Omitted when empty rather
// than published as `[]`, so their presence always means something happened.
const failed = idsWith('failed');
const errored = idsWith('error');
if (failed.length > 0) declaration.failed_cases = failed;
if (errored.length > 0) declaration.error_cases = errored;
return declaration;
}
/**
* Run one case and return { missing, extra } within the case's scope.
* Anything the runtime raises that is not a lexicon pattern is out of scope and
@ -282,9 +369,23 @@ describe('commons conformance corpus — exact-within-scope through scanForInjec
for (const caseId of applicableCases) {
it(`${caseId}`, () => {
const { missing, extra } = runCase(caseId);
assert.deepEqual(missing, [], `case ${caseId}: expected findings the runtime did not raise`);
assert.deepEqual(extra, [], `case ${caseId}: lexicon findings the runtime raised but the case does not list`);
// The verdict is recorded here, from the case that produced it, because
// §1.1's declaration must be a measurement rather than a summary written
// afterwards. `failed` and `error` are kept apart on the same distinction
// the spec turns on: an AssertionError means the case ran and disagreed;
// anything else means the runtime could not run it at all (§1).
let verdict = 'error';
try {
const { missing, extra } = runCase(caseId);
assert.deepEqual(missing, [], `case ${caseId}: expected findings the runtime did not raise`);
assert.deepEqual(extra, [], `case ${caseId}: lexicon findings the runtime raised but the case does not list`);
verdict = 'passed';
} catch (err) {
verdict = err instanceof assert.AssertionError ? 'failed' : 'error';
throw err;
} finally {
verdicts.set(caseId, verdict);
}
});
}
@ -296,16 +397,78 @@ describe('commons conformance corpus — exact-within-scope through scanForInjec
it.skip(`${caseId} — not-applicable: scoped to ${scope}, which this runtime does not declare`, () => {});
}
after(() => {
// The declaration §1.1 requires us to publish alongside the result. Pass
// and fail counts are the runner's to report; what belongs here is the
// declared set and the enumerated not-applicable cases.
// Declared LAST on purpose, and a test rather than an `after` hook.
//
// Last, because it can only publish what the cases above have already
// decided; node:test runs subtests in declaration order unless concurrency
// is opted into, and the `verdicts.size` assertion below fails loudly if
// that ever stops holding.
//
// A test and not a hook, because MEASURED on Node 25.8.2: an assertion that
// fails inside `after()` prints under "failing tests" and marks the suite ✖,
// but leaves `fail 0` and an exit code of ZERO. `npm test` and CI would read
// that run as green. This gate spent its first draft in that hook, which
// made it exactly the "reports success without running" failure it exists to
// catch. Verified by mutation: falsifying a count now exits non-zero.
it('publishes a spec §1.1 declaration derived from this run', () => {
// The declaration §1.1 requires us to publish alongside the result, in the
// shape commons' schema/conformance-declaration.schema.json 0.1.0 defines.
// Until this landed, what we published was a console.log summary — the
// right facts in a format only its author could parse, which is most of
// what §1.1 exists to prevent.
const declaration = buildDeclaration();
// The schema states two invariants it cannot express in JSON Schema, and
// notes that "a validator that checks the schema and not these has checked
// the shape of a claim without checking the claim". They are asserted
// before the write, so an unsound declaration is never published.
const { total, passed, failed, not_applicable: notApplicable, error } = declaration.result;
assert.equal(passed + failed + notApplicable + error, total,
'denominator_intact: the four verdict counts do not sum to total — cases were dropped between the run and the publication');
assert.equal(total, manifest.count,
'total_matches_corpus: fewer cases were enumerated than the corpus holds at this commons commit — §1 forbids skipping');
// Every applicable case must have reached a verdict. Without this, a case
// whose body never ran would leave `total` intact while quietly reducing
// the numerator, which is the same shrunken gate §1 forbids.
assert.equal(verdicts.size, applicableCases.length,
'an applicable case produced no verdict — the declaration would understate what was measured');
// `declaration_source: derived-from-runner` is a claim about where the
// array above came from, and commons is explicit that the distinction is
// not cosmetic. Found by mutation: substituting a literal list for
// DECLARED_TABLES in the builder left every other assertion green, because
// they all read the constant rather than what was published. So the
// published array is tied back to the constant the runner routes cases
// with — which is what makes the field a measurement instead of a label.
assert.deepEqual(declaration.declared_tables, [...DECLARED_TABLES],
'declared_tables was published as something other than the runner\'s own constant — declaration_source claims derived-from-runner, so these cannot differ');
assert.equal(declaration.declaration_source, 'derived-from-runner');
// The schema closes `additionalProperties` on both objects and requires all
// five counts including the zeros. Validating that properly needs a JSON
// Schema implementation, and this suite has no external dependencies — the
// full 2020-12 validation was run once against the vendored schema when
// this landed (valid, with six negative controls rejected). What is worth
// holding continuously is the cheap half that actually drifts: a field
// added to the builder without commons publishing it, or a count dropped.
assert.deepEqual(Object.keys(declaration).sort(), [
'commons_commit', 'commons_version', 'declaration_source', 'declared_tables',
'measured_date', 'not_applicable_cases', 'result', 'runtime', 'runtime_commit',
], 'declaration key set drifted from the schema — additionalProperties is closed, and failed_cases/error_cases appear only when non-zero');
assert.deepEqual(Object.keys(declaration.result).sort(),
['error', 'failed', 'match', 'not_applicable', 'passed', 'total'],
'result key set drifted from the schema — all five counts are required, zeros included');
mkdirSync(dirname(DECLARATION_PATH), { recursive: true });
writeFileSync(DECLARATION_PATH, `${JSON.stringify(declaration, null, 2)}\n`, 'utf8');
const lines = [
'',
` commons conformance — ${manifest.id} v${manifest.version} (${manifest.count} cases)`,
` declared tables (spec §1.1): ${DECLARED_TABLES.join(', ')}`,
` applicable: ${applicableCases.length} not-applicable: ${notApplicableCases.length}`,
` ${passed} passed, ${failed} failed, ${error} error, ${notApplicable} not-applicable (of ${total})`,
...notApplicableCases.map((c) => ` not-applicable ${c} [${caseSpecs.get(c).scope.join(', ')}]`),
` declaration: ${DECLARATION_PATH}`,
'',
];
console.log(lines.join('\n'));