fix(llm-security): golden gate - coverage block measured something else
The `coverage` block never looked at the 61 hook invocations. It probed every
payload STRING against every injection regex in-process, so `47/83` meant
"if you threw all 61 strings at all 83 patterns, 47 would match" - not "the
reference run exercised 47". The pre-bash-destructive payloads never reach
injection-patterns at all, yet their strings were in the probe set and could
mark a pattern exercised.
That number was asserted as reference-run coverage in three places that
instruct a future session: tests/golden/README.md, the STATE golden-gate
section, and the generator's summary line. In a repo whose v7.8.2 lesson was
"the check reported success without running", a figure that measures one
thing while labelled another is the same defect wearing a different hat.
Relabelled rather than re-measured - the probe still honestly bounds the gate
(an unreachable pattern is one the corpus cannot protect under ANY
attribution), it just has to say what it is:
coverage.kind = 'static-reachability', with the caveat in a `note` field.
patternsExercised -> patternsReachable
uncoveredPatterns -> unreachablePatterns
tablesExercised -> corpusContains (a payload CONTAINS a homoglyph; it
does not say the run folded one)
The gate now pins both the `kind` and the note, so the honest label cannot be
dropped quietly by a later edit.
Also surfaced the gap the old wording hid: the four OWASP maps have NO
behavioural coverage - they are scanner-side and no hook in this corpus
reaches them. They are precisely the tables the dump was widened to cover, so
the table digest is their only protection. Stated in the README next to the
47/83 line, where a reader was previously left to infer the reference run
backed them.
Flakiness check for the new gate (61 sequential spawns, ~12s, the most
process-heavy file in the suite, added to a suite npm test runs concurrently):
three consecutive full runs, 2053/2053, 0 fail. The three known
timing-sensitive files did not destabilise.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BB4vXvwvtW4dxbPRd6vsez
This commit is contained in:
parent
8d990e06d3
commit
2fe29152b3
5 changed files with 87 additions and 37 deletions
|
|
@ -70,8 +70,8 @@ console.log(
|
|||
`\n file digests : ${dump.counts.file}` +
|
||||
`\n reference run : ${run.summary.total} cases, ` +
|
||||
`${run.summary.matchingExpectation} match expectation` +
|
||||
`\n coverage : ${run.coverage.patternsExercised}/${run.coverage.patternsTotal} patterns` +
|
||||
` | HOMOGLYPH_MAP ${run.coverage.tablesExercised.HOMOGLYPH_MAP ? 'exercised' : 'NOT exercised'}`
|
||||
`\n reachability : ${run.coverage.patternsReachable}/${run.coverage.patternsTotal} patterns` +
|
||||
` (STATIC probe, not observed coverage — see tests/golden/README.md)`
|
||||
);
|
||||
|
||||
const suite = artifacts.find(([n]) => n === 'suite-counts.json');
|
||||
|
|
|
|||
|
|
@ -269,12 +269,23 @@ export async function buildReferenceRun(root) {
|
|||
}
|
||||
|
||||
/**
|
||||
* Which injection patterns the corpus actually reaches.
|
||||
* STATIC REACHABILITY — not observed coverage. Read the distinction before
|
||||
* quoting the number anywhere.
|
||||
*
|
||||
* Byte-identity over a corpus that trips 5 of 90 patterns would prove almost
|
||||
* nothing, so the uncovered ones are listed by key rather than silently
|
||||
* dropped — a shrinking number is a gate failure, and a large uncovered list
|
||||
* is an honest statement of what this gate does NOT cover.
|
||||
* This probes every injection pattern against every payload string
|
||||
* in-process. It answers "if you threw all 61 payload strings at all 83
|
||||
* regexes, how many would match?" It does NOT answer "how many patterns did
|
||||
* the 61 hook invocations actually evaluate": the `pre-bash-destructive`
|
||||
* payloads never reach injection-patterns at all, yet their strings are in
|
||||
* the probe set and can mark a pattern reachable.
|
||||
*
|
||||
* It is kept because it still bounds the gate honestly — an unreachable
|
||||
* pattern is one the corpus cannot protect under ANY attribution — but it is
|
||||
* named for what it measures. A number labelled as measuring one thing while
|
||||
* measuring another is this repo's v7.8.2 defect class.
|
||||
*
|
||||
* Not covered here at all: the four OWASP maps are scanner-side and no hook
|
||||
* in this corpus reaches them. Their only protection is the table digest.
|
||||
*/
|
||||
async function buildCoverage(root, payloads) {
|
||||
const ip = await import(resolve(root, 'scanners/lib/injection-patterns.mjs'));
|
||||
|
|
@ -289,22 +300,30 @@ async function buildCoverage(root, payloads) {
|
|||
}
|
||||
|
||||
const texts = payloads.flatMap((p) => [p.payload, su.normalizeForScan(p.payload)]);
|
||||
const exercised = [];
|
||||
const uncovered = [];
|
||||
const reachable = [];
|
||||
const unreachable = [];
|
||||
for (const { key, re } of entries) {
|
||||
const probe = new RegExp(re.source, re.flags.replace('g', ''));
|
||||
(texts.some((t) => probe.test(t)) ? exercised : uncovered).push(key);
|
||||
(texts.some((t) => probe.test(t)) ? reachable : unreachable).push(key);
|
||||
}
|
||||
|
||||
const homoglyphHit = texts.some((t) => su.foldHomoglyphs(t) !== t);
|
||||
|
||||
return {
|
||||
kind: 'static-reachability',
|
||||
note:
|
||||
'Payload strings probed against patterns in-process. NOT a measurement ' +
|
||||
'of what the 61 hook invocations evaluated — pre-bash-destructive ' +
|
||||
'payloads never reach injection-patterns yet are in the probe set. ' +
|
||||
'The four OWASP maps are scanner-side and have no behavioural coverage ' +
|
||||
'here at all; their only protection is the table digest.',
|
||||
patternsTotal: entries.length,
|
||||
patternsExercised: exercised.length,
|
||||
uncoveredPatterns: uncovered,
|
||||
tablesExercised: {
|
||||
HOMOGLYPH_MAP: homoglyphHit,
|
||||
normalizeForScan: texts.some((t, i) => i % 2 === 1 && t !== payloads[(i - 1) / 2].payload),
|
||||
patternsReachable: reachable.length,
|
||||
unreachablePatterns: unreachable,
|
||||
corpusContains: {
|
||||
// "a payload contains a homoglyph", not "the run folded one".
|
||||
homoglyphChars: texts.some((t) => su.foldHomoglyphs(t) !== t),
|
||||
payloadsAlteredByNormalize: payloads.filter(
|
||||
(p) => su.normalizeForScan(p.payload) !== p.payload
|
||||
).length,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ The gate is `tests/lib/golden-baseline.test.mjs`.
|
|||
| `patterns.json` | `.source` + `.flags` of every RegExp reachable from the walked modules' exports | After a swap a pattern is `new RegExp(jsonString, flags)`. The plan's named hazard — JSON backslash-doubling — is visible **only** on the compiled object. |
|
||||
| ↳ table records | key/value digest of `HOMOGLYPH_MAP`, `TYPOSQUAT_SUSPICIOUS_TOKENS`, `SEVERITY`, the four OWASP maps | Most of what Phase 4 moves is not a regex. A regex-only dump is blind to a broken homoglyph or OWASP-map swap, i.e. blind to the bulk of the payload. |
|
||||
| ↳ file records | sha256 of the five source files in the moving set | The completeness layer, complete **by construction**: it covers regexes inlined in function bodies (`NAMED` at `string-utils.mjs:291`, the BIDI/tag/PUA ranges at 357–404) that no export walk can reach. |
|
||||
| `reference-run.json` | the 61 showcase payloads through the real hook entry points, plus coverage | Byte identity over a corpus that trips a handful of patterns would prove almost nothing, so coverage is recorded and uncovered patterns are listed by name. |
|
||||
| `reference-run.json` | the 61 showcase payloads through the real hook entry points, plus a static reachability probe | Byte identity over a corpus that trips a handful of patterns would prove almost nothing, so reachability is recorded and unreachable patterns are listed by name. |
|
||||
| `suite-counts.json` | per-file pass/fail, each file run alone | A total is unattributable, and `npm test` runs files concurrently where three timing-sensitive files flake. Running each alone is the only reproducible recording. |
|
||||
|
||||
## Why there is no regex enumerator
|
||||
|
|
@ -37,10 +37,30 @@ A diff here means the swap changed observable behaviour. **Roll the swap back.**
|
|||
`--write` is for deliberately re-blessing a change you have already decided is
|
||||
correct, not for making the gate quiet.
|
||||
|
||||
## Known coverage gap
|
||||
## Known coverage gaps — read before quoting a number
|
||||
|
||||
The reference run exercises 47 of 83 injection patterns. The other 36 are
|
||||
listed by key under `coverage.uncoveredPatterns` — a swap that breaks one of
|
||||
those is not caught by the reference run, only by the pattern dump. That is an
|
||||
honest limit of the corpus, not a bug in the gate; closing it means growing
|
||||
the conformance corpus (Phase 5 step 5).
|
||||
**`coverage` is static reachability, NOT observed coverage.** It probes the 61
|
||||
payload strings against the 83 injection patterns in-process: "if you threw
|
||||
every payload string at every regex, how many would match?" It does **not**
|
||||
measure what the 61 hook invocations evaluated — the `pre-bash-destructive`
|
||||
payloads never reach injection-patterns at all, yet their strings are in the
|
||||
probe set and can mark a pattern reachable. `47/83` is therefore an upper
|
||||
bound on what the corpus could protect, not a measurement of what it did.
|
||||
|
||||
The block is named `kind: "static-reachability"` and carries that caveat in a
|
||||
`note` field; the gate asserts both, so the honest label cannot be dropped
|
||||
quietly.
|
||||
|
||||
Two gaps follow:
|
||||
|
||||
- **36 unreachable patterns**, listed by key under
|
||||
`coverage.unreachablePatterns`. A swap that breaks one of those is caught by
|
||||
the pattern dump only. Closing it means growing the conformance corpus
|
||||
(Phase 5 step 5).
|
||||
- **The four OWASP maps have no behavioural coverage at all.** They are
|
||||
scanner-side and no hook in this corpus reaches them. They were the reason
|
||||
the dump was widened to table records, and the digest is their *only*
|
||||
protection — do not read the reference run as backing them.
|
||||
|
||||
`corpusContains` says a payload *contains* a homoglyph or is altered by
|
||||
`normalizeForScan`. It does not say the run folded one.
|
||||
|
|
|
|||
|
|
@ -2,9 +2,11 @@
|
|||
"artifact": "golden-reference-run",
|
||||
"schema": 1,
|
||||
"coverage": {
|
||||
"kind": "static-reachability",
|
||||
"note": "Payload strings probed against patterns in-process. NOT a measurement of what the 61 hook invocations evaluated — pre-bash-destructive payloads never reach injection-patterns yet are in the probe set. The four OWASP maps are scanner-side and have no behavioural coverage here at all; their only protection is the table digest.",
|
||||
"patternsTotal": 83,
|
||||
"patternsExercised": 47,
|
||||
"uncoveredPatterns": [
|
||||
"patternsReachable": 47,
|
||||
"unreachablePatterns": [
|
||||
"CRITICAL_PATTERNS[5]",
|
||||
"CRITICAL_PATTERNS[8]",
|
||||
"CRITICAL_PATTERNS[15]",
|
||||
|
|
@ -42,9 +44,9 @@
|
|||
"HYBRID_PATTERNS[1]",
|
||||
"HYBRID_PATTERNS[7]"
|
||||
],
|
||||
"tablesExercised": {
|
||||
"HOMOGLYPH_MAP": true,
|
||||
"normalizeForScan": true
|
||||
"corpusContains": {
|
||||
"homoglyphChars": true,
|
||||
"payloadsAlteredByNormalize": 5
|
||||
}
|
||||
},
|
||||
"summary": {
|
||||
|
|
|
|||
|
|
@ -204,22 +204,31 @@ describe('golden baseline — reference run (61 payloads, real entry points)', (
|
|||
}
|
||||
});
|
||||
|
||||
it('records pattern coverage and fails if it shrinks', async () => {
|
||||
it('records static pattern reachability, labelled as such, and fails if it shrinks', async () => {
|
||||
const run = JSON.parse(readArtifact(REFERENCE_ARTIFACT));
|
||||
assert.ok(run.coverage, 'reference run carries no coverage block');
|
||||
|
||||
// The label is load-bearing. This block probes payload strings against
|
||||
// patterns in-process; it does NOT measure what the 61 hook invocations
|
||||
// evaluated. A future session quoting it as observed coverage would be
|
||||
// quoting a number that measures something else — pinned here so the
|
||||
// honest name cannot be quietly dropped.
|
||||
assert.equal(run.coverage.kind, 'static-reachability');
|
||||
assert.match(run.coverage.note, /NOT a measurement/);
|
||||
|
||||
assert.ok(
|
||||
run.coverage.patternsExercised > 0,
|
||||
'coverage says zero patterns fired — a byte-identical gate over a ' +
|
||||
'corpus that exercises nothing proves nothing'
|
||||
run.coverage.patternsReachable > 0,
|
||||
'zero patterns reachable — a byte-identical gate over a corpus that ' +
|
||||
'reaches nothing proves nothing'
|
||||
);
|
||||
assert.ok(
|
||||
Array.isArray(run.coverage.uncoveredPatterns),
|
||||
'uncovered patterns must be listed explicitly, not silently dropped'
|
||||
Array.isArray(run.coverage.unreachablePatterns),
|
||||
'unreachable patterns must be listed explicitly, not silently dropped'
|
||||
);
|
||||
assert.equal(
|
||||
run.coverage.patternsExercised + run.coverage.uncoveredPatterns.length,
|
||||
run.coverage.patternsReachable + run.coverage.unreachablePatterns.length,
|
||||
run.coverage.patternsTotal,
|
||||
'coverage accounting does not add up'
|
||||
'reachability accounting does not add up'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue