config-audit/tests/scanners/cache-prefix.test.mjs
Kjell Tore Guttormsen 7a794b47eb fix(scanners)!: a finding ID names the check, not the emission (M-BUG-28)
BREAKING CHANGE: the {NNN} in CA-{SCANNER}-{NNN} identifies the check that
produced the finding. It used to be the finding's position in that scanner's
output for that run, which made it unstable across CONFIGURATIONS, not just
across releases as STATE framed it. Measured on two fixtures: "No custom
subagents" was CA-GAP-007 on minimal-project and CA-GAP-004 on healthy-project.
A user who fixed an unrelated earlier gap silently renumbered every later one,
so a .config-audit-ignore pin retargeted to a neighbouring finding with no
version change at all.

Second measured arm: README already documented the opposite scheme. It and the
scanner headers describe ~20 numbers as check codes (CA-SKL-003 = oversized
body, CA-PLH-015 = folder shadowing, CA-TOK-006 = schema deferral), and the
counter could only produce those in the all-fire case -- source-order positions
are 4, 3 and 8. The documentation described the scheme; the implementation was
what was wrong. Every published number is preserved by construction and pinned
exhaustively in tests/lib/finding-codes.test.mjs.

scanners/lib/finding-codes.mjs is the single authority. Every finding() call
passes a `code`; an undeclared or missing one THROWS. No counter fallback --
that would reproduce D1's findGapId -> 'unknown' silent degradation and let a
half-converted scanner ship IDs that look valid. findingCounter/resetCounter
are deleted outright, not left as no-ops. Retirement is now a mechanism:
RETIRED_CODES tombstones a withdrawn key so its number is never reissued,
seeded with GAP t3_8 -- the D1 removal that opened this chunk.

IDs are consequently NOT unique per finding: one check failing in three files
emits three findings sharing an ID. That inverts which consumer is correct, so
every f.id/findingId site was classified before the change. diff-engine and
most of fix-engine already keyed on scanner+title+file (drift was never lying);
fix-engine's verification did not, and keyed on the ID alone -- fixing one of
two sibling instances marked both fixed, and the untouched one, still present
in the re-scan, was reported as a REGRESSION. Red test first, then keyed on
(findingId, file), which both planFixes and applyFixes already carry.
plugin-health's crossIds Set was measured and is a clean negative: cross
findings are allFindings.slice(crossPluginStart) and codes 18/19 are emitted
only in that tail, so the partition holds by construction.

unknownSuppressions() reports a pin that names no declared check, in the
--output-file payload (ux-rules rule 2 -- a stderr-only warning is invisible to
the commands) and only when one exists, so a clean config is byte-identical.
That is what makes the break safe: a stale pin goes loud instead of dying quiet.

Frozen tests/snapshots/v5.0.0/ untouched on disk. IDs are masked out of that
comparison (mask-finding-ids.mjs) rather than re-derived -- re-deriving
positional IDs would assert the retired scheme against itself, and #58's
isGapEntry off-by-one is the measured example of that misfiring. The dead
re-derivation is removed from strip-retired-gap.mjs. default-output snapshots
re-approved after confirming the diff is IDs and nothing else.

Guards, each seen red against its own defect: a missing code (scanner errors
out mid-sweep), an orphan declaration, a resurrected retired key, and a
documented ID naming no check. The sweep asserts the union across all 16
scanners, never per scanner -- a per-scanner assertion goes green on a partial
conversion.

Fasit written before implementation: docs/mbug28-id-semantics-fasit.local.md,
including one correction made before running (CML has 12 checks over 13 call
sites -- the anchored and calibrated char-budget arms are one check, which a
repeated-title sweep found and my call-site count had missed).

Suite 1535 -> 1573, 0 failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MyqCQKK2ornJ1jFWwqx17E
2026-08-09 23:26:36 +02:00

162 lines
8.4 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { scan } from '../../scanners/cache-prefix-scanner.mjs';
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
const __dirname = fileURLToPath(new URL('.', import.meta.url));
const FIXTURES = resolve(__dirname, '../fixtures');
async function runScanner(fixtureName) {
const path = resolve(FIXTURES, fixtureName);
const discovery = await discoverConfigFiles(path);
return scan(path, discovery);
}
describe('CPS scanner — basic structure', () => {
it('reports scanner prefix CPS', async () => {
const result = await runScanner('volatile-mid-section/volatile-line-60');
assert.equal(result.scanner, 'CPS');
});
it('finding IDs match CA-CPS-NNN pattern', async () => {
const result = await runScanner('volatile-mid-section/volatile-line-60');
for (const f of result.findings) {
assert.match(f.id, /^CA-CPS-\d{3}$/);
}
});
});
describe('CPS scanner — volatile content within cached prefix', () => {
it('flags !git log at line 60 (medium severity)', async () => {
const result = await runScanner('volatile-mid-section/volatile-line-60');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.ok(f, `expected volatile-prefix finding; got: ${result.findings.map(x => x.title).join(' | ')}`);
assert.equal(f.severity, 'medium', `expected medium, got ${f.severity}`);
assert.match(String(f.evidence || ''), /line 60/);
assert.match(String(f.evidence || ''), /shell-exec/i);
});
});
describe('CPS scanner — volatile content beyond cache window', () => {
it('does NOT flag volatility at line 200+ (outside 150-line window)', async () => {
const result = await runScanner('volatile-mid-section/volatile-line-200');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(f, undefined,
`expected no finding for line-200 fixture; got: ${f?.title}`);
});
});
describe('CPS scanner — does not duplicate TOK Pattern A territory', () => {
it('volatility at lines 130 is left for TOK Pattern A (no CPS finding)', async () => {
// The opus-47/cache-breaking fixture has volatile content at the very top.
// CPS skips lines 130 to avoid duplicating Pattern A's territory.
const result = await runScanner('opus-47/cache-breaking');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(f, undefined,
`expected no CPS finding when volatility is only in lines 130 (Pattern A's range)`);
});
});
describe('CPS scanner — volatile content in @imported files (v5.10 B6)', () => {
it('flags volatile content inside an @imported file even when the root file is stable', async () => {
const result = await runScanner('import-volatile/positive');
const f = result.findings.find(x => /volatile content in @imported file/i.test(x.title || ''));
assert.ok(f, `expected @import-volatile finding; got: ${result.findings.map(x => x.title).join(' | ')}`);
assert.equal(f.severity, 'medium', `expected medium, got ${f.severity}`);
assert.match(String(f.evidence || ''), /imported by/i);
assert.match(String(f.evidence || ''), /BUILD_TIMESTAMP|git log|VAR substitution|shell-exec/i);
assert.equal(f.category, 'token-efficiency');
});
it('does NOT emit the in-file CPS finding when the importing file itself is stable', async () => {
// Root CLAUDE.md carries no volatile lines; only the @imported file does.
const result = await runScanner('import-volatile/positive');
const inFile = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(inFile, undefined, 'a byte-stable importing file must not trip the in-file finding');
});
it('does NOT flag a clean @imported file', async () => {
const result = await runScanner('import-volatile/clean');
const f = result.findings.find(x => /volatile content in @imported file/i.test(x.title || ''));
assert.equal(f, undefined, `expected no finding for a clean import; got: ${f?.title}`);
});
});
describe('CPS scanner — CC-stable vars are not cache-busters (M-BUG-7)', () => {
it('does NOT flag ${CLAUDE_PLUGIN_ROOT} / ${CLAUDE_PROJECT_DIR} in prose', async () => {
// These are CC-provided path substitutions that resolve to a stable value
// every turn — referencing them in CLAUDE.md prose is not a cache-buster.
const result = await runScanner('cps-stable-vars/whitelisted');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(f, undefined,
`CC-stable path vars must not trip CPS; got: ${f?.evidence}`);
});
it('STILL flags a genuine non-CC ${VAR} substitution (whitelist is selective)', async () => {
const result = await runScanner('cps-stable-vars/non-whitelisted');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.ok(f, `expected a finding for a genuine runtime var; got none`);
assert.match(String(f.evidence || ''), /line 40/);
assert.match(String(f.evidence || ''), /\$\{VAR\} substitution/i);
});
});
describe('CPS scanner — volatility inside fenced code blocks is documentation (M-BUG-7)', () => {
it('does NOT flag volatile-looking lines inside a ```fence```', async () => {
// Content inside a fenced code block is illustrative, byte-stable literal
// text — not runtime volatility — so it must not break the cached prefix.
const result = await runScanner('cps-fenced/inside-fence');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(f, undefined,
`fenced code content must not trip CPS; got: ${f?.evidence}`);
});
it('STILL flags genuine volatility in live prose outside the fence', async () => {
const result = await runScanner('cps-fenced/mixed');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.ok(f, `expected a finding for the out-of-fence volatile line; got none`);
assert.match(String(f.evidence || ''), /line 50/);
assert.match(String(f.evidence || ''), /shell-exec/i);
assert.doesNotMatch(String(f.evidence || ''), /line 3[567]/,
'fenced lines 3537 must not appear in the evidence');
});
});
describe('CPS scanner — volatility inside inline code is documentation (M-BUG-7)', () => {
it('does NOT flag a {date} placeholder shown inside `inline code`', async () => {
// A filename template like `.claude/plans/run-{date}.md` in backticks is
// literal documentation text — byte-stable, not a runtime substitution.
const result = await runScanner('cps-inline-code/pure');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.equal(f, undefined,
`inline-code documentation must not trip CPS; got: ${f?.evidence}`);
});
it('STILL flags a live ${VAR} that sits outside backticks', async () => {
const result = await runScanner('cps-inline-code/mixed');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.ok(f, `expected a finding for the out-of-backtick volatile line; got none`);
assert.match(String(f.evidence || ''), /line 50/);
assert.doesNotMatch(String(f.evidence || ''), /line 40/,
'the backticked {date} on line 40 must not appear in the evidence');
});
});
describe('CPS scanner — orchestrator wiring', () => {
it('CPS appears in scan-orchestrator scanner list', async () => {
const orch = await import('../../scanners/scan-orchestrator.mjs');
const path = resolve(FIXTURES, 'volatile-mid-section/volatile-line-60');
const env = await orch.runAllScanners(path, { filterFixtures: false });
const cps = env.scanners.find(r => r.scanner === 'CPS');
assert.ok(cps, `expected CPS in orchestrator results; got: ${env.scanners.map(r => r.scanner).join(', ')}`);
});
it('CPS findings carry the token-efficiency category', async () => {
const result = await runScanner('volatile-mid-section/volatile-line-60');
const f = result.findings.find(x => /volatile content inside cached prefix/i.test(x.title || ''));
assert.ok(f);
assert.equal(f.category, 'token-efficiency');
});
});