fix(scanners)!: a finding ID names the check, not the emission (M-BUG-28)
BREAKING CHANGE: the {NNN} in CA-{SCANNER}-{NNN} identifies the check that
produced the finding. It used to be the finding's position in that scanner's
output for that run, which made it unstable across CONFIGURATIONS, not just
across releases as STATE framed it. Measured on two fixtures: "No custom
subagents" was CA-GAP-007 on minimal-project and CA-GAP-004 on healthy-project.
A user who fixed an unrelated earlier gap silently renumbered every later one,
so a .config-audit-ignore pin retargeted to a neighbouring finding with no
version change at all.
Second measured arm: README already documented the opposite scheme. It and the
scanner headers describe ~20 numbers as check codes (CA-SKL-003 = oversized
body, CA-PLH-015 = folder shadowing, CA-TOK-006 = schema deferral), and the
counter could only produce those in the all-fire case -- source-order positions
are 4, 3 and 8. The documentation described the scheme; the implementation was
what was wrong. Every published number is preserved by construction and pinned
exhaustively in tests/lib/finding-codes.test.mjs.
scanners/lib/finding-codes.mjs is the single authority. Every finding() call
passes a `code`; an undeclared or missing one THROWS. No counter fallback --
that would reproduce D1's findGapId -> 'unknown' silent degradation and let a
half-converted scanner ship IDs that look valid. findingCounter/resetCounter
are deleted outright, not left as no-ops. Retirement is now a mechanism:
RETIRED_CODES tombstones a withdrawn key so its number is never reissued,
seeded with GAP t3_8 -- the D1 removal that opened this chunk.
IDs are consequently NOT unique per finding: one check failing in three files
emits three findings sharing an ID. That inverts which consumer is correct, so
every f.id/findingId site was classified before the change. diff-engine and
most of fix-engine already keyed on scanner+title+file (drift was never lying);
fix-engine's verification did not, and keyed on the ID alone -- fixing one of
two sibling instances marked both fixed, and the untouched one, still present
in the re-scan, was reported as a REGRESSION. Red test first, then keyed on
(findingId, file), which both planFixes and applyFixes already carry.
plugin-health's crossIds Set was measured and is a clean negative: cross
findings are allFindings.slice(crossPluginStart) and codes 18/19 are emitted
only in that tail, so the partition holds by construction.
unknownSuppressions() reports a pin that names no declared check, in the
--output-file payload (ux-rules rule 2 -- a stderr-only warning is invisible to
the commands) and only when one exists, so a clean config is byte-identical.
That is what makes the break safe: a stale pin goes loud instead of dying quiet.
Frozen tests/snapshots/v5.0.0/ untouched on disk. IDs are masked out of that
comparison (mask-finding-ids.mjs) rather than re-derived -- re-deriving
positional IDs would assert the retired scheme against itself, and #58's
isGapEntry off-by-one is the measured example of that misfiring. The dead
re-derivation is removed from strip-retired-gap.mjs. default-output snapshots
re-approved after confirming the diff is IDs and nothing else.
Guards, each seen red against its own defect: a missing code (scanner errors
out mid-sweep), an orphan declaration, a resurrected retired key, and a
documented ID naming no check. The sweep asserts the union across all 16
scanners, never per scanner -- a per-scanner assertion goes green on a partial
conversion.
Fasit written before implementation: docs/mbug28-id-semantics-fasit.local.md,
including one correction made before running (CML has 12 checks over 13 call
sites -- the anchored and calibrated char-budget arms are one check, which a
repeated-title sweep found and my call-site count had missed).
Suite 1535 -> 1573, 0 failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MyqCQKK2ornJ1jFWwqx17E
This commit is contained in:
parent
4027cdcf54
commit
7a794b47eb
62 changed files with 1092 additions and 279 deletions
|
|
@ -3,7 +3,6 @@ import assert from 'node:assert/strict';
|
|||
import { join } from 'node:path';
|
||||
import { mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/agent-listing-scanner.mjs';
|
||||
import { PER_AGENT_DESC_SOFT_CAP } from '../../scanners/lib/agent-listing-budget.mjs';
|
||||
|
||||
|
|
@ -33,7 +32,6 @@ function uniqueDir(suffix) {
|
|||
* override HOME, run, restore — never rely on the developer's real ~/.claude.
|
||||
*/
|
||||
async function runScannerWithHome(home) {
|
||||
resetCounter();
|
||||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import { describe, it } from 'node:test';
|
|||
import assert from 'node:assert/strict';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/cache-prefix-scanner.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
|
||||
|
|
@ -10,7 +9,6 @@ const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
|||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
||||
async function runScanner(fixtureName) {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, fixtureName);
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
return scan(path, discovery);
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/claude-md-linter.mjs';
|
||||
|
||||
|
|
@ -14,7 +13,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('CML scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -52,7 +50,6 @@ describe('CML scanner — healthy project', () => {
|
|||
describe('CML scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -81,7 +78,6 @@ describe('CML scanner — broken project', () => {
|
|||
describe('CML scanner — broken project: 200-tier stays MEDIUM (regression lock)', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -101,7 +97,6 @@ describe('CML scanner — large cascade (>500 lines): reframed, not absolute-adh
|
|||
// HIGH@500 + "significantly reduce adherence" claim is now-wrong.
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'large-cascade'));
|
||||
result = await scan(resolve(FIXTURES, 'large-cascade'), discovery);
|
||||
});
|
||||
|
|
@ -137,7 +132,6 @@ describe('CML scanner — char budget mirrors CC startup warning (CC 2.1.169)',
|
|||
// 200-line count, so it isolates the char-budget check from the line check.
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'large-claude-chars'));
|
||||
result = await scan(resolve(FIXTURES, 'large-claude-chars'), discovery);
|
||||
});
|
||||
|
|
@ -178,7 +172,6 @@ describe('CML scanner — context-window calibration (B8)', () => {
|
|||
r.findings.find((f) => /performance-warning threshold/i.test(f.title || ''));
|
||||
|
||||
async function scanWithCtx(contextWindow) {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(FIXTURE);
|
||||
return scan(FIXTURE, discovery, { contextWindow });
|
||||
}
|
||||
|
|
@ -197,7 +190,6 @@ describe('CML scanner — context-window calibration (B8)', () => {
|
|||
});
|
||||
|
||||
it('no opts (default) is unchanged: fires medium at the 40k anchor', async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(FIXTURE);
|
||||
const result = await scan(FIXTURE, discovery);
|
||||
assert.equal(charFinding(result)?.severity, 'medium', 'default must stay byte-stable: medium');
|
||||
|
|
@ -210,7 +202,6 @@ describe('CML scanner — large-by-lines but under the char budget (no false cha
|
|||
// proving the check keys on chars, not raw size or line count.
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'large-cascade'));
|
||||
result = await scan(resolve(FIXTURES, 'large-cascade'), discovery);
|
||||
});
|
||||
|
|
@ -224,7 +215,6 @@ describe('CML scanner — large-by-lines but under the char budget (no false cha
|
|||
describe('CML scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
@ -243,7 +233,6 @@ describe('CML scanner — empty project', () => {
|
|||
describe('CML scanner — minimal project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'minimal-project'));
|
||||
result = await scan(resolve(FIXTURES, 'minimal-project'), discovery);
|
||||
});
|
||||
|
|
@ -262,7 +251,6 @@ describe('CML — nested CLAUDE.md not re-injected after compaction (A)', () =>
|
|||
let result;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-cml-nested-'));
|
||||
await mkdir(join(tmpRoot, 'src'), { recursive: true });
|
||||
await writeFile(join(tmpRoot, 'CLAUDE.md'), '# Root\n\nProject overview goes here.\n', 'utf8');
|
||||
|
|
|
|||
|
|
@ -8,6 +8,7 @@ import { readFile, writeFile, unlink, mkdir, access } from 'node:fs/promises';
|
|||
import { hermeticEnv, HERMETIC_HOME } from '../helpers/hermetic-home.mjs';
|
||||
import { stripHotspotLoadPattern } from '../helpers/strip-hotspot-load-pattern.mjs';
|
||||
import { stripRetiredGap } from '../helpers/strip-retired-gap.mjs';
|
||||
import { maskFindingIds } from '../helpers/mask-finding-ids.mjs';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
|
@ -44,7 +45,7 @@ async function runCli(cliPath, args, env = {}) {
|
|||
function normalizeTokenHotspotsPayload(p) {
|
||||
const out = JSON.parse(JSON.stringify(p));
|
||||
out.duration_ms = 0;
|
||||
return stripRetiredGap(stripHotspotLoadPattern(out));
|
||||
return maskFindingIds(stripRetiredGap(stripHotspotLoadPattern(out)));
|
||||
}
|
||||
|
||||
function normalizeManifestOutput(o) {
|
||||
|
|
@ -77,7 +78,7 @@ function normalizePluginHealthOutput(o) {
|
|||
|
||||
function normalizeDriftOutput(o) {
|
||||
// Drift result has no time fields; just round-trip through JSON.
|
||||
return stripRetiredGap(JSON.parse(JSON.stringify(o)));
|
||||
return maskFindingIds(stripRetiredGap(JSON.parse(JSON.stringify(o))));
|
||||
}
|
||||
|
||||
// ============================================================================
|
||||
|
|
@ -319,14 +320,14 @@ describe('fix-cli humanizer (Step 7)', () => {
|
|||
const { stdout } = await runCli(CLI, [FIXTURE, '--json']);
|
||||
const actual = JSON.parse(stdout);
|
||||
const expected = JSON.parse(await readFile(SNAPSHOT, 'utf-8'));
|
||||
assert.deepStrictEqual(stripRetiredGap(actual), stripRetiredGap(expected));
|
||||
assert.deepStrictEqual(maskFindingIds(stripRetiredGap(actual)), maskFindingIds(stripRetiredGap(expected)));
|
||||
});
|
||||
|
||||
it('--raw: stdout JSON byte-equal v5.0.0 snapshot', async () => {
|
||||
const { stdout } = await runCli(CLI, [FIXTURE, '--raw']);
|
||||
const actual = JSON.parse(stdout);
|
||||
const expected = JSON.parse(await readFile(SNAPSHOT, 'utf-8'));
|
||||
assert.deepStrictEqual(stripRetiredGap(actual), stripRetiredGap(expected));
|
||||
assert.deepStrictEqual(maskFindingIds(stripRetiredGap(actual)), maskFindingIds(stripRetiredGap(expected)));
|
||||
});
|
||||
|
||||
it('default mode stderr differs from --raw stderr when findings have humanizer translations', async () => {
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/collision-scanner.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
|
|
@ -20,7 +19,6 @@ function uniqueDir(suffix) {
|
|||
* Tests must override HOME, run, and restore — never rely on user-state.
|
||||
*/
|
||||
async function runScannerWithHome(home) {
|
||||
resetCounter();
|
||||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join, sep } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/conflict-detector.mjs';
|
||||
|
||||
|
|
@ -14,7 +13,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('CNF scanner — conflict project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'conflict-project'));
|
||||
result = await scan(resolve(FIXTURES, 'conflict-project'), discovery);
|
||||
});
|
||||
|
|
@ -77,7 +75,6 @@ describe('CNF scanner — conflict project', () => {
|
|||
describe('CNF scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -94,7 +91,6 @@ describe('CNF scanner — healthy project', () => {
|
|||
describe('CNF scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
@ -111,7 +107,6 @@ describe('CNF scanner — empty project', () => {
|
|||
describe('CNF scanner — minimal project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'minimal-project'));
|
||||
result = await scan(resolve(FIXTURES, 'minimal-project'), discovery);
|
||||
});
|
||||
|
|
@ -132,7 +127,6 @@ describe('CNF scanner — param-qualified cross-scope conflicts', () => {
|
|||
// Agent: deny model:opus vs allow model:sonnet → disjoint → NO conflict.
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'param-conflict-project'));
|
||||
result = await scan(resolve(FIXTURES, 'param-conflict-project'), discovery);
|
||||
});
|
||||
|
|
@ -186,11 +180,9 @@ describe('CNF scanner — cache exclusion drops duplicate-hook count (B3)', () =
|
|||
},
|
||||
}));
|
||||
|
||||
resetCounter();
|
||||
const dIncl = await discoverConfigFiles(dir);
|
||||
includeCount = (await scan(dir, dIncl)).findings.filter(f => f.title.includes('Duplicate hook')).length;
|
||||
|
||||
resetCounter();
|
||||
const dExcl = await discoverConfigFiles(dir, { excludeCache: true });
|
||||
excludeCount = (await scan(dir, dExcl)).findings.filter(f => f.title.includes('Duplicate hook')).length;
|
||||
});
|
||||
|
|
@ -255,7 +247,6 @@ describe('CNF scanner — excludes plugin-bundled configs (M-BUG-2)', () => {
|
|||
permissions: { allow: ['Bash(rm:*)'] },
|
||||
}));
|
||||
|
||||
resetCounter();
|
||||
discovery = await discoverConfigFiles(dir);
|
||||
result = await scan(dir, discovery);
|
||||
});
|
||||
|
|
@ -289,7 +280,6 @@ describe('CNF scanner — genuine live conflict still flagged (M-BUG-2 guard)',
|
|||
await mkdir(claude, { recursive: true });
|
||||
await writeFile(join(claude, 'settings.json'), JSON.stringify({ model: 'opus' }));
|
||||
await writeFile(join(claude, 'settings.local.json'), JSON.stringify({ model: 'haiku' }));
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(dir);
|
||||
result = await scan(dir, discovery);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import { describe, it } from 'node:test';
|
|||
import assert from 'node:assert/strict';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/disabled-in-schema-scanner.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
|
||||
|
|
@ -10,7 +9,6 @@ const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
|||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
||||
async function runScanner(fixtureName) {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, fixtureName);
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
return scan(path, discovery);
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join, dirname } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan, opportunitySummary, bundledSkillsLeverFinding, cliOverMcpLeverFinding, filterHookLeverFinding, GAP_CHECKS } from '../../scanners/feature-gap-scanner.mjs';
|
||||
import { TITLE_TO_ID as GAP_TITLE_TO_ID, TIER_COUNTS, TOTAL_DIMENSIONS } from '../../scanners/lib/scoring.mjs';
|
||||
import { TRANSLATIONS } from '../../scanners/lib/humanizer-data.mjs';
|
||||
|
|
@ -23,7 +22,6 @@ async function fixtureDiscovery(name) {
|
|||
describe('GAP scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('healthy-project');
|
||||
// Hermetic HOME: scan() now enumerates active skills via process.env.HOME
|
||||
// (the disableBundledSkills lever check). An empty HOME keeps these fixture
|
||||
|
|
@ -97,7 +95,6 @@ describe('GAP scanner — healthy project', () => {
|
|||
describe('GAP scanner — minimal project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('minimal-project');
|
||||
result = await withHermeticHome(() => scan(resolve(FIXTURES, 'minimal-project'), discovery));
|
||||
});
|
||||
|
|
@ -131,7 +128,6 @@ describe('GAP scanner — minimal project', () => {
|
|||
});
|
||||
|
||||
it('has more findings than healthy project', async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('healthy-project');
|
||||
const healthyResult = await withHermeticHome(() => scan(resolve(FIXTURES, 'healthy-project'), discovery));
|
||||
assert.ok(result.findings.length > healthyResult.findings.length);
|
||||
|
|
@ -141,7 +137,6 @@ describe('GAP scanner — minimal project', () => {
|
|||
describe('GAP scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('empty-project');
|
||||
result = await withHermeticHome(() => scan(resolve(FIXTURES, 'empty-project'), discovery));
|
||||
});
|
||||
|
|
@ -230,7 +225,6 @@ describe('bundledSkillsLeverFinding — pure decision', () => {
|
|||
});
|
||||
|
||||
it('returns a finding when the lever is un-pulled AND the listing is over budget', () => {
|
||||
resetCounter();
|
||||
const f = bundledSkillsLeverFinding({ leverPulled: false, aggregate: overBudget });
|
||||
assert.ok(f, 'expected a finding');
|
||||
assert.match(f.id, /^CA-GAP-\d{3}$/);
|
||||
|
|
@ -267,7 +261,6 @@ describe('GAP scanner — disableBundledSkills lever wiring (HOME-scoped)', () =
|
|||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(project, { includeGlobal: true });
|
||||
const result = await scan(project, discovery);
|
||||
return result;
|
||||
|
|
@ -351,7 +344,6 @@ describe('cliOverMcpLeverFinding (CLI-over-MCP lever, v5.10 B4)', () => {
|
|||
});
|
||||
|
||||
it('fires a low-severity opportunity when MCP schemas are forced upfront', () => {
|
||||
resetCounter();
|
||||
const f = cliOverMcpLeverFinding({
|
||||
assessment: {
|
||||
forcedUpfront: true,
|
||||
|
|
@ -375,7 +367,6 @@ describe('filterHookLeverFinding (filter-before-Claude-reads lever, v5.10 B5)',
|
|||
});
|
||||
|
||||
it('fires an info opportunity when ≥1 chatty hook is detected', () => {
|
||||
resetCounter();
|
||||
const f = filterHookLeverFinding({
|
||||
flaggedHooks: [{ event: 'SessionStart', scriptPath: '/x/hooks/scripts/chatty.sh' }],
|
||||
});
|
||||
|
|
@ -389,7 +380,6 @@ describe('filterHookLeverFinding (filter-before-Claude-reads lever, v5.10 B5)',
|
|||
|
||||
describe('GAP scanner — filter-before lever wiring (chatty hook fixture)', () => {
|
||||
it('emits the filter-before lever when scanning a repo with a chatty hook', async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('hooks-additional-context');
|
||||
const result = await withHermeticHome(() =>
|
||||
scan(resolve(FIXTURES, 'hooks-additional-context'), discovery),
|
||||
|
|
@ -402,7 +392,6 @@ describe('GAP scanner — filter-before lever wiring (chatty hook fixture)', ()
|
|||
});
|
||||
|
||||
it('does NOT emit the lever for a repo with only quiet hooks', async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('hooks-quiet');
|
||||
const result = await withHermeticHome(() =>
|
||||
scan(resolve(FIXTURES, 'hooks-quiet'), discovery),
|
||||
|
|
@ -432,7 +421,6 @@ describe('GAP scanner — test/demo data must not mask real gaps (M-BUG-13)', ()
|
|||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(project, { includeGlobal: true });
|
||||
const result = await scan(project, discovery);
|
||||
return result;
|
||||
|
|
@ -529,7 +517,6 @@ describe('GAP scanner — retired dimensions (D1)', () => {
|
|||
const RETIRED_TITLES = ['No autoMode classifier'];
|
||||
|
||||
it('emits no retired dimension on a fixture that lacks the feature', async () => {
|
||||
resetCounter();
|
||||
const discovery = await fixtureDiscovery('healthy-project');
|
||||
const result = await withHermeticHome(
|
||||
() => scan(resolve(FIXTURES, 'healthy-project'), discovery),
|
||||
|
|
|
|||
82
tests/scanners/finding-code-coverage.test.mjs
Normal file
82
tests/scanners/finding-code-coverage.test.mjs
Normal file
|
|
@ -0,0 +1,82 @@
|
|||
/**
|
||||
* Blanket invariants over the whole scanner set (M-BUG-28).
|
||||
*
|
||||
* Deliberately NOT per-scanner: a per-scanner assertion goes green on a partial
|
||||
* conversion, which is the class #51/#57/#58 kept reproducing. Both directions
|
||||
* are asserted — no finding without a declared code, and no declared code
|
||||
* without a call site (an orphan declaration is as wrong as a missing one).
|
||||
*/
|
||||
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFile, readdir } from 'node:fs/promises';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { runAllScanners } from '../../scanners/scan-orchestrator.mjs';
|
||||
import { FINDING_CODES, allFindingIds } from '../../scanners/lib/finding-codes.mjs';
|
||||
import { GAP_CHECKS } from '../../scanners/feature-gap-scanner.mjs';
|
||||
import { withHermeticHome } from '../helpers/hermetic-home.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
const SCANNERS_DIR = resolve(__dirname, '../../scanners');
|
||||
|
||||
// Fixtures chosen to spread across scanners: broken/conflicting config fires the
|
||||
// validators, healthy config fires the gap dimensions.
|
||||
const SWEEP = [
|
||||
'broken-project',
|
||||
'conflict-project',
|
||||
'healthy-project',
|
||||
'minimal-project',
|
||||
'fixable-project',
|
||||
'large-cascade',
|
||||
];
|
||||
|
||||
describe('every emitted finding carries a declared check code', () => {
|
||||
it('holds across all orchestrated scanners and a spread of fixtures', async () => {
|
||||
const declaredIds = allFindingIds();
|
||||
const offenders = [];
|
||||
const scannerErrors = [];
|
||||
|
||||
for (const fixture of SWEEP) {
|
||||
const env = await withHermeticHome(async () =>
|
||||
runAllScanners(resolve(FIXTURES, fixture), { filterFixtures: false, suppress: false })
|
||||
);
|
||||
|
||||
for (const r of env.scanners || []) {
|
||||
if (r.status === 'error') scannerErrors.push(`${fixture}/${r.scanner}: ${r.error}`);
|
||||
for (const f of r.findings || []) {
|
||||
if (!declaredIds.has(f.id)) offenders.push(`${fixture}/${r.scanner}: ${f.id} — ${f.title}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A scanner that throws returns status:'error' with zero findings, so an
|
||||
// unconverted call site would otherwise hide as "no findings".
|
||||
assert.deepEqual(scannerErrors, [], 'a scanner failed during the sweep');
|
||||
assert.deepEqual(offenders, [], 'a finding carries an ID outside the registry');
|
||||
});
|
||||
});
|
||||
|
||||
describe('every declared code is claimed by a call site', () => {
|
||||
it('finds no orphan declarations', async () => {
|
||||
const files = (await readdir(SCANNERS_DIR)).filter((f) => f.endsWith('.mjs'));
|
||||
const sources = await Promise.all(
|
||||
files.map((f) => readFile(resolve(SCANNERS_DIR, f), 'utf-8'))
|
||||
);
|
||||
const blob = sources.join('\n');
|
||||
|
||||
const gapIds = new Set(GAP_CHECKS.map((g) => g.id));
|
||||
const orphans = [];
|
||||
|
||||
for (const [scanner, table] of Object.entries(FINDING_CODES)) {
|
||||
for (const key of Object.keys(table)) {
|
||||
// GAP dimension codes are claimed via `code: gap.id` over GAP_CHECKS.
|
||||
if (scanner === 'GAP' && gapIds.has(key)) continue;
|
||||
if (!blob.includes(`code: '${key}'`)) orphans.push(`${scanner}.${key}`);
|
||||
}
|
||||
}
|
||||
|
||||
assert.deepEqual(orphans, [], 'a declared code is emitted by nothing');
|
||||
});
|
||||
});
|
||||
69
tests/scanners/finding-id-stability.test.mjs
Normal file
69
tests/scanners/finding-id-stability.test.mjs
Normal file
|
|
@ -0,0 +1,69 @@
|
|||
/**
|
||||
* M-BUG-28 — finding IDs identify the CHECK, not the emission position.
|
||||
*
|
||||
* The defect these tests pin (measured 2026-08-09, session #59): `CA-{SCANNER}-{NNN}`
|
||||
* was rendered from a module-global emission counter, so NNN was the finding's
|
||||
* position in that scanner's output for THAT run. Two configurations that both
|
||||
* fail the same check got different IDs for it, and a `.config-audit-ignore`
|
||||
* entry silently retargeted to a neighbouring finding whenever an earlier check
|
||||
* started or stopped firing.
|
||||
*
|
||||
* These assert BEHAVIOUR (run the scanner, read the id), never the mechanism.
|
||||
*/
|
||||
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { scan as gapScan } from '../../scanners/feature-gap-scanner.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { withHermeticHome } from '../helpers/hermetic-home.mjs';
|
||||
import { startScannerRun } from '../helpers/scanner-run.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
||||
async function gapIdsByTitle(fixture) {
|
||||
return withHermeticHome(async () => {
|
||||
// Mirror what scan-orchestrator does before each scanner, so this measures
|
||||
// ONLY config-driven ID drift and not counter carry-over between two scans
|
||||
// in one test process. Becomes a no-op once IDs are check-derived.
|
||||
startScannerRun();
|
||||
const target = resolve(FIXTURES, fixture);
|
||||
const discovery = await discoverConfigFiles(target);
|
||||
const result = await gapScan(target, discovery);
|
||||
const map = new Map();
|
||||
for (const f of result.findings) map.set(f.title, f.id);
|
||||
return map;
|
||||
});
|
||||
}
|
||||
|
||||
describe('finding IDs are stable across configurations (M-BUG-28)', () => {
|
||||
it('gives a check the same ID no matter which other checks fired', async () => {
|
||||
// minimal-project fails more early checks than healthy-project, so under the
|
||||
// emission-counter scheme every shared check downstream of the first
|
||||
// divergence was renumbered.
|
||||
const minimal = await gapIdsByTitle('minimal-project');
|
||||
const healthy = await gapIdsByTitle('healthy-project');
|
||||
|
||||
const shared = [...minimal.keys()].filter((t) => healthy.has(t));
|
||||
assert.ok(shared.length >= 10, `expected overlapping checks, got ${shared.length}`);
|
||||
|
||||
const drifted = shared
|
||||
.filter((t) => minimal.get(t) !== healthy.get(t))
|
||||
.map((t) => `${t}: minimal=${minimal.get(t)} healthy=${healthy.get(t)}`);
|
||||
|
||||
assert.deepEqual(drifted, [], 'a check must carry one ID regardless of config');
|
||||
});
|
||||
|
||||
it('keeps a suppression pinned to the check the user pinned', async () => {
|
||||
// The user-visible consequence: pin the ID you see, fix an unrelated earlier
|
||||
// gap, and the pin must still name the same check.
|
||||
const minimal = await gapIdsByTitle('minimal-project');
|
||||
const healthy = await gapIdsByTitle('healthy-project');
|
||||
|
||||
const pinned = 'No custom subagents';
|
||||
assert.ok(minimal.has(pinned) && healthy.has(pinned), 'fixture drifted — check still fires?');
|
||||
assert.equal(minimal.get(pinned), healthy.get(pinned));
|
||||
});
|
||||
});
|
||||
|
|
@ -5,7 +5,6 @@ import { fileURLToPath } from 'node:url';
|
|||
import { cp, rm, readFile, writeFile, stat } from 'node:fs/promises';
|
||||
import { mkdirSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { runAllScanners } from '../../scanners/scan-orchestrator.mjs';
|
||||
import { planFixes, applyFixes, verifyFixes, FIX_TYPES } from '../../scanners/fix-engine.mjs';
|
||||
|
|
@ -30,7 +29,6 @@ describe('planFixes', () => {
|
|||
let envelope;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
envelope = await runAllScanners(FIXABLE);
|
||||
});
|
||||
|
||||
|
|
@ -159,7 +157,6 @@ describe('applyFixes dry-run', () => {
|
|||
let envelope;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
envelope = await runAllScanners(FIXABLE);
|
||||
});
|
||||
|
||||
|
|
@ -190,7 +187,6 @@ describe('applyFixes on tmp copy', () => {
|
|||
|
||||
beforeEach(async () => {
|
||||
tmpDir = await createTmpCopy();
|
||||
resetCounter();
|
||||
envelope = await runAllScanners(tmpDir);
|
||||
});
|
||||
|
||||
|
|
@ -282,7 +278,6 @@ describe('applyFixes on tmp copy', () => {
|
|||
before.effortLevel = raw;
|
||||
await writeFile(settingsPath, `${JSON.stringify(before, null, 2)}\n`, 'utf-8');
|
||||
|
||||
resetCounter();
|
||||
const env = await runAllScanners(dir);
|
||||
const { fixes } = planFixes(env);
|
||||
const effortFix = fixes.filter(f => f.key === 'effortLevel');
|
||||
|
|
@ -360,7 +355,6 @@ describe('verifyFixes', () => {
|
|||
});
|
||||
|
||||
it('confirms fixed findings are gone', async () => {
|
||||
resetCounter();
|
||||
const envelope = await runAllScanners(tmpDir);
|
||||
const { fixes } = planFixes(envelope);
|
||||
|
||||
|
|
@ -383,7 +377,6 @@ describe('planFixes ordering (M-BUG-29)', () => {
|
|||
// Both defects at once: undocumented `globs:` AND a non-.md extension.
|
||||
await writeFile(join(rulesDir, 'both.txt'), '---\nglobs: "**/*.ts"\n---\n\nBody.\n');
|
||||
|
||||
resetCounter();
|
||||
const envelope = await runAllScanners(dir, { includeGlobal: false });
|
||||
const { fixes } = planFixes(envelope);
|
||||
|
||||
|
|
@ -419,7 +412,6 @@ describe('verifyFixes scope (F5)', () => {
|
|||
const originalHome = process.env.HOME;
|
||||
process.env.HOME = fakeHome;
|
||||
try {
|
||||
resetCounter();
|
||||
const envelope = await runAllScanners(dir, { includeGlobal: true });
|
||||
const globalOnly = envelope.scanners
|
||||
.flatMap((s) => s.findings)
|
||||
|
|
|
|||
77
tests/scanners/fix-verify-instance-identity.test.mjs
Normal file
77
tests/scanners/fix-verify-instance-identity.test.mjs
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
/**
|
||||
* Fix verification must identify a finding INSTANCE, not just its check
|
||||
* (M-BUG-28, prediction 7).
|
||||
*
|
||||
* A finding ID names the check, so one check failing in two files yields two
|
||||
* findings sharing an ID. If verification keys on the ID alone, fixing one of
|
||||
* them marks both as fixed — and the untouched one, still present in the
|
||||
* re-scan, is then reported as a REGRESSION. The user is told a fix came back
|
||||
* undone when it was never applied.
|
||||
*
|
||||
* `(findingId, file)` is the instance key, and both `planFixes` and `applyFixes`
|
||||
* already carry `file` next to `findingId`.
|
||||
*/
|
||||
|
||||
import { describe, it, before, after } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { runAllScanners } from '../../scanners/scan-orchestrator.mjs';
|
||||
import { planFixes, applyFixes, verifyFixes } from '../../scanners/fix-engine.mjs';
|
||||
|
||||
const RULE = (name) => `---
|
||||
globs:
|
||||
- "src/**/*.ts"
|
||||
---
|
||||
|
||||
# ${name}
|
||||
|
||||
A path-scoped rule that uses the undocumented \`globs:\` key.
|
||||
`;
|
||||
|
||||
describe('fix verification distinguishes two instances of one check', () => {
|
||||
let dir;
|
||||
|
||||
before(async () => {
|
||||
dir = await mkdtemp(join(tmpdir(), 'ca-mbug28-'));
|
||||
await mkdir(join(dir, '.claude', 'rules'), { recursive: true });
|
||||
await writeFile(join(dir, 'CLAUDE.md'), '# Project\n\nInstructions.\n');
|
||||
await writeFile(join(dir, '.claude', 'rules', 'alpha.md'), RULE('Alpha'));
|
||||
await writeFile(join(dir, '.claude', 'rules', 'beta.md'), RULE('Beta'));
|
||||
});
|
||||
|
||||
after(async () => {
|
||||
if (dir) await rm(dir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('does not report the untouched instance as a regression', async () => {
|
||||
const envelope = await runAllScanners(dir, { filterFixtures: false, suppress: false });
|
||||
|
||||
const globsFindings = envelope.scanners
|
||||
.flatMap((s) => s.findings)
|
||||
.filter((f) => f.title === 'Rule uses "globs" instead of documented "paths"');
|
||||
|
||||
assert.equal(globsFindings.length, 2, 'fixture must produce one finding per rule file');
|
||||
assert.equal(
|
||||
globsFindings[0].id,
|
||||
globsFindings[1].id,
|
||||
'both instances name the same check — that is the scheme, not a defect'
|
||||
);
|
||||
|
||||
const { fixes } = planFixes(envelope);
|
||||
const alphaFix = fixes.filter((p) => p.file && p.file.endsWith('alpha.md'));
|
||||
assert.equal(alphaFix.length, 1, 'expected exactly one fix plan for alpha.md');
|
||||
|
||||
const backupDir = join(dir, '.backups');
|
||||
await mkdir(backupDir, { recursive: true });
|
||||
const { applied } = await applyFixes(alphaFix, { backupDir });
|
||||
assert.equal(applied.length, 1);
|
||||
|
||||
const { verified, regressions } = await verifyFixes(envelope, applied);
|
||||
|
||||
// beta.md was never fixed. It is still open — which is not a regression.
|
||||
assert.deepEqual(regressions, [], 'an unfixed sibling instance must not count as a regression');
|
||||
assert.equal(verified.length, 1, 'exactly the fixed instance is verified');
|
||||
});
|
||||
});
|
||||
|
|
@ -4,9 +4,9 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/hook-validator.mjs';
|
||||
import { findingId } from '../../scanners/lib/finding-codes.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
|
@ -14,7 +14,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('HKV scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -42,7 +41,6 @@ describe('HKV scanner — healthy project', () => {
|
|||
describe('HKV scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -54,8 +52,7 @@ describe('HKV scanner — broken project', () => {
|
|||
});
|
||||
|
||||
it('detects object matcher (should be string)', () => {
|
||||
// CA-HKV-002 in broken-project, evidence contains the object matcher snippet.
|
||||
const found = result.findings.some(f => f.scanner === 'HKV' && f.id === 'CA-HKV-002');
|
||||
const found = result.findings.some(f => f.scanner === 'HKV' && f.id === findingId('HKV', 'matcher-not-string'));
|
||||
assert.ok(found, 'Should detect nested object matcher');
|
||||
});
|
||||
|
||||
|
|
@ -80,7 +77,6 @@ describe('HKV scanner — broken project', () => {
|
|||
|
||||
describe('HKV scanner — verbose hook output (v5 M5)', () => {
|
||||
it('flags hook script with > 50 console.log/stdout.write lines (low)', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'hooks-verbose');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -92,7 +88,6 @@ describe('HKV scanner — verbose hook output (v5 M5)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a quiet hook script', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'hooks-quiet');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -103,7 +98,6 @@ describe('HKV scanner — verbose hook output (v5 M5)', () => {
|
|||
|
||||
describe('HKV scanner — additionalContext injection advisory (v5.10 B5)', () => {
|
||||
it('flags a hook that injects unfiltered command output into additionalContext (info advisory)', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'hooks-additional-context');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -119,7 +113,6 @@ describe('HKV scanner — additionalContext injection advisory (v5.10 B5)', () =
|
|||
});
|
||||
|
||||
it('does NOT flag the filtered (grep|head) sibling hook', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'hooks-additional-context');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -130,7 +123,6 @@ describe('HKV scanner — additionalContext injection advisory (v5.10 B5)', () =
|
|||
});
|
||||
|
||||
it('does NOT flag a quiet hook with no additionalContext', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'hooks-quiet');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -150,7 +142,6 @@ describe('HKV scanner — CC 2.1.152 MessageDisplay event (Batch 1 false-positiv
|
|||
const VALID_NEW = ['MessageDisplay'];
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-hkv-events-'));
|
||||
await mkdir(join(tmpRoot, '.claude'), { recursive: true });
|
||||
const settings = {
|
||||
|
|
@ -197,7 +188,6 @@ describe('HKV scanner — post-session is NOT a settings.json hook event (Verifi
|
|||
let result;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-hkv-postsession-'));
|
||||
await mkdir(join(tmpRoot, '.claude'), { recursive: true });
|
||||
const settings = {
|
||||
|
|
@ -236,7 +226,6 @@ describe('HKV scanner — Setup/UserPromptExpansion/PostToolBatch events (Batch
|
|||
const NEW_EVENTS = ['Setup', 'UserPromptExpansion', 'PostToolBatch'];
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-hkv-events2-'));
|
||||
await mkdir(join(tmpRoot, '.claude'), { recursive: true });
|
||||
const settings = {
|
||||
|
|
@ -278,7 +267,6 @@ describe('HKV scanner — Setup/UserPromptExpansion/PostToolBatch events (Batch
|
|||
describe('HKV scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import { describe, it, beforeEach } from 'node:test';
|
|||
import assert from 'node:assert/strict';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/import-resolver.mjs';
|
||||
|
||||
|
|
@ -12,7 +11,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('IMP scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -44,7 +42,6 @@ describe('IMP scanner — healthy project', () => {
|
|||
describe('IMP scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -85,7 +82,6 @@ describe('IMP scanner — broken project', () => {
|
|||
describe('IMP scanner — minimal project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'minimal-project'));
|
||||
result = await scan(resolve(FIXTURES, 'minimal-project'), discovery);
|
||||
});
|
||||
|
|
@ -102,7 +98,6 @@ describe('IMP scanner — minimal project', () => {
|
|||
describe('IMP scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/mcp-config-validator.mjs';
|
||||
|
||||
|
|
@ -14,7 +13,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('MCP scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -54,7 +52,6 @@ describe('MCP scanner — healthy project', () => {
|
|||
describe('MCP scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -104,7 +101,6 @@ describe('MCP scanner — stray `trust` field is an unknown field (verify-first,
|
|||
let tmpRoot;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-mcp-trust-'));
|
||||
// `trust` is NOT a field in the official .mcp.json schema (verified against
|
||||
// code.claude.com/docs/en/mcp + /settings, 2026-06-18). Approval is
|
||||
|
|
@ -141,7 +137,6 @@ describe('MCP scanner — `alwaysLoad` is a valid field (v5.10 B4, verify-first
|
|||
let tmpRoot;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-mcp-alwaysload-'));
|
||||
// alwaysLoad exempts a server from MCP tool-schema deferral (CC v2.1.121+).
|
||||
// Verified against code.claude.com/docs/en/mcp.md#exempt-a-server-from-deferral.
|
||||
|
|
@ -171,7 +166,6 @@ describe('MCP scanner — env-var false positives (CC 2.1.139/2.1.142, Batch 1)'
|
|||
let envFindings;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-mcp-env-'));
|
||||
const mcp = {
|
||||
mcpServers: {
|
||||
|
|
@ -219,7 +213,6 @@ describe('MCP scanner — env-var false positives (CC 2.1.139/2.1.142, Batch 1)'
|
|||
describe('MCP scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
@ -236,7 +229,6 @@ describe('MCP scanner — empty project', () => {
|
|||
describe('MCP scanner — minimal project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'minimal-project'));
|
||||
result = await scan(resolve(FIXTURES, 'minimal-project'), discovery);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -15,7 +15,6 @@ import assert from 'node:assert/strict';
|
|||
import { join } from 'node:path';
|
||||
import { mkdtemp, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/optimization-lens-scanner.mjs';
|
||||
|
||||
const tmp = () => mkdtemp(join(tmpdir(), 'ca-opt-'));
|
||||
|
|
@ -29,7 +28,6 @@ async function runOpt(claudeMd, relPath = 'CLAUDE.md') {
|
|||
const repo = await tmp();
|
||||
const abs = join(repo, relPath.replace(/\//g, '-'));
|
||||
await writeFile(abs, claudeMd, 'utf-8');
|
||||
resetCounter();
|
||||
try {
|
||||
return await scan(repo, {
|
||||
files: [{ absPath: abs, relPath, type: 'claude-md', scope: 'project', size: claudeMd.length }],
|
||||
|
|
@ -50,7 +48,6 @@ describe('OPT scanner — result shape', () => {
|
|||
|
||||
it('emits nothing and stays ok when no claude-md files are discovered', async () => {
|
||||
const repo = await tmp();
|
||||
resetCounter();
|
||||
try {
|
||||
const result = await scan(repo, { files: [] });
|
||||
assert.equal(result.status, 'ok');
|
||||
|
|
|
|||
|
|
@ -16,7 +16,6 @@ import assert from 'node:assert/strict';
|
|||
import { join } from 'node:path';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/output-style-scanner.mjs';
|
||||
|
||||
function tmp(prefix) {
|
||||
|
|
@ -66,7 +65,6 @@ async function runOst(opts = {}) {
|
|||
const originalProfile = process.env.USERPROFILE;
|
||||
process.env.HOME = home;
|
||||
process.env.USERPROFILE = home;
|
||||
resetCounter();
|
||||
try {
|
||||
return await scan(repo, { files: [] });
|
||||
} finally {
|
||||
|
|
|
|||
|
|
@ -5,8 +5,8 @@ import { fileURLToPath } from 'node:url';
|
|||
import { mkdtemp, mkdir, writeFile, rm, readFile } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan, discoverPlugins } from '../../scanners/plugin-health-scanner.mjs';
|
||||
import { findingId } from '../../scanners/lib/finding-codes.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
|
@ -39,14 +39,12 @@ describe('discoverPlugins', () => {
|
|||
|
||||
describe('scan on valid test-plugin', () => {
|
||||
it('returns ok status', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
assert.equal(result.scanner, 'PLH');
|
||||
assert.equal(result.status, 'ok');
|
||||
});
|
||||
|
||||
it('finds commands and agents', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
assert.ok(result.files_scanned >= 1, 'Should scan at least 1 plugin');
|
||||
// Valid plugin should have few or no findings
|
||||
|
|
@ -55,7 +53,6 @@ describe('scan on valid test-plugin', () => {
|
|||
});
|
||||
|
||||
it('no findings for missing plugin.json fields', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
// Anchor on PLH + a title-substring stable across humanizer rewrites.
|
||||
// Raw: "Missing required field in plugin.json: <field>". Humanized: "A plugin's manifest is missing a required field".
|
||||
|
|
@ -66,7 +63,6 @@ describe('scan on valid test-plugin', () => {
|
|||
});
|
||||
|
||||
it('no findings for missing CLAUDE.md sections', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
// Raw: "CLAUDE.md missing '<name>' section". Humanized: "A plugin's instructions file is missing a recommended section".
|
||||
const missingSections = result.findings.filter(f =>
|
||||
|
|
@ -78,7 +74,6 @@ describe('scan on valid test-plugin', () => {
|
|||
|
||||
describe('CLAUDE.md section findings track present components', () => {
|
||||
it('flags a missing section only for components the plugin actually ships', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(SECTION_COV);
|
||||
// section-coverage ships commands/ but no agents/ or hooks, and its CLAUDE.md omits the
|
||||
// Commands section. Per the component-aware rule: flag the present component's missing
|
||||
|
|
@ -97,33 +92,27 @@ describe('CLAUDE.md section findings track present components', () => {
|
|||
|
||||
describe('scan on broken-plugin', () => {
|
||||
it('detects missing plugin.json fields', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(BROKEN_PLUGIN);
|
||||
// CA-PLH-001 (description) and CA-PLH-002 (version) in broken-plugin.
|
||||
// One check ("missing required field"), two instances (description, version).
|
||||
const missingFields = result.findings.filter(f =>
|
||||
f.scanner === 'PLH' && (f.id === 'CA-PLH-001' || f.id === 'CA-PLH-002')
|
||||
f.scanner === 'PLH' && f.id === findingId('PLH', 'missing-required-field')
|
||||
);
|
||||
assert.ok(missingFields.length >= 2, 'Should detect missing description and version');
|
||||
});
|
||||
|
||||
it('detects missing CLAUDE.md', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(BROKEN_PLUGIN);
|
||||
// CA-PLH-003 in broken-plugin = Missing CLAUDE.md.
|
||||
const missingMd = result.findings.filter(f => f.scanner === 'PLH' && f.id === 'CA-PLH-003');
|
||||
const missingMd = result.findings.filter(f => f.scanner === 'PLH' && f.id === findingId('PLH', 'missing-claude-md'));
|
||||
assert.equal(missingMd.length, 1, 'Should detect missing CLAUDE.md');
|
||||
});
|
||||
|
||||
it('detects command without frontmatter', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(BROKEN_PLUGIN);
|
||||
// CA-PLH-004 in broken-plugin = Command missing frontmatter.
|
||||
const noFrontmatter = result.findings.filter(f => f.scanner === 'PLH' && f.id === 'CA-PLH-004');
|
||||
const noFrontmatter = result.findings.filter(f => f.scanner === 'PLH' && f.id === findingId('PLH', 'command-missing-frontmatter'));
|
||||
assert.equal(noFrontmatter.length, 1, 'Should detect command without frontmatter');
|
||||
});
|
||||
|
||||
it('flags missing required agent field (description) but not optional model/tools', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(BROKEN_PLUGIN);
|
||||
// Per CC sub-agents docs: only `name` and `description` are required; `model` and `tools`
|
||||
// are optional (inherit / all-tools by default). bad-agent.md has `name` only.
|
||||
|
|
@ -138,11 +127,9 @@ describe('scan on broken-plugin', () => {
|
|||
|
||||
describe('scan with no plugins', () => {
|
||||
it('returns info finding for empty directory', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(resolve(FIXTURES, 'empty-project'));
|
||||
assert.equal(result.findings.length, 1);
|
||||
// CA-PLH-001 in empty-project = No plugins found.
|
||||
assert.equal(result.findings[0].id, 'CA-PLH-001');
|
||||
assert.equal(result.findings[0].id, findingId('PLH', 'no-plugins-found'));
|
||||
assert.equal(result.findings[0].scanner, 'PLH');
|
||||
assert.equal(result.findings[0].severity, 'info');
|
||||
});
|
||||
|
|
@ -150,7 +137,6 @@ describe('scan with no plugins', () => {
|
|||
|
||||
describe('cross-plugin command conflict detection', () => {
|
||||
it('scans fixtures dir and reports findings for all plugins', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(FIXTURES);
|
||||
assert.equal(result.scanner, 'PLH');
|
||||
assert.ok(result.files_scanned >= 2, 'Should scan multiple plugins');
|
||||
|
|
@ -161,7 +147,6 @@ describe('plugin namespace collision detection', () => {
|
|||
const COLLISION_RE = /namespace collision/i;
|
||||
|
||||
it('flags two plugins that declare the same name', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(DUP_NAME);
|
||||
const collisions = result.findings.filter(f =>
|
||||
f.scanner === 'PLH' && COLLISION_RE.test(f.title || '')
|
||||
|
|
@ -181,7 +166,6 @@ describe('plugin namespace collision detection', () => {
|
|||
});
|
||||
|
||||
it('excludes name-less plugins from the collision map', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(DUP_NAME);
|
||||
// gamma + delta declare no name; they must NOT form an undefined/empty collision.
|
||||
const collisions = result.findings.filter(f =>
|
||||
|
|
@ -195,7 +179,6 @@ describe('plugin namespace collision detection', () => {
|
|||
});
|
||||
|
||||
it('does not flag a single plugin as a collision', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
const collisions = result.findings.filter(f =>
|
||||
f.scanner === 'PLH' && COLLISION_RE.test(f.title || '')
|
||||
|
|
@ -208,7 +191,6 @@ describe('cross-plugin command name ambiguity (COL-level)', () => {
|
|||
const CMD_RE = /used by multiple plugins/i;
|
||||
|
||||
it('flags a command name shared across different plugin namespaces as low', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(DUP_CMD);
|
||||
const amb = result.findings.filter(f => f.scanner === 'PLH' && CMD_RE.test(f.title || ''));
|
||||
assert.equal(amb.length, 1, `Expected one command-ambiguity finding, got ${amb.length}`);
|
||||
|
|
@ -221,7 +203,6 @@ describe('cross-plugin command name ambiguity (COL-level)', () => {
|
|||
});
|
||||
|
||||
it('labels plugins by declared name, not folder basename', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(DUP_CMD);
|
||||
const f = result.findings.find(x => x.scanner === 'PLH' && CMD_RE.test(x.title || ''));
|
||||
const sources = f.details.namespaces.map(n => n.source).sort();
|
||||
|
|
@ -230,7 +211,6 @@ describe('cross-plugin command name ambiguity (COL-level)', () => {
|
|||
});
|
||||
|
||||
it('does not emit a high-severity command conflict (legacy behavior removed)', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(DUP_CMD);
|
||||
const high = result.findings.filter(f =>
|
||||
f.scanner === 'PLH' && /command name conflict/i.test(f.title || '') && f.severity === 'high'
|
||||
|
|
@ -239,7 +219,6 @@ describe('cross-plugin command name ambiguity (COL-level)', () => {
|
|||
});
|
||||
|
||||
it('does not flag a shared command WITHIN a colliding namespace (namespace-collision covers it)', async () => {
|
||||
resetCounter();
|
||||
// alpha + beta both declare name "dup" and both ship a "hello" command.
|
||||
const result = await scan(DUP_NAME);
|
||||
const amb = result.findings.filter(f => f.scanner === 'PLH' && CMD_RE.test(f.title || ''));
|
||||
|
|
@ -252,7 +231,6 @@ describe('plugin-folder shadowing (CA-PLH-015)', () => {
|
|||
const SHADOW_RE = /shadows the default/i;
|
||||
|
||||
it('flags a manifest path that shadows the default commands/ folder', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(SHADOW);
|
||||
const shadows = result.findings.filter(f => f.scanner === 'PLH' && SHADOW_RE.test(f.title || ''));
|
||||
assert.equal(shadows.length, 1, `Expected exactly one shadow finding, got ${shadows.length}: ${shadows.map(f => f.title).join(' | ')}`);
|
||||
|
|
@ -266,7 +244,6 @@ describe('plugin-folder shadowing (CA-PLH-015)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a default folder addressed explicitly in the manifest array', async () => {
|
||||
resetCounter();
|
||||
// agents: ["./agents/", "./more-agents/"] addresses the default agents/ folder → no warning.
|
||||
const result = await scan(SHADOW);
|
||||
const agentShadows = result.findings.filter(f =>
|
||||
|
|
@ -276,7 +253,6 @@ describe('plugin-folder shadowing (CA-PLH-015)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag skills (adds to default, not replace)', async () => {
|
||||
resetCounter();
|
||||
// skills: "./custom-skills/" ADDS to the default skills/ scan; both load → no shadow.
|
||||
const result = await scan(SHADOW);
|
||||
const skillShadows = result.findings.filter(f =>
|
||||
|
|
@ -286,7 +262,6 @@ describe('plugin-folder shadowing (CA-PLH-015)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag when the default folder is absent', async () => {
|
||||
resetCounter();
|
||||
// outputStyles: "./styles/" is declared, but there is no output-styles/ folder → nothing ignored.
|
||||
const result = await scan(SHADOW);
|
||||
const osShadows = result.findings.filter(f =>
|
||||
|
|
@ -296,7 +271,6 @@ describe('plugin-folder shadowing (CA-PLH-015)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a plugin with no component-path keys', async () => {
|
||||
resetCounter();
|
||||
// test-plugin has commands/ and agents/ folders but declares no custom paths → nothing shadowed.
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
const shadows = result.findings.filter(f => f.scanner === 'PLH' && SHADOW_RE.test(f.title || ''));
|
||||
|
|
@ -309,7 +283,6 @@ describe('skills:-array entry validation (CA-PLH-016)', () => {
|
|||
const SKILLS_RE = /^plugin\.json "skills" entry/;
|
||||
|
||||
it('flags one finding per bad entry (file, missing, escape, non-string) and none for a valid dir', async () => {
|
||||
resetCounter();
|
||||
// skills: ["./valid-skill/", "./a-file.md", "./missing-dir/", "../escape", 42]
|
||||
const result = await scan(SKILLS_ARR);
|
||||
const bad = result.findings.filter(f => f.scanner === 'PLH' && SKILLS_RE.test(f.title || ''));
|
||||
|
|
@ -329,7 +302,6 @@ describe('skills:-array entry validation (CA-PLH-016)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag the valid skill directory', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(SKILLS_ARR);
|
||||
const validFlagged = result.findings.some(f =>
|
||||
f.scanner === 'PLH' && SKILLS_RE.test(f.title || '') && /valid-skill/.test(f.title || '')
|
||||
|
|
@ -338,7 +310,6 @@ describe('skills:-array entry validation (CA-PLH-016)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a plugin with no skills: key', async () => {
|
||||
resetCounter();
|
||||
// test-plugin declares no skills: field.
|
||||
const result = await scan(TEST_PLUGIN);
|
||||
const skillsFindings = result.findings.filter(f => f.scanner === 'PLH' && SKILLS_RE.test(f.title || ''));
|
||||
|
|
@ -348,7 +319,6 @@ describe('skills:-array entry validation (CA-PLH-016)', () => {
|
|||
|
||||
describe('finding format', () => {
|
||||
it('findings have standard fields', async () => {
|
||||
resetCounter();
|
||||
const result = await scan(BROKEN_PLUGIN);
|
||||
assert.ok(result.findings.length > 0);
|
||||
const f = result.findings[0];
|
||||
|
|
@ -383,7 +353,6 @@ describe('PLH — plugin agent declares fields Claude Code ignores (E)', () => {
|
|||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-plh-agentdead-'));
|
||||
await writePlugin(tmpRoot, 'permissionMode: plan\nhooks: present\nmcpServers: present\n');
|
||||
result = await scan(tmpRoot);
|
||||
|
|
@ -410,7 +379,6 @@ describe('PLH — plugin agent declares fields Claude Code ignores (E)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a clean agent (name + description only)', async () => {
|
||||
resetCounter();
|
||||
const clean = await mkdtemp(join(tmpdir(), 'ca-plh-agentclean-'));
|
||||
try {
|
||||
await writePlugin(clean, '');
|
||||
|
|
@ -578,7 +546,6 @@ describe('.claude-plugin/marketplace.json is not an unknown file (F9)', () => {
|
|||
}
|
||||
|
||||
it('does not flag marketplace.json (documented catalog location)', async () => {
|
||||
resetCounter();
|
||||
await writeMarketplacePlugin(dir);
|
||||
const result = await scan(dir);
|
||||
const unknown = result.findings.filter(f => /Unknown file/i.test(f.title || ''));
|
||||
|
|
@ -587,7 +554,6 @@ describe('.claude-plugin/marketplace.json is not an unknown file (F9)', () => {
|
|||
});
|
||||
|
||||
it('still flags a genuinely unexpected file', async () => {
|
||||
resetCounter();
|
||||
await writeMarketplacePlugin(dir, { 'notes.txt': 'scratch' });
|
||||
const result = await scan(dir);
|
||||
const unknown = result.findings.filter(f => /Unknown file/i.test(f.title || ''));
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@ import { hermeticEnv } from '../helpers/hermetic-home.mjs';
|
|||
import { stripHotspotLoadPattern } from '../helpers/strip-hotspot-load-pattern.mjs';
|
||||
import { stripAddedScanners, stripAddedScannerStderr } from '../helpers/strip-added-scanner.mjs';
|
||||
import { stripRetiredGap, maskGapTallyStderr } from '../helpers/strip-retired-gap.mjs';
|
||||
import { maskFindingIds } from '../helpers/mask-finding-ids.mjs';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
|
@ -42,7 +43,7 @@ function normalizePosture(p) {
|
|||
}
|
||||
}
|
||||
}
|
||||
return stripRetiredGap(stripAddedScanners(stripHotspotLoadPattern(out)));
|
||||
return maskFindingIds(stripRetiredGap(stripAddedScanners(stripHotspotLoadPattern(out))));
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/rules-validator.mjs';
|
||||
|
||||
|
|
@ -14,7 +13,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('RUL scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -42,7 +40,6 @@ describe('RUL scanner — healthy project', () => {
|
|||
describe('RUL scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -83,7 +80,6 @@ describe('RUL scanner — broken project', () => {
|
|||
describe('RUL scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
@ -120,7 +116,6 @@ describe('RUL — large path-scoped rule lost after compaction (A)', () => {
|
|||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-rul-durability-'));
|
||||
await writeProject(tmpRoot, 60);
|
||||
const discovery = await discoverConfigFiles(tmpRoot);
|
||||
|
|
@ -138,7 +133,6 @@ describe('RUL — large path-scoped rule lost after compaction (A)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a small path-scoped rule', async () => {
|
||||
resetCounter();
|
||||
const small = await mkdtemp(join(tmpdir(), 'ca-rul-small-'));
|
||||
try {
|
||||
await writeProject(small, 3);
|
||||
|
|
@ -177,7 +171,6 @@ describe('RUL — block-sequence-scoped rule is correctly scoped (parser regress
|
|||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-rul-blockseq-'));
|
||||
await writeBlockSeqProject(tmpRoot, 60);
|
||||
const discovery = await discoverConfigFiles(tmpRoot);
|
||||
|
|
@ -235,7 +228,6 @@ describe('RUL — nested-repo rule glob resolves to its own project root (M-BUG-
|
|||
}
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-rul-nested-'));
|
||||
await writeNestedRepo(tmpRoot);
|
||||
const discovery = await discoverConfigFiles(tmpRoot);
|
||||
|
|
@ -266,7 +258,6 @@ describe('RUL — user-global rule is not flagged "matches no files" (M-BUG-9 gu
|
|||
let result;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpHome = await mkdtemp(join(tmpdir(), 'ca-rul-home-'));
|
||||
savedHome = process.env.HOME;
|
||||
process.env.HOME = tmpHome;
|
||||
|
|
@ -302,7 +293,6 @@ describe('RUL — mid-pattern /**/ glob matches intermediate dirs (M-BUG-19)', (
|
|||
let result;
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-rul-globstar-'));
|
||||
await mkdir(join(tmpRoot, '.claude', 'rules'), { recursive: true });
|
||||
await mkdir(join(tmpRoot, 'posts', '2026-01-23-slug'), { recursive: true });
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import { hermeticEnv } from '../helpers/hermetic-home.mjs';
|
|||
import { stripHotspotLoadPattern } from '../helpers/strip-hotspot-load-pattern.mjs';
|
||||
import { stripAddedScanners } from '../helpers/strip-added-scanner.mjs';
|
||||
import { stripRetiredGap } from '../helpers/strip-retired-gap.mjs';
|
||||
import { maskFindingIds } from '../helpers/mask-finding-ids.mjs';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
|
|
@ -39,7 +40,7 @@ function normalizeEnvelope(env) {
|
|||
}
|
||||
}
|
||||
}
|
||||
return stripRetiredGap(stripAddedScanners(stripHotspotLoadPattern(out)));
|
||||
return maskFindingIds(stripRetiredGap(stripAddedScanners(stripHotspotLoadPattern(out))));
|
||||
}
|
||||
|
||||
async function runOrchestrator(flags) {
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdtemp, mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { scan } from '../../scanners/settings-validator.mjs';
|
||||
import { findingId } from '../../scanners/lib/finding-codes.mjs';
|
||||
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
const FIXTURES = resolve(__dirname, '../fixtures');
|
||||
|
|
@ -14,7 +14,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
describe('SET scanner — healthy project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'healthy-project'));
|
||||
result = await scan(resolve(FIXTURES, 'healthy-project'), discovery);
|
||||
});
|
||||
|
|
@ -42,7 +41,6 @@ describe('SET scanner — healthy project', () => {
|
|||
describe('SET scanner — broken project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'broken-project'));
|
||||
result = await scan(resolve(FIXTURES, 'broken-project'), discovery);
|
||||
});
|
||||
|
|
@ -87,7 +85,6 @@ describe('SET scanner — broken project', () => {
|
|||
|
||||
describe('SET scanner — additionalDirectories (v5 M6)', () => {
|
||||
it('does NOT flag additionalDirectories as unknown key', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'additional-dirs-ok');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -100,7 +97,6 @@ describe('SET scanner — additionalDirectories (v5 M6)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag 2 entries as too many', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'additional-dirs-ok');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -112,12 +108,10 @@ describe('SET scanner — additionalDirectories (v5 M6)', () => {
|
|||
});
|
||||
|
||||
it('flags > 2 entries as low finding', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'additional-dirs-many');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
// CA-SET-001 in additional-dirs-many = the additionalDirectories threshold finding.
|
||||
const f = result.findings.find(x => x.scanner === 'SET' && x.id === 'CA-SET-001');
|
||||
const f = result.findings.find(x => x.scanner === 'SET' && x.id === findingId('SET', 'many-additional-dirs'));
|
||||
assert.ok(f, `expected additionalDirectories threshold finding; got: ${result.findings.map(x => x.id).join(' | ')}`);
|
||||
assert.equal(f.severity, 'low', `expected low severity, got ${f.severity}`);
|
||||
});
|
||||
|
|
@ -137,7 +131,6 @@ describe('SET scanner — CC 2.1.114→181 valid keys (Batch 1 false-positive fi
|
|||
];
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-set-2181-'));
|
||||
await mkdir(join(tmpRoot, '.claude'), { recursive: true });
|
||||
const settings = {
|
||||
|
|
@ -206,7 +199,6 @@ describe('SET scanner — unknown-key typo gate (M-BUG-10)', () => {
|
|||
const TYPO_TITLE = 'Possible typo in settings key';
|
||||
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
tmpRoot = await mkdtemp(join(tmpdir(), 'ca-set-typo-'));
|
||||
await mkdir(join(tmpRoot, '.claude'), { recursive: true });
|
||||
const settings = {
|
||||
|
|
@ -254,7 +246,6 @@ describe('SET scanner — unknown-key typo gate (M-BUG-10)', () => {
|
|||
describe('SET scanner — empty project', () => {
|
||||
let result;
|
||||
beforeEach(async () => {
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(resolve(FIXTURES, 'empty-project'));
|
||||
result = await scan(resolve(FIXTURES, 'empty-project'), discovery);
|
||||
});
|
||||
|
|
@ -274,7 +265,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
const amFindings = (result) => result.findings.filter(f => f.scanner === 'SET' && AM_RE.test(f.title || ''));
|
||||
|
||||
it('flags structural problems: wrong sub-key type, non-string array entry, unknown sub-key', async () => {
|
||||
resetCounter();
|
||||
// automode-structure/.claude/settings.local.json (scope=local → no dead-config):
|
||||
// environment valid; allow not-array; soft_deny ["ok",7]; bogus unknown sub-key.
|
||||
const path = resolve(FIXTURES, 'automode-structure');
|
||||
|
|
@ -295,7 +285,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a valid string-array sub-key (incl. "$defaults")', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'automode-structure');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -305,7 +294,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
});
|
||||
|
||||
it('flags autoMode that is not an object', async () => {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, 'automode-nonobject');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
const result = await scan(path, discovery);
|
||||
|
|
@ -316,7 +304,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
});
|
||||
|
||||
it('flags autoMode in shared project settings as dead config (and not in local scope)', async () => {
|
||||
resetCounter();
|
||||
// automode-shared/.claude/settings.json → scope=project → dead-config.
|
||||
const sharedPath = resolve(FIXTURES, 'automode-shared');
|
||||
const sharedResult = await scan(sharedPath, await discoverConfigFiles(sharedPath));
|
||||
|
|
@ -327,7 +314,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
assert.equal(amFindings(sharedResult).length, 1, 'valid structure in shared scope yields only the dead-config finding');
|
||||
|
||||
// The same-shaped autoMode in LOCAL scope (automode-structure) must NOT be dead config.
|
||||
resetCounter();
|
||||
const localPath = resolve(FIXTURES, 'automode-structure');
|
||||
const localResult = await scan(localPath, await discoverConfigFiles(localPath));
|
||||
const localDead = amFindings(localResult).filter(f => f.details && f.details.problem === 'shared-project-scope');
|
||||
|
|
@ -335,7 +321,6 @@ describe('SET scanner — autoMode validation (CA-SET-NNN)', () => {
|
|||
});
|
||||
|
||||
it('does NOT flag a settings file without autoMode', async () => {
|
||||
resetCounter();
|
||||
// healthy-project has .claude/settings.json but no autoMode.
|
||||
const path = resolve(FIXTURES, 'healthy-project');
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
|
|
|
|||
|
|
@ -3,7 +3,6 @@ import assert from 'node:assert/strict';
|
|||
import { join } from 'node:path';
|
||||
import { mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/skill-listing-scanner.mjs';
|
||||
|
||||
const CAP = 1536; // verified per-description listing cap (CC 2.1.105, changelog L1502)
|
||||
|
|
@ -18,7 +17,6 @@ function uniqueDir(suffix) {
|
|||
* never rely on the developer's real ~/.claude.
|
||||
*/
|
||||
async function runScannerWithHome(home) {
|
||||
resetCounter();
|
||||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
|
|
@ -30,7 +28,6 @@ async function runScannerWithHome(home) {
|
|||
|
||||
/** Like runScannerWithHome but threads a resolved { window, advisory } (B8 calibration). */
|
||||
async function runScannerWithCtx(home, contextWindow) {
|
||||
resetCounter();
|
||||
const original = process.env.HOME;
|
||||
process.env.HOME = home;
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -2,7 +2,6 @@ import { describe, it } from 'node:test';
|
|||
import assert from 'node:assert/strict';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/token-hotspots.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { withHermeticHome } from '../helpers/hermetic-home.mjs';
|
||||
|
|
@ -14,7 +13,6 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
// ~/.claude.json plugin MCP servers into the fixture result (mirrors the other
|
||||
// TOK tests). The deferral check reads project+local settings + project .mcp.json.
|
||||
async function runScanner(fixtureName) {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, fixtureName);
|
||||
const discovery = await discoverConfigFiles(path);
|
||||
return withHermeticHome(() => scan(path, discovery));
|
||||
|
|
|
|||
|
|
@ -4,7 +4,6 @@ import { resolve, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { mkdir, writeFile, rm } from 'node:fs/promises';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { resetCounter } from '../../scanners/lib/output.mjs';
|
||||
import { scan } from '../../scanners/token-hotspots.mjs';
|
||||
import { discoverConfigFiles } from '../../scanners/lib/file-discovery.mjs';
|
||||
import { withHermeticHome } from '../helpers/hermetic-home.mjs';
|
||||
|
|
@ -17,7 +16,6 @@ async function fixtureDiscovery(name) {
|
|||
}
|
||||
|
||||
async function runScanner(fixtureName) {
|
||||
resetCounter();
|
||||
const path = resolve(FIXTURES, fixtureName);
|
||||
const discovery = await fixtureDiscovery(fixtureName);
|
||||
// Hermetic HOME: scan() calls readActiveConfig, which resolves the user
|
||||
|
|
@ -368,7 +366,6 @@ describe('TOK scanner — H stale plugin-cache versions (v5.9 B3)', () => {
|
|||
};
|
||||
|
||||
it('emits a low-severity stale-cache finding when stale versions exist', async () => {
|
||||
resetCounter();
|
||||
const result = await withHermeticHome(() => scan('/tmp/x', staleDiscovery));
|
||||
const f = result.findings.find(x => /stale plugin-cache/i.test(x.title || ''));
|
||||
assert.ok(f, 'expected a stale plugin-cache finding');
|
||||
|
|
@ -377,7 +374,6 @@ describe('TOK scanner — H stale plugin-cache versions (v5.9 B3)', () => {
|
|||
});
|
||||
|
||||
it('states zero live-context impact and lists the stale keys', async () => {
|
||||
resetCounter();
|
||||
const result = await withHermeticHome(() => scan('/tmp/x', staleDiscovery));
|
||||
const f = result.findings.find(x => /stale plugin-cache/i.test(x.title || ''));
|
||||
assert.match(f.evidence, /zero live-context impact/i);
|
||||
|
|
@ -386,7 +382,6 @@ describe('TOK scanner — H stale plugin-cache versions (v5.9 B3)', () => {
|
|||
});
|
||||
|
||||
it('warns against deleting a version a running session still uses (/exit caveat)', async () => {
|
||||
resetCounter();
|
||||
const result = await withHermeticHome(() => scan('/tmp/x', staleDiscovery));
|
||||
const f = result.findings.find(x => /stale plugin-cache/i.test(x.title || ''));
|
||||
assert.match(f.recommendation, /running session|active session/i,
|
||||
|
|
@ -396,13 +391,11 @@ describe('TOK scanner — H stale plugin-cache versions (v5.9 B3)', () => {
|
|||
});
|
||||
|
||||
it('does NOT fire when there are no stale versions', async () => {
|
||||
resetCounter();
|
||||
const result = await withHermeticHome(() => scan('/tmp/x', { files: [], staleCacheVersions: [] }));
|
||||
assert.ok(!result.findings.some(x => /stale plugin-cache/i.test(x.title || '')));
|
||||
});
|
||||
|
||||
it('does NOT fire when discovery omits staleCacheVersions entirely', async () => {
|
||||
resetCounter();
|
||||
const result = await withHermeticHome(() => scan('/tmp/x', { files: [] }));
|
||||
assert.ok(!result.findings.some(x => /stale plugin-cache/i.test(x.title || '')));
|
||||
});
|
||||
|
|
@ -423,7 +416,6 @@ describe('TOK scanner — CLAUDE.md HTML-comment token discount (M-BUG-6)', () =
|
|||
const content = `# Root\n\n${comment}\n\nReal instruction body.\n`;
|
||||
const rawBytes = Buffer.byteLength(content, 'utf8');
|
||||
await writeFile(join(dir, 'CLAUDE.md'), content);
|
||||
resetCounter();
|
||||
const discovery = await discoverConfigFiles(dir);
|
||||
const result = await withHermeticHome(() => scan(dir, discovery));
|
||||
const hs = result.hotspots.find(
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue