BREAKING CHANGE: the {NNN} in CA-{SCANNER}-{NNN} identifies the check that
produced the finding. It used to be the finding's position in that scanner's
output for that run, which made it unstable across CONFIGURATIONS, not just
across releases as STATE framed it. Measured on two fixtures: "No custom
subagents" was CA-GAP-007 on minimal-project and CA-GAP-004 on healthy-project.
A user who fixed an unrelated earlier gap silently renumbered every later one,
so a .config-audit-ignore pin retargeted to a neighbouring finding with no
version change at all.
Second measured arm: README already documented the opposite scheme. It and the
scanner headers describe ~20 numbers as check codes (CA-SKL-003 = oversized
body, CA-PLH-015 = folder shadowing, CA-TOK-006 = schema deferral), and the
counter could only produce those in the all-fire case -- source-order positions
are 4, 3 and 8. The documentation described the scheme; the implementation was
what was wrong. Every published number is preserved by construction and pinned
exhaustively in tests/lib/finding-codes.test.mjs.
scanners/lib/finding-codes.mjs is the single authority. Every finding() call
passes a `code`; an undeclared or missing one THROWS. No counter fallback --
that would reproduce D1's findGapId -> 'unknown' silent degradation and let a
half-converted scanner ship IDs that look valid. findingCounter/resetCounter
are deleted outright, not left as no-ops. Retirement is now a mechanism:
RETIRED_CODES tombstones a withdrawn key so its number is never reissued,
seeded with GAP t3_8 -- the D1 removal that opened this chunk.
IDs are consequently NOT unique per finding: one check failing in three files
emits three findings sharing an ID. That inverts which consumer is correct, so
every f.id/findingId site was classified before the change. diff-engine and
most of fix-engine already keyed on scanner+title+file (drift was never lying);
fix-engine's verification did not, and keyed on the ID alone -- fixing one of
two sibling instances marked both fixed, and the untouched one, still present
in the re-scan, was reported as a REGRESSION. Red test first, then keyed on
(findingId, file), which both planFixes and applyFixes already carry.
plugin-health's crossIds Set was measured and is a clean negative: cross
findings are allFindings.slice(crossPluginStart) and codes 18/19 are emitted
only in that tail, so the partition holds by construction.
unknownSuppressions() reports a pin that names no declared check, in the
--output-file payload (ux-rules rule 2 -- a stderr-only warning is invisible to
the commands) and only when one exists, so a clean config is byte-identical.
That is what makes the break safe: a stale pin goes loud instead of dying quiet.
Frozen tests/snapshots/v5.0.0/ untouched on disk. IDs are masked out of that
comparison (mask-finding-ids.mjs) rather than re-derived -- re-deriving
positional IDs would assert the retired scheme against itself, and #58's
isGapEntry off-by-one is the measured example of that misfiring. The dead
re-derivation is removed from strip-retired-gap.mjs. default-output snapshots
re-approved after confirming the diff is IDs and nothing else.
Guards, each seen red against its own defect: a missing code (scanner errors
out mid-sweep), an orphan declaration, a resurrected retired key, and a
documented ID naming no check. The sweep asserts the union across all 16
scanners, never per scanner -- a per-scanner assertion goes green on a partial
conversion.
Fasit written before implementation: docs/mbug28-id-semantics-fasit.local.md,
including one correction made before running (CML has 12 checks over 13 call
sites -- the anchored and calibrated char-budget arms are one check, which a
repeated-title sweep found and my call-site count had missed).
Suite 1535 -> 1573, 0 failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MyqCQKK2ornJ1jFWwqx17E
345 lines
14 KiB
JavaScript
345 lines
14 KiB
JavaScript
#!/usr/bin/env node
|
|
|
|
/**
|
|
* Config-Audit Scan Orchestrator
|
|
* Runs all registered scanners sequentially, collects findings, outputs JSON envelope.
|
|
* Usage: node scan-orchestrator.mjs <target-path> [--output-file path] [--save-baseline] [--baseline path]
|
|
* Zero external dependencies.
|
|
*/
|
|
|
|
import { resolve, sep } from 'node:path';
|
|
import { readFile, writeFile } from 'node:fs/promises';
|
|
import { writeOutputFile } from './lib/write-output.mjs';
|
|
import { requireTargetDir } from './lib/require-target-dir.mjs';
|
|
import { envelope } from './lib/output.mjs';
|
|
import { discoverConfigFiles, discoverConfigFilesMulti, discoverFullMachinePaths } from './lib/file-discovery.mjs';
|
|
import { loadSuppressions, applySuppressions, formatSuppressionSummary, unknownSuppressions } from './lib/suppression.mjs';
|
|
import { humanizeEnvelope } from './lib/humanizer.mjs';
|
|
import { resolveContextWindow } from './lib/context-window.mjs';
|
|
import { resolveActiveModel } from './lib/active-model.mjs';
|
|
|
|
// Scanner registry — import order determines execution order
|
|
import { scan as scanClaudeMd } from './claude-md-linter.mjs';
|
|
import { scan as scanSettings } from './settings-validator.mjs';
|
|
import { scan as scanHooks } from './hook-validator.mjs';
|
|
import { scan as scanRules } from './rules-validator.mjs';
|
|
import { scan as scanMcp } from './mcp-config-validator.mjs';
|
|
import { scan as scanImports } from './import-resolver.mjs';
|
|
import { scan as scanConflicts } from './conflict-detector.mjs';
|
|
import { scan as scanGap } from './feature-gap-scanner.mjs';
|
|
import { scan as scanTokenHotspots } from './token-hotspots.mjs';
|
|
import { scan as scanCachePrefix } from './cache-prefix-scanner.mjs';
|
|
import { scan as scanDisabledInSchema } from './disabled-in-schema-scanner.mjs';
|
|
import { scan as scanCollision } from './collision-scanner.mjs';
|
|
import { scan as scanSkillListing } from './skill-listing-scanner.mjs';
|
|
import { scan as scanAgentListing } from './agent-listing-scanner.mjs';
|
|
import { scan as scanOutputStyle } from './output-style-scanner.mjs';
|
|
import { scan as scanOptimizationLens } from './optimization-lens-scanner.mjs';
|
|
import { requireValidArgs } from './lib/cli-args.mjs';
|
|
|
|
/** Flag surface, measured 2026-08-09. Anything else is exit 3. */
|
|
const ARG_SPEC = {
|
|
boolean: [
|
|
'--json', '--raw', '--global', '--full-machine', '--no-suppress',
|
|
'--include-fixtures', '--exclude-cache', '--no-exclude-cache', '--save-baseline',
|
|
],
|
|
value: ['--output-file', '--context-window', '--baseline'],
|
|
};
|
|
|
|
// Directory names that identify test fixture / example directories
|
|
const FIXTURE_DIR_NAMES = ['tests', 'examples', '__tests__', 'test-fixtures'];
|
|
|
|
/**
|
|
* Check if a finding originates from a test fixture or example directory
|
|
* relative to the scan target. Only filters when the finding's path extends
|
|
* beyond the target into a fixture subdirectory — if the target itself is
|
|
* a fixture directory, findings are NOT filtered.
|
|
* @param {object} f - Finding object
|
|
* @param {string} targetPath - Resolved scan target path
|
|
* @returns {boolean}
|
|
*/
|
|
function isFixturePath(f, targetPath) {
|
|
const p = f.file || f.path || f.location || '';
|
|
if (!p || !p.startsWith(targetPath)) return false;
|
|
// Get the path relative to target, then check if it passes through a fixture dir
|
|
const rel = p.slice(targetPath.length);
|
|
return FIXTURE_DIR_NAMES.some(dir => rel.includes(sep + dir + sep));
|
|
}
|
|
|
|
const SCANNERS = [
|
|
{ name: 'CML', fn: scanClaudeMd, label: 'CLAUDE.md Linter' },
|
|
{ name: 'SET', fn: scanSettings, label: 'Settings Validator' },
|
|
{ name: 'HKV', fn: scanHooks, label: 'Hook Validator' },
|
|
{ name: 'RUL', fn: scanRules, label: 'Rules Validator' },
|
|
{ name: 'MCP', fn: scanMcp, label: 'MCP Config Validator' },
|
|
{ name: 'IMP', fn: scanImports, label: 'Import Resolver' },
|
|
{ name: 'CNF', fn: scanConflicts, label: 'Conflict Detector' },
|
|
{ name: 'GAP', fn: scanGap, label: 'Feature Gap Scanner' },
|
|
{ name: 'TOK', fn: scanTokenHotspots, label: 'Token Hotspots' },
|
|
{ name: 'CPS', fn: scanCachePrefix, label: 'Cache-Prefix Stability' },
|
|
{ name: 'DIS', fn: scanDisabledInSchema, label: 'Disabled-In-Schema' },
|
|
{ name: 'COL', fn: scanCollision, label: 'Plugin Skill Collision' },
|
|
{ name: 'SKL', fn: scanSkillListing, label: 'Skill-Listing Budget' },
|
|
{ name: 'AGT', fn: scanAgentListing, label: 'Agent-Listing Budget' },
|
|
{ name: 'OST', fn: scanOutputStyle, label: 'Output-Style Validation' },
|
|
{ name: 'OPT', fn: scanOptimizationLens, label: 'Optimization Lens' },
|
|
];
|
|
|
|
/**
|
|
* Run all scanners against target path.
|
|
* @param {string} targetPath
|
|
* @param {object} [opts]
|
|
* @param {boolean} [opts.includeGlobal=false]
|
|
* @param {boolean} [opts.fullMachine=false] - Scan all known locations across the machine
|
|
* @param {boolean} [opts.suppress=true] - Apply suppressions from .config-audit-ignore
|
|
* @param {boolean} [opts.filterFixtures=true] - Exclude findings from test/example paths
|
|
* @param {boolean} [opts.excludeCache=true] - Drop stale ~/.claude/plugins/cache versions so findings reflect live config (B3)
|
|
* @returns {Promise<object>} Full envelope with all results
|
|
*/
|
|
// Exported for testing
|
|
export { isFixturePath, FIXTURE_DIR_NAMES };
|
|
|
|
export async function runAllScanners(targetPath, opts = {}) {
|
|
const start = Date.now();
|
|
const resolvedPath = resolve(targetPath);
|
|
|
|
// Default ON: stale cached plugin versions otherwise inflate token hotspots
|
|
// and CNF duplicate-hook findings with config that loads on zero turns. (B3)
|
|
const excludeCache = opts.excludeCache !== false;
|
|
|
|
// B8 — resolve the context window once and thread it to budget-aware scanners
|
|
// (SKL, CML). Undefined opts.contextWindow → conservative 200k anchor, which is
|
|
// byte-identical to the pre-B8 default; other scanners ignore the third arg.
|
|
// B8b — `--context-window auto` probes the configured model (settings cascade /
|
|
// ANTHROPIC_MODEL) so a 1M-tier host self-calibrates; unknown/unpinned → advisory.
|
|
let probedModel = null;
|
|
if (String(opts.contextWindow ?? '').trim().toLowerCase() === 'auto') {
|
|
probedModel = await resolveActiveModel(resolvedPath, { env: process.env });
|
|
}
|
|
const contextWindow = resolveContextWindow(opts.contextWindow, { model: probedModel });
|
|
|
|
// Shared file discovery — scanners reuse this
|
|
let discovery;
|
|
if (opts.fullMachine) {
|
|
const roots = await discoverFullMachinePaths();
|
|
discovery = await discoverConfigFilesMulti(roots, { excludeCache });
|
|
} else {
|
|
discovery = await discoverConfigFiles(resolvedPath, {
|
|
includeGlobal: opts.includeGlobal || false,
|
|
excludeCache,
|
|
});
|
|
}
|
|
|
|
const results = [];
|
|
|
|
for (const scanner of SCANNERS) {
|
|
const scanStart = Date.now();
|
|
try {
|
|
const result = await scanner.fn(resolvedPath, discovery, { contextWindow });
|
|
results.push(result);
|
|
const count = result.findings.length;
|
|
const label = opts.humanizedProgress
|
|
? `\`[${scanner.name}] ${scanner.label}\``
|
|
: `[${scanner.name}] ${scanner.label}`;
|
|
process.stderr.write(` ${label}: ${count} finding(s) (${Date.now() - scanStart}ms)\n`);
|
|
} catch (err) {
|
|
results.push({
|
|
scanner: scanner.name,
|
|
status: 'error',
|
|
files_scanned: 0,
|
|
duration_ms: Date.now() - scanStart,
|
|
findings: [],
|
|
counts: { critical: 0, high: 0, medium: 0, low: 0, info: 0 },
|
|
error: err.message,
|
|
});
|
|
const label = opts.humanizedProgress
|
|
? `\`[${scanner.name}] ${scanner.label}\``
|
|
: `[${scanner.name}] ${scanner.label}`;
|
|
process.stderr.write(` ${label}: ERROR — ${err.message}\n`);
|
|
}
|
|
}
|
|
|
|
// Filter findings from test fixtures / examples (unless disabled)
|
|
const shouldFilterFixtures = opts.filterFixtures !== false;
|
|
let fixtureFindings = [];
|
|
|
|
if (shouldFilterFixtures) {
|
|
for (const result of results) {
|
|
const active = [];
|
|
const fixture = [];
|
|
for (const f of result.findings) {
|
|
if (isFixturePath(f, resolvedPath)) {
|
|
fixture.push(f);
|
|
} else {
|
|
active.push(f);
|
|
}
|
|
}
|
|
if (fixture.length > 0) {
|
|
fixtureFindings.push(...fixture);
|
|
result.findings = active;
|
|
result.counts = { critical: 0, high: 0, medium: 0, low: 0, info: 0 };
|
|
for (const f of active) {
|
|
if (result.counts[f.severity] !== undefined) result.counts[f.severity]++;
|
|
}
|
|
}
|
|
}
|
|
if (fixtureFindings.length > 0) {
|
|
process.stderr.write(` ${fixtureFindings.length} finding(s) from test fixtures excluded\n`);
|
|
}
|
|
}
|
|
|
|
// Apply suppressions (unless disabled)
|
|
const shouldSuppress = opts.suppress !== false;
|
|
let suppressedFindings = [];
|
|
|
|
let deadSuppressions = [];
|
|
|
|
if (shouldSuppress) {
|
|
const { suppressions } = await loadSuppressions(resolvedPath);
|
|
// A pin that names no declared check can never match. Report it: a silently
|
|
// dead suppression leaves the user believing a finding is hidden when it is
|
|
// not (M-BUG-28).
|
|
deadSuppressions = unknownSuppressions(suppressions);
|
|
if (suppressions.length > 0) {
|
|
for (const result of results) {
|
|
const { active, suppressed } = applySuppressions(result.findings, suppressions);
|
|
suppressedFindings.push(...suppressed);
|
|
result.findings = active;
|
|
// Recalculate counts
|
|
result.counts = { critical: 0, high: 0, medium: 0, low: 0, info: 0 };
|
|
for (const f of active) {
|
|
if (result.counts[f.severity] !== undefined) result.counts[f.severity]++;
|
|
}
|
|
}
|
|
if (suppressedFindings.length > 0) {
|
|
process.stderr.write(` ${formatSuppressionSummary(suppressedFindings)}\n`);
|
|
}
|
|
}
|
|
}
|
|
|
|
const totalMs = Date.now() - start;
|
|
const env = envelope(resolvedPath, results, totalMs);
|
|
if (fixtureFindings.length > 0) {
|
|
env.fixture_findings = fixtureFindings;
|
|
}
|
|
if (suppressedFindings.length > 0) {
|
|
env.suppressed_findings = suppressedFindings;
|
|
}
|
|
// ux-rules rule 2: commands run scanners with `2>/dev/null`, so anything they
|
|
// must ACT on rides in the payload, never in a stderr-only warning. Added only
|
|
// when a dead pin exists, so a config without one is byte-identical.
|
|
if (deadSuppressions.length > 0) {
|
|
env.unknown_suppressions = deadSuppressions;
|
|
}
|
|
return env;
|
|
}
|
|
|
|
// --- CLI entry point ---
|
|
async function main() {
|
|
const args = process.argv.slice(2);
|
|
if (!requireValidArgs(args, ARG_SPEC)) return;
|
|
let targetPath = '.';
|
|
let outputFile = null;
|
|
let saveBaseline = false;
|
|
let baselinePath = null;
|
|
let contextWindow = null;
|
|
|
|
for (let i = 0; i < args.length; i++) {
|
|
if (args[i] === '--output-file' && args[i + 1]) {
|
|
outputFile = args[++i];
|
|
} else if (args[i] === '--context-window' && args[i + 1]) {
|
|
contextWindow = args[++i];
|
|
} else if (args[i] === '--save-baseline') {
|
|
saveBaseline = true;
|
|
} else if (args[i] === '--baseline' && args[i + 1]) {
|
|
baselinePath = args[++i];
|
|
} else if (args[i] === '--global') {
|
|
// handled below
|
|
} else if (args[i] === '--full-machine') {
|
|
// handled below
|
|
} else if (args[i] === '--no-suppress') {
|
|
// handled below
|
|
} else if (args[i] === '--include-fixtures') {
|
|
// handled below
|
|
} else if (args[i] === '--exclude-cache' || args[i] === '--no-exclude-cache') {
|
|
// handled below
|
|
} else if (args[i] === '--json') {
|
|
// handled below — explicit machine-readable mode (bypass humanizer)
|
|
} else if (args[i] === '--raw') {
|
|
// handled below — v5.0.0 verbatim mode (bypass humanizer)
|
|
} else if (!args[i].startsWith('-')) {
|
|
targetPath = args[i];
|
|
}
|
|
}
|
|
|
|
const includeGlobal = args.includes('--global');
|
|
const fullMachine = args.includes('--full-machine');
|
|
const suppress = !args.includes('--no-suppress');
|
|
const filterFixtures = !args.includes('--include-fixtures');
|
|
const excludeCache = !args.includes('--no-exclude-cache');
|
|
const jsonMode = args.includes('--json');
|
|
const rawMode = args.includes('--raw');
|
|
|
|
if (!(await requireTargetDir(resolve(targetPath)))) {
|
|
process.exitCode = 3;
|
|
return;
|
|
}
|
|
|
|
const humanizedProgress = !jsonMode && !rawMode;
|
|
process.stderr.write(humanizedProgress ? `Config-Audit v2.2.0\n` : `Config-Audit Scanner v2.2.0\n`);
|
|
process.stderr.write(`Target: ${resolve(targetPath)}\n`);
|
|
process.stderr.write(`Scope: ${fullMachine ? 'full-machine' : includeGlobal ? 'global' : 'project'}\n`);
|
|
process.stderr.write(`Fixtures: ${filterFixtures ? 'excluded' : 'included'}\n\n`);
|
|
|
|
const result = await runAllScanners(targetPath, {
|
|
includeGlobal,
|
|
fullMachine,
|
|
suppress,
|
|
filterFixtures,
|
|
excludeCache,
|
|
humanizedProgress,
|
|
contextWindow,
|
|
});
|
|
|
|
// Default mode runs the humanizer; --json and --raw bypass for v5.0.0 byte-equal output.
|
|
const output = (jsonMode || rawMode) ? result : humanizeEnvelope(result);
|
|
const json = JSON.stringify(output, null, 2);
|
|
|
|
if (outputFile) {
|
|
await writeOutputFile(outputFile, json, 'utf-8');
|
|
process.stderr.write(`\nResults written to ${outputFile}\n`);
|
|
} else {
|
|
process.stdout.write(json + '\n');
|
|
}
|
|
|
|
if (saveBaseline) {
|
|
const bPath = baselinePath || resolve(targetPath, '.config-audit-baseline.json');
|
|
// Always save baselines as raw v5.0.0-shape envelope so future humanizer
|
|
// changes don't trigger false-positive drift findings.
|
|
await writeFile(bPath, JSON.stringify(result, null, 2), 'utf-8');
|
|
process.stderr.write(`Baseline saved to ${bPath}\n`);
|
|
}
|
|
|
|
// Summary
|
|
const agg = result.aggregate;
|
|
process.stderr.write(`\n--- Summary ---\n`);
|
|
process.stderr.write(`Findings: ${agg.total_findings} (C:${agg.counts.critical} H:${agg.counts.high} M:${agg.counts.medium} L:${agg.counts.low} I:${agg.counts.info})\n`);
|
|
process.stderr.write(`Risk: ${agg.risk_score}/100 (${agg.risk_band})\n`);
|
|
process.stderr.write(`Verdict: ${agg.verdict}\n`);
|
|
|
|
// Exit code. Set, never process.exit(): stdout is written ASYNCHRONOUSLY when
|
|
// it is a pipe, and process.exit() discards whatever is still buffered. Piping
|
|
// this envelope used to yield truncated, unparseable JSON (246 854 bytes to a
|
|
// file vs 65 536 to a pipe) — a corruption that looks like a bad file, not a
|
|
// cut-off. Letting Node exit naturally drains stdout first.
|
|
process.exitCode = agg.verdict === 'FAIL' ? 2 : agg.verdict === 'WARNING' ? 1 : 0;
|
|
}
|
|
|
|
// Only run CLI if invoked directly
|
|
const isDirectRun = process.argv[1] && resolve(process.argv[1]) === resolve(new URL(import.meta.url).pathname);
|
|
if (isDirectRun) {
|
|
main().catch(err => {
|
|
process.stderr.write(`Fatal: ${err.message}\n`);
|
|
process.exitCode = 3;
|
|
});
|
|
}
|