test(fix-cli): close missed HOME-leak — isolate spawns + lock SKL/COL out

Devil's advocate gap-verification (read-only Workflow, 9 skeptics) refuted the
blanket "all closed" claim by finding fix-cli.test.mjs was the one CLI-spawning
test still reading the real ~/.claude. fix-cli runs the SKL skill-listing
scanner (HOME-scoped) even with includeGlobal:false, so its manual findings
include CA-SKL-001 on a dev machine but not in clean CI.

This directly corrects 325182d, which listed fix-cli.test.mjs as "Proven safe,
left as-is (output byte-identical real vs empty HOME — fixable findings are
project-local HKV/RUL/SET, never SKL/COL)". That reasoning predated SKL being
wired into scan-orchestrator (7bb2547/66433fe) and was false: real HOME yields
manual=6 (incl. CA-SKL-001), hermetic manual=5 (5230 vs 4798 bytes).

- wrap all 5 fix-cli spawns in hermeticEnv() (matches the other 11 CLI tests)
- add a regression lock: a project-scoped run must surface no CA-SKL/CA-COL
- redirect the --apply backup check to HERMETIC_HOME — the test was also
  writing backups into the real ~/.config-audit/backups on every run
- docs: stale "26 hook events" -> 28 (README:528, scanner-internals:73);
  hook-validator.mjs comment April -> June 2026 (functional count already 28)

Re-audited all 12 CLI-spawning test files: 11 hermetic-helper, manifest custom
HOME-env, post-edit-verify safe-by-construction (early-exit only, never reaches
scanners). HOME-leak class now actually closed. Suite 875/875, no snapshot drift.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ter3E2JSi1Khgmuf2kady8
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 18:50:20 +02:00
commit 5b28e84966
4 changed files with 24 additions and 6 deletions

View file

@ -525,7 +525,7 @@ Reference documents that inform the feature-gap agent and context-aware recommen
| `claude-code-capabilities.md` | Feature register: 18 config surfaces, Anthropic guidance, relevance table |
| `configuration-best-practices.md` | Per-layer best practices (cache-stability guidance) |
| `anti-patterns.md` | Common mistakes mapped to scanner IDs |
| `hook-events-reference.md` | All 26 hook events with details |
| `hook-events-reference.md` | All 28 hook events with details |
| `feature-evolution.md` | Feature timeline for staleness detection |
| `gap-closure-templates.md` | Config-specific templates for closing gaps |
| `prompt-cache-patterns.md` | Token-cost dynamics (prompt-cache patterns) — patterns powering the TOK scanner |

View file

@ -70,7 +70,7 @@ Scanner CLI: `node scanners/scan-orchestrator.mjs <path> [--global] [--full-mach
| `claude-code-capabilities.md` | Feature register: 18 config surfaces, Anthropic guidance, relevance table |
| `configuration-best-practices.md` | Per-layer best practices (v5: cache-stability guidance replaces Sonnet-era 200-line rule) |
| `anti-patterns.md` | Common mistakes mapped to scanner IDs |
| `hook-events-reference.md` | All 26 hook events with details |
| `hook-events-reference.md` | All 28 hook events with details |
| `feature-evolution.md` | Feature timeline for staleness detection |
| `gap-closure-templates.md` | Config-specific templates for closing gaps |
| `prompt-cache-patterns.md` | Token-cost dynamics (prompt-cache patterns) — patterns powering the TOK scanner |

View file

@ -13,7 +13,7 @@ import { resolve, dirname } from 'node:path';
const SCANNER = 'HKV';
/** All valid hook events (as of April 2026) */
/** All valid hook events (as of June 2026) */
const VALID_EVENTS = new Set([
'SessionStart', 'InstructionsLoaded', 'UserPromptSubmit',
'PreToolUse', 'PermissionRequest', 'PermissionDenied',

View file

@ -4,8 +4,9 @@ import { resolve, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { cp, rm, readFile, stat } from 'node:fs/promises';
import { mkdirSync, existsSync, readdirSync } from 'node:fs';
import { tmpdir, homedir } from 'node:os';
import { tmpdir } from 'node:os';
import { execFileSync } from 'node:child_process';
import { hermeticEnv, HERMETIC_HOME } from '../helpers/hermetic-home.mjs';
const __dirname = fileURLToPath(new URL('.', import.meta.url));
const FIXTURES = resolve(__dirname, '../fixtures');
@ -25,18 +26,31 @@ describe('fix-cli dry-run', () => {
const result = execFileSync('node', [FIX_CLI, FIXABLE, '--json'], {
encoding: 'utf-8',
timeout: 30000,
env: hermeticEnv(),
});
const output = JSON.parse(result);
assert.ok(Array.isArray(output.planned), 'Should have planned array');
assert.ok(output.planned.length > 0, 'Should have planned fixes');
assert.strictEqual(output.backupId, null, 'No backup in dry-run');
assert.ok(Array.isArray(output.manual), 'Should have manual array');
// Regression lock (HOME-leak class): a project-scoped fix run must surface only
// project-local findings, never HOME-scoped SKL/COL findings read from the
// developer's real ~/.claude. Without a hermetic HOME the spawned CLI picks up
// CA-SKL-001 from installed skills, making this test machine-dependent.
const homeScoped = output.manual.filter((m) => /^CA-(SKL|COL)-/.test(m.findingId));
assert.deepStrictEqual(
homeScoped.map((m) => m.findingId),
[],
'fix-cli leaked HOME-scoped findings — spawn is not HOME-isolated',
);
});
it('outputs valid JSON with --json flag', () => {
const result = execFileSync('node', [FIX_CLI, FIXABLE, '--json'], {
encoding: 'utf-8',
timeout: 30000,
env: hermeticEnv(),
});
assert.doesNotThrow(() => JSON.parse(result), 'Output should be valid JSON');
});
@ -57,13 +71,15 @@ describe('fix-cli --apply', () => {
const result = execFileSync('node', [FIX_CLI, tmpDir, '--apply', '--json'], {
encoding: 'utf-8',
timeout: 30000,
env: hermeticEnv(),
});
const output = JSON.parse(result);
assert.ok(output.applied.length > 0, 'Should have applied fixes');
assert.ok(output.backupId, 'Should have a backup ID');
// Verify backup exists
const backupDir = join(homedir(), '.config-audit', 'backups', output.backupId);
// Verify backup exists. The CLI runs with a hermetic HOME (see execFileSync env
// below), so its backups land under HERMETIC_HOME, not the developer's real home.
const backupDir = join(HERMETIC_HOME, '.config-audit', 'backups', output.backupId);
assert.ok(existsSync(backupDir), 'Backup directory should exist');
});
@ -71,6 +87,7 @@ describe('fix-cli --apply', () => {
execFileSync('node', [FIX_CLI, tmpDir, '--apply'], {
encoding: 'utf-8',
timeout: 30000,
env: hermeticEnv(),
});
// Check that settings.json was fixed
@ -83,6 +100,7 @@ describe('fix-cli --apply', () => {
const result = execFileSync('node', [FIX_CLI, tmpDir, '--apply', '--json'], {
encoding: 'utf-8',
timeout: 30000,
env: hermeticEnv(),
});
const output = JSON.parse(result);
assert.ok(Array.isArray(output.verified), 'Should have verified array');