open/config-audit
1
0
Fork 0
Code Issues Releases 1 Activity

test(snapshots): make byte/snapshot tests hermetic + re-seed baseline

Browse source 
The COL collision-scanner and the CLAUDE.md cascade resolve ~/.claude from
process.env.HOME (active-config-reader). Snapshot/byte CLIs were spawned with
the developer's real HOME, so they picked up installed plugins/skills and the
user CLAUDE.md — making the v5.0.0 + default-output snapshots machine- and
time-dependent. They were seeded 2026-05-01 with COL=1 (a real ~/.claude skill
collision) and drifted to COL=0 after the polyrepo split: 26 pre-existing
failures unrelated to Batch 1.

Fix (test-only, no production change):
- tests/helpers/hermetic-home.mjs — empty temp HOME, mirroring the pattern
  collision.test.mjs already uses for the COL unit test.
- 7 harnesses spawn CLIs (or call lint()) under the hermetic HOME, so output
  depends only on committed fixtures. Determinism verified across runs.
- Re-seeded all snapshots under hermetic HOME via SEED_SNAPSHOT/UPDATE_SNAPSHOT
  (added a SEED guard to the frozen v5.0.0 byte tests). Snapshots now reflect
  the fixture alone (COL=0, fixture-only activeConfig counts).
- Also re-seeded the unused env-aware snapshots (manifest/whats-active/
  plugin-health), which had baked dozens of real ~/.claude skill/plugin names
  into the committed repo — privacy cleanup.

Full suite: 812/812 green, stable across 3 runs.
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 12:26:00 +02:00
commit 8216fb4175
20 changed files with 326 additions and 3386 deletions
Download patch file Download diff file Expand all files Collapse all files

15
tests/raw-backcompat.test.mjs
View file

@ -23,8 +23,9 @@ import { resolve, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { readFile, access, mkdir } from 'node:fs/promises';
import { readFile, writeFile, access, mkdir } from 'node:fs/promises';
import { homedir } from 'node:os';
import { hermeticEnv, HERMETIC_HOME } from './helpers/hermetic-home.mjs';
const exec = promisify(execFile);
const __dirname = dirname(fileURLToPath(import.meta.url));
@ -32,8 +33,10 @@ const REPO = resolve(__dirname, '..');
const FIXTURE = resolve(REPO, 'tests/fixtures/marketplace-medium');
const SNAPSHOT_DIR = resolve(REPO, 'tests/snapshots/v5.0.0');
const STDERR_SNAPSHOT_DIR = resolve(REPO, 'tests/snapshots/v5.0.0-stderr');
const BASELINE_DIR = resolve(homedir(), '.config-audit/baselines');
// Baseline under the hermetic HOME so drift is isolated from ~/.config-audit.
const BASELINE_DIR = resolve(HERMETIC_HOME, '.config-audit/baselines');
const DEFAULT_BASELINE = resolve(BASELINE_DIR, 'default.json');
const SEED = process.env.SEED_SNAPSHOT === '1';
async function runCli(scriptPath, args) {
try {
@ -41,6 +44,7 @@ async function runCli(scriptPath, args) {
timeout: 60000,
cwd: REPO,
maxBuffer: 10 * 1024 * 1024,
env: hermeticEnv(),
});
return { stdout: stdout || '', stderr: stderr || '' };
} catch (err) {
@ -263,7 +267,12 @@ describe('SC-7 --raw posture stderr scorecard verbatim', () => {
it('posture --raw stderr matches tests/snapshots/v5.0.0-stderr/posture.txt (modulo Xms)', async () => {
const script = resolve(REPO, 'scanners/posture.mjs');
const { stderr } = await runCli(script, [FIXTURE, '--raw']);
const expected = await readFile(resolve(STDERR_SNAPSHOT_DIR, 'posture.txt'), 'utf8');
const stderrSnapshotPath = resolve(STDERR_SNAPSHOT_DIR, 'posture.txt');
if (SEED) {
await writeFile(stderrSnapshotPath, stderr, 'utf8');
return;
}
const expected = await readFile(stderrSnapshotPath, 'utf8');
assert.equal(
normalizeStderrDurations(stderr.trim()),
normalizeStderrDurations(expected.trim()),