open/config-audit
1
0
Fork 0
Code Issues Pull requests Releases Activity

fix(commands): stop writing files no later step can read, and payloads nobody asked for

Browse source 
Dogfooding the four read commands (posture, tokens, manifest, whats-active)
surfaced four defect classes, all in the seam between what a command template
promises and what the scanner behind it actually does.

M-BUG-40, fifth arm: posture wrote four temp files it could never read back.
#49 closed the $$/cross-block class in four commands, but posture survived it —
and so did the guard written to prevent exactly this. The guard compared each
$$ path to the block that created it, so a path written once and then read via
prose had no second occurrence to flag. Measured live: written from PID 21614,
read attempted from PID 23772. The invariant is now blanket (no $$ in any temp
path), which also caught fix.md and feature-gap.md.

M-BUG-43: 6 of 7 scanners write their payload to stdout when --raw/--json is
set even when --output-file was given, and the templates redirected only
stderr. Measured: posture 255 182 B, whats-active 35 922 B, drift 28 316 B,
manifest 23 825 B, tokens 8 768 B. fix and feature-gap never read the file they
wrote, so both recovered one letter grade from a quarter-megabyte dump.

tokens swallowed --json and --with-telemetry-recipe: documented, never
threaded, so --json returned the humanized payload where the docs promise
byte-stable v5.0.0 output.

M-BUG-42: manifest's render contract asked for {load}; the payload carries
loadPattern, so the Load column rendered blank for all 96 rows.

Four new tests (1449 -> 1453), each verified red before the fix. The
render-contract test checks {field} names against a live payload from a
fixture, since a hardcoded key list would drift. Frozen v5.0.0 snapshots
untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VGCk9o27eWo9uXLjkZTXEq
This commit is contained in:
Kjell Tore Guttormsen 2026-08-01 20:40:07 +02:00
commit acd1cf1248
13 changed files with 336 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

35
CHANGELOG.md
View file

@ -8,6 +8,41 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased]
### Fixed
- **`M-BUG-40`, fifth arm — `posture` wrote four temp files it could never read back.** #49 closed the
`$$`/cross-block class in four commands, but `posture.md` survived it, and so did the guard written
to prevent exactly this. The guard compared each `$$` path against the block that created it, so a
path written **once** and then read via prose ("Read the JSON output file using the Read tool") had
no second occurrence to flag. Measured live: the scanner wrote `/tmp/config-audit-posture-21614.json`
from PID 21614 while the next Bash call ran as PID 23772, and the read step had no path to hand the
Read tool at all. The invariant is now blanket — **no `$$` in any temp path in any command file** —
which also caught `fix.md` and `feature-gap.md`. All five sites now use fixed literal paths, repeated
literally in every step that needs them.
- **`M-BUG-43` — commands leaked whole JSON payloads into the transcript.** Every scanner except
`scan-orchestrator` writes its payload to stdout when `--raw`/`--json` is set, **even when
`--output-file` was given** — and the command templates redirected only stderr. Measured on a real
repo: `posture` 255 182 B, `whats-active` 35 922 B, `drift` 28 316 B, `manifest` 23 825 B, `tokens`
8 768 B. `fix` and `feature-gap` were the worst case: both ran posture with `--json`, never read the
file they wrote, and in practice recovered a single letter grade from a quarter-megabyte dump — in
the plugin that exists to cut token cost. 13 invocations across 10 command files now redirect stdout,
and the two commands that needed the data read it from their output file instead.
- **`tokens` swallowed two documented flags.** `--json` and `--with-telemetry-recipe` were listed as
recognized flags but never threaded into the CLI call, so `--json` returned the *humanized* payload
where the docs promised byte-stable v5.0.0 output (measured: 4/4 findings carried humanizer fields,
and the title read "Your file starts with content that changes between turns" instead of
"Cache-breaking volatile content at top of CLAUDE.md"), while `--with-telemetry-recipe` silently
produced no `telemetry_recipe_path` — the very flag the command's own closing tip recommends.
- **`M-BUG-42` — `manifest` asked for a field the scanner never emits.** The render contract used
`{load}`; the payload carries `loadPattern`. The Load column — which the command's own prose calls
the whole point of the view — would render blank for all 96 rows. `posture`'s headline had the same
shape (`{qualityAreaCount}`, never emitted) and now takes its count from the humanized scorecard
rather than `areas.length`, which counts a Feature Coverage row the table below deliberately excludes.
### Added
- Four command-template shape tests (1449 → 1453), each verified to fail before the fix: a blanket
`$$` ban, stdout-redirect discipline for any scanner invoked with `--output-file` in raw/json mode,
flag threading from prose to shell, and a render-contract test that checks every `{field}` against a
**live payload generated from a fixture** rather than a hardcoded key list, which would drift.
- **`M-BUG-40` — command templates assumed shell state survives between fenced blocks.** It does not:
every ```` ```bash ```` fence is executed as its own Bash call, in its own process. A variable
assigned in one block is empty in the next, and `$$` (the PID) differs between calls, so a

2
commands/config-audit.md
View file

@ -94,7 +94,7 @@ if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
# A placeholder in square brackets does not start with a dash, so both CLIs'
# arg loops would take it as the TARGET PATH instead of a flag.
SCOPE_FLAGS="" # e.g. SCOPE_FLAGS="--full-machine" or SCOPE_FLAGS="--global"
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/findings/scan-results.json $SCOPE_FLAGS $RAW_FLAG 2>/dev/null; node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/posture.json $SCOPE_FLAGS $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/findings/scan-results.json $SCOPE_FLAGS $RAW_FLAG >/dev/null 2>/dev/null; node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/posture.json $SCOPE_FLAGS $RAW_FLAG 2>/dev/null; echo $?
```
Use `--full-machine` for `full` scope, `--global` for `home` scope. For `repo` and `current`, pass the resolved path directly.

2
commands/discover.md
View file

@ -77,7 +77,7 @@ if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
# the orchestrator's arg loop takes it as the SCAN TARGET and silently scans a
# path that does not exist.
SCOPE_FLAGS="" # e.g. SCOPE_FLAGS="--full-machine" or SCOPE_FLAGS="--global"
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/findings/scan-results.json $SCOPE_FLAGS $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/findings/scan-results.json $SCOPE_FLAGS $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
Check exit code: 0/1/2 → normal. 3 → "Discovery encountered an error. Try a narrower scope."

2
commands/drift.md
View file

@ -50,7 +50,7 @@ Tell the user: **"Comparing current configuration against baseline..."**
```bash
RAW_FLAG=""
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
node ${CLAUDE_PLUGIN_ROOT}/scanners/drift-cli.mjs <path> --baseline <name> --output-file /tmp/config-audit-drift.json $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/drift-cli.mjs <path> --baseline <name> --output-file /tmp/config-audit-drift.json $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
Exit codes: `0` = stable/improving, `1` = degrading (both normal — present the result either way), `3` = a real error.

7
commands/feature-gap.md
View file

@ -42,7 +42,7 @@ Generate session ID (`YYYYMMDD_HHmmss`) if no active session exists.
mkdir -p ~/.claude/config-audit/sessions/{session-id}/findings 2>/dev/null
RAW_FLAG=""
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/posture.json $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file ~/.claude/config-audit/sessions/{session-id}/posture.json $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
If exit code is non-zero: "Assessment couldn't run. Check that the path exists and contains configuration files."
@ -159,9 +159,12 @@ Implementing 3 recommendations...
4. **Verify** by re-running posture:
```bash
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --json --output-file /tmp/config-audit-verify-$$.json 2>/dev/null
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --json --output-file /tmp/config-audit-verify.json >/dev/null 2>/dev/null
```
Use the Read tool on `/tmp/config-audit-verify.json` for the new `overallGrade`
and score — stdout is discarded on purpose (the same envelope is 255 KB).
### Step 7: Show results
```markdown

9
commands/fix.md
View file

@ -43,7 +43,7 @@ if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
# A placeholder in square brackets does not start with a dash, so the arg loop
# would take it as the scan/fix TARGET instead of a flag.
GLOBAL_FLAG=""
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <path> --output-file /tmp/config-audit-fix-scan.json $GLOBAL_FLAG $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/scan-orchestrator.mjs <path> --output-file /tmp/config-audit-fix-scan.json $GLOBAL_FLAG $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
Exit code 3 → tell user: "Scanner error. Try `/config-audit posture` to check your configuration."
@ -120,9 +120,14 @@ Read `/tmp/config-audit-fix-applied.json` with the Read tool to get applied/fail
Run a quick posture check to measure improvement:
```bash
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <path> --json --output-file /tmp/config-audit-fix-posture-$$.json 2>/dev/null
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <path> --json --output-file /tmp/config-audit-fix-posture.json >/dev/null 2>/dev/null
```
Use the Read tool on `/tmp/config-audit-fix-posture.json` and take `overallGrade`
and the score from there. That read is the only source for the numbers below:
`--json` prints the same envelope to stdout, but 255 KB of raw JSON in the
transcript to recover one grade is exactly the waste this plugin exists to find.
Present results:
```markdown

4
commands/manifest.md
View file

@ -41,7 +41,7 @@ Tell the user: **"Building token-source manifest for `<path>`..."**
```bash
RAW_FLAG=""
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
node ${CLAUDE_PLUGIN_ROOT}/scanners/manifest.mjs <path> --output-file /tmp/config-audit-manifest.json $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/manifest.mjs <path> --output-file /tmp/config-audit-manifest.json $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
**Exit code handling:**
@ -69,7 +69,7 @@ Use the Read tool on `/tmp/config-audit-manifest.json`. Extract `meta.repoPath`,
| Rank | Kind | Name | Source | Tokens | Load |
|------|------|------|--------|--------|------|
| 1 | {kind} | `<name>` | {source} | ~{estimated_tokens} | {load} |
| 1 | {kind} | `<name>` | {source} | ~{estimated_tokens} | {loadPattern} |
| ... | ... | ... | ... | ... | ... |
_Load column: **always** / **on-demand** / **external**. Append `°` when `derivationConfidence` is `inferred` (no primary-doc row pins it exactly)._

2
commands/plugin-health.md
View file

@ -37,7 +37,7 @@ Run silently for each plugin. Default mode writes a humanized JSON payload to `-
```bash
RAW_FLAG=""
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
node ${CLAUDE_PLUGIN_ROOT}/scanners/plugin-health-scanner.mjs <path> --output-file /tmp/config-audit-plugin-health.json $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/plugin-health-scanner.mjs <path> --output-file /tmp/config-audit-plugin-health.json $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
Read `/tmp/config-audit-plugin-health.json` with the Read tool. Exit codes 0, 1 and 2 are normal; only 3 is a real error.

30
commands/posture.md
View file

@ -42,27 +42,35 @@ Run silently — JSON goes to a file, the humanized scorecard prints to stderr (
```bash
RAW_FLAG=""
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file /tmp/config-audit-posture-$$.json $RAW_FLAG 2>/tmp/config-audit-posture-stderr-$$.txt; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --output-file /tmp/config-audit-posture.json $RAW_FLAG >/dev/null 2>/tmp/config-audit-posture-stderr.txt; echo $?
```
Both paths are fixed literals, repeated literally in every later step: each
```bash fence is its own process, so a `$$`-derived path could never be named
again. `>/dev/null` is required, not cosmetic — with `--raw` the scanner writes
the full envelope to stdout *as well as* the file (`posture.mjs:101`), which is
255 KB on a real repo.
If exit code is non-zero, tell the user: "Assessment couldn't complete. Check that the path exists and contains Claude Code configuration files."
If `--raw` was passed, treat the captured stderr as v5.0.0-shape verbatim text and present it as-is in a code block; skip the humanized rendering steps below.
### Step 3: Read and interpret results
Read the JSON output file using the Read tool. Extract:
Use the Read tool on `/tmp/config-audit-posture.json`. Extract:
- `overallGrade`, `opportunityCount`
- `areas[]` — each with `name`, `grade`, `score`, `findingCount`
- `scannerEnvelope.scanners[].findings[]` — when surfacing individual findings, prefer the humanizer-provided fields: `userImpactCategory` (e.g., "Configuration mistake", "Wasted tokens"), `userActionLanguage` (e.g., "Fix this now", "Fix soon", "Optional cleanup"), and `relevanceContext` ("affects-everyone", "affects-this-machine-only", "test-fixture-no-impact"). These let you group and prioritize without hardcoded severity-to-prose mappings.
Also Read the captured stderr file — its body is the humanized scorecard (grade headline, area-score block, opportunity hint). You can present it verbatim or interleave its lines with the JSON-driven table.
Also use the Read tool on `/tmp/config-audit-posture-stderr.txt` — its body is the humanized scorecard (grade headline, area-score block, opportunity hint). You can present it verbatim or interleave its lines with the JSON-driven table.
### Step 4: Present the scorecard
```markdown
**Health: {overallGrade}** | {qualityAreaCount} areas scanned
**Health: {overallGrade}** | (area count: take it from the humanized scorecard's
"N areas reviewed" line — do NOT use `areas.length`, which counts Feature
Coverage; the table below excludes it, so the two would disagree)
{Use the headline line from the humanized stderr scorecard — it carries grade-context prose already (e.g., " Health: A (97/100) — Healthy setup, only minor polish needed"). Do not re-derive an A/B/C/D prose table here; the humanizer owns that vocabulary.}
@ -93,19 +101,19 @@ Avoid hardcoded grade-to-prose ladders here — the humanized scorecard headline
Run drift comparison silently:
```bash
node ${CLAUDE_PLUGIN_ROOT}/scanners/drift-cli.mjs <target-path> --output-file /tmp/config-audit-posture-drift-$$.json 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/drift-cli.mjs <target-path> --output-file /tmp/config-audit-posture-drift.json 2>/dev/null; echo $?
```
Read that file and append a "Configuration Drift" section showing what changed since the last baseline. Both scanners report to stderr in default mode, which `2>/dev/null` discards — the payload is the only readable output.
Use the Read tool on `/tmp/config-audit-posture-drift.json` and append a "Configuration Drift" section showing what changed since the last baseline. Both scanners report to stderr in default mode, which `2>/dev/null` discards — the payload is the only readable output.
**If `--plugin-health` flag is present:**
Run plugin health scanner silently:
```bash
node ${CLAUDE_PLUGIN_ROOT}/scanners/plugin-health-scanner.mjs <target-path> --output-file /tmp/config-audit-posture-plh-$$.json 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/plugin-health-scanner.mjs <target-path> --output-file /tmp/config-audit-posture-plh.json 2>/dev/null; echo $?
```
Read that file and append a "Plugin Health" section, using its `plugins[]` rows for per-plugin grades.
Use the Read tool on `/tmp/config-audit-posture-plh.json` and append a "Plugin Health" section, using its `plugins[]` rows for per-plugin grades.
**If both flags:** Use `scanners/lib/report-generator.mjs` to produce a unified markdown report.
@ -113,5 +121,9 @@ Read that file and append a "Plugin Health" section, using its `plugins[]` rows
If a config-audit session exists, save results:
```bash
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --json --output-file ~/.claude/config-audit/sessions/<session-id>/posture.json 2>/dev/null
node ${CLAUDE_PLUGIN_ROOT}/scanners/posture.mjs <target-path> --json --output-file ~/.claude/config-audit/sessions/<session-id>/posture.json >/dev/null 2>/dev/null
```
This is a second scan on purpose: the session file stores the raw v5.0.0 shape,
while step 2 wrote the humanized one. `>/dev/null` matters most here — `--json`
sends the same envelope to stdout regardless of `--output-file`.

11
commands/tokens.md
View file

@ -47,9 +47,18 @@ if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
# loop would take it as the TARGET PATH instead of a flag.
GLOBAL_FLAG="" # --global
CACHE_FLAG="" # --no-exclude-cache
node ${CLAUDE_PLUGIN_ROOT}/scanners/token-hotspots-cli.mjs <path> --output-file /tmp/config-audit-tokens.json $GLOBAL_FLAG $CACHE_FLAG $RAW_FLAG 2>/dev/null; echo $?
JSON_FLAG="" # --json
TELEMETRY_FLAG="" # --with-telemetry-recipe
node ${CLAUDE_PLUGIN_ROOT}/scanners/token-hotspots-cli.mjs <path> --output-file /tmp/config-audit-tokens.json $GLOBAL_FLAG $CACHE_FLAG $JSON_FLAG $TELEMETRY_FLAG $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
`--json` and `--with-telemetry-recipe` must be threaded here, not just
documented: the CLI is what turns them on. `--json`/`--raw` make the payload
byte-stable v5.0.0 (the humanizer is skipped, `token-hotspots-cli.mjs:127`), and
`--with-telemetry-recipe` is what adds `telemetry_recipe_path`. `>/dev/null` is
required because those two modes also print the payload to stdout even with
`--output-file` set (`token-hotspots-cli.mjs:137`).
**Exit code handling:**
- `0` → continue
- `3` → tell user: "Couldn't analyse tokens. Check that the path exists and is a directory." Stop.

2
commands/whats-active.md
View file

@ -40,7 +40,7 @@ if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
# arg loop would take it as the TARGET PATH instead of a flag.
VERBOSE_FLAG="" # --verbose
SUGGEST_FLAG="" # --suggest-disables
node ${CLAUDE_PLUGIN_ROOT}/scanners/whats-active.mjs <path> --output-file /tmp/config-audit-whats-active.json $VERBOSE_FLAG $SUGGEST_FLAG $RAW_FLAG 2>/dev/null; echo $?
node ${CLAUDE_PLUGIN_ROOT}/scanners/whats-active.mjs <path> --output-file /tmp/config-audit-whats-active.json $VERBOSE_FLAG $SUGGEST_FLAG $RAW_FLAG >/dev/null 2>/dev/null; echo $?
```
**Exit code handling:**

222
tests/commands/command-output-discipline.test.mjs Normal file
View file

@ -0,0 +1,222 @@
/**
* Session #50 — command-template output-discipline tests.
*
* Dogfooding the four read commands (`posture`, `tokens`, `manifest`,
* `whats-active`) surfaced three defect classes that all live in the same
* seam: what a command template PROMISES versus what the scanner behind it
* actually does.
*
* 1. M-BUG-43 — a scanner invoked with `--raw`/`--json` writes the payload
* to stdout *even when `--output-file` is set*
* (`token-hotspots-cli.mjs:137`, `manifest.mjs:293`,
* `whats-active.mjs:60`, `posture.mjs:101`, `drift-cli.mjs`,
* `plugin-health-scanner.mjs`). The command templates redirect only
* stderr, so the payload lands in the transcript. Measured on this repo:
* posture 255 182 B, whats-active 35 922 B, drift 28 316 B,
* manifest 23 825 B, tokens 8 768 B. `.claude/rules/ux-rules.md` rule 1
* says NEVER show raw JSON — and a plugin that exists to cut token cost
* must not dump a quarter-megabyte to report one grade.
*
* 2. Flags documented in a command's prose that never reach any shell
* (`tokens.md`: `--json`, `--with-telemetry-recipe`). The scanner
* supports them; the template silently swallows them. Same "green is
* worse than an error" shape as the arg-sink class.
*
* 3. Render contracts that name a field the scanner never emits
* (`manifest.md` asked for `{load}`; the payload carries `loadPattern`),
* so the column the command's own prose calls the whole point renders
* blank for every row.
*
* Test 3 runs the real scanners against a fixture rather than asserting
* against a hardcoded key list — a hardcoded list drifts, a live payload
* cannot.
*/
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { readFile, readdir, mkdtemp, rm } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { execFile } from 'node:child_process';
import { promisify } from 'node:util';
import { resolve, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
const execFileAsync = promisify(execFile);
const __dirname = dirname(fileURLToPath(import.meta.url));
const ROOT = resolve(__dirname, '..', '..');
const COMMANDS_DIR = join(ROOT, 'commands');
const FIXTURE = join(ROOT, 'tests', 'fixtures', 'marketplace-medium');
async function commandFiles() {
const entries = await readdir(COMMANDS_DIR);
return entries.filter((e) => e.endsWith('.md')).sort();
}
/** Strip `#` comments so the tests never match their own explanatory prose. */
function stripComment(line) {
const h = line.indexOf('#');
return h === -1 ? line : line.slice(0, h);
}
/** Yield [lineNumber, line] for lines inside ```bash fences only. */
function bashLines(content) {
const out = [];
let open = null;
content.split('\n').forEach((line, i) => {
const m = line.match(/^\s*```(\w*)/);
if (m) {
open = open === null ? m[1] : null;
return;
}
if (open === 'bash') out.push([i + 1, line]);
});
return out;
}
/** Yield lines inside ```markdown fences — the render contracts. */
function markdownFenceLines(content) {
const out = [];
let open = null;
content.split('\n').forEach((line) => {
const m = line.match(/^\s*```(\w*)/);
if (m) {
open = open === null ? m[1] : null;
return;
}
if (open === 'markdown') out.push(line);
});
return out;
}
test('Output discipline: a scanner asked for --output-file never also prints to stdout', async () => {
// The rule is blanket and needs no per-scanner allowlist: if the command
// asked for the payload in a FILE, the same invocation must not let it reach
// the transcript. Invocations with no --output-file are out of scope — there
// the payload has nowhere else to go (e.g. `drift --save`, 112 B).
const violations = [];
for (const name of await commandFiles()) {
const content = await readFile(join(COMMANDS_DIR, name), 'utf-8');
for (const [lineNo, raw] of bashLines(content)) {
const line = stripComment(raw);
if (!/node \$\{CLAUDE_PLUGIN_ROOT\}\/scanners\//.test(line)) continue;
if (!line.includes('--output-file')) continue;
const rawish = /--json\b|--raw\b|\$RAW_FLAG/.test(line);
if (!rawish) continue;
// Mask `2>` so a stderr redirect is never mistaken for a stdout one.
const masked = line.replace(/2>/g, '2@');
if (/(^|\s)>\s*\S+/.test(masked)) continue;
violations.push(
`${name}:${lineNo} runs a scanner in raw/json mode with --output-file but never redirects stdout — the payload lands in the transcript`,
);
}
}
assert.deepEqual(violations, [], `Unredirected scanner payloads:\n${violations.join('\n')}`);
});
test('Flag threading: every flag documented in prose reaches a shell', async () => {
// A flag the user is told to pass must arrive somewhere. Two legitimate
// destinations exist: a bash fence (threaded to the scanner) or a documented
// control-flow branch handled by the model. The allowlist below carries only
// the second kind, each verified by reading the code:
// posture --drift / --plugin-health : step 5 runs *different* scanners
// fix --dry-run : dry-run is fix-cli's DEFAULT and the
// flag is an explicit no-op alias
// (`scanners/fix-cli.mjs:18-21`)
// knowledge-refresh --no-candidates : skips a web-poll step; never a CLI flag
// manifest/whats-active --json : served by `cat` of the payload, whose
// content is byte-identical to --raw
// (both scanners document --raw as a
// no-op; verified by diffing payloads)
const PROSE_HANDLED = new Set([
'posture.md:--drift',
'posture.md:--plugin-health',
'fix.md:--dry-run',
'knowledge-refresh.md:--no-candidates',
'manifest.md:--json',
'whats-active.md:--json',
]);
const violations = [];
for (const name of await commandFiles()) {
const content = await readFile(join(COMMANDS_DIR, name), 'utf-8');
const bash = bashLines(content).map(([, l]) => l).join('\n');
const documented = new Set();
for (const line of content.split('\n')) {
const m = line.match(/^\s*[-*]\s+`(--[a-z][a-z0-9-]*)`/);
if (m) documented.add(m[1]);
}
for (const flag of documented) {
if (bash.includes(flag)) continue;
if (PROSE_HANDLED.has(`${name}:${flag}`)) continue;
violations.push(`${name} documents \`${flag}\` but no bash fence ever passes it`);
}
}
assert.deepEqual(violations, [], `Flags dropped between docs and shell:\n${violations.join('\n')}`);
});
test('Render contract: every {field} in a render fence exists in the real payload', async () => {
// Derived values the model computes rather than reads. Kept deliberately
// short — every entry here is a field the render fence does NOT get from the
// scanner, so a long list would hollow out the test.
const DERIVED = new Set(['rank']);
const CASES = [
{ command: 'manifest.md', scanner: 'manifest.mjs', args: [] },
{ command: 'tokens.md', scanner: 'token-hotspots-cli.mjs', args: [] },
{
command: 'whats-active.md',
scanner: 'whats-active.mjs',
args: ['--verbose', '--suggest-disables'],
},
];
const dir = await mkdtemp(join(tmpdir(), 'ca-render-'));
try {
const violations = [];
for (const { command, scanner, args } of CASES) {
const outFile = join(dir, `${scanner}.json`);
await execFileAsync('node', [
join(ROOT, 'scanners', scanner), FIXTURE, '--output-file', outFile, ...args,
]);
const payload = JSON.parse(await readFile(outFile, 'utf-8'));
const keys = new Set();
const walk = (node) => {
if (Array.isArray(node)) node.slice(0, 5).forEach(walk);
else if (node && typeof node === 'object') {
for (const k of Object.keys(node)) {
keys.add(k);
walk(node[k]);
}
}
};
walk(payload);
const content = await readFile(join(COMMANDS_DIR, command), 'utf-8');
const refs = new Set();
for (const line of markdownFenceLines(content)) {
const re = /\{([a-zA-Z][a-zA-Z0-9_.]*)\}/g;
let m;
while ((m = re.exec(line)) !== null) refs.add(m[1]);
}
for (const ref of refs) {
if (DERIVED.has(ref)) continue;
if (ref.endsWith('.length')) {
// `{foo.length}` is a count of a collection — assert the collection
// itself exists, since that is the part the scanner owns.
const base = ref.slice(0, -'.length'.length).split('.').pop();
if (!keys.has(base)) {
violations.push(`${command}: {${ref}} — no \`${base}\` collection in the payload`);
}
continue;
}
const leaf = ref.split('.').pop();
if (!keys.has(leaf) && !keys.has(ref)) {
violations.push(`${command}: {${ref}} is never emitted by ${scanner}`);
}
}
}
assert.deepEqual(violations, [], `Render fields the scanner never emits:\n${violations.join('\n')}`);
} finally {
await rm(dir, { recursive: true, force: true });
}
});

29
tests/commands/command-shell-state-shape.test.mjs
View file

@ -151,6 +151,35 @@ test('Shell state: no $$ temp path is referenced outside the block that created
assert.deepEqual(violations, [], `Unresolvable $$ temp paths:\n${violations.join('\n')}`);
});
test('Shell state: no $$ appears in any temp path at all', async () => {
// Session #50 closed a blind spot in the test above: it only flags a `$$`
// path that is *referenced twice*, because it compares each occurrence to
// the block that created it. A path written once and then read via prose
// ("Read the JSON output file using the Read tool") has no second
// occurrence — so posture.md sat green through #49 while being unreadable
// by construction: the PID is never printed, so no later step can name the
// file. Measured live: written by PID 21614, read attempted from PID 23772.
//
// The invariant is therefore blanket, not relational: a command template
// must not put `$$` in a temp path at all. The hardened pattern from
// drift.md — one fixed literal path, repeated literally — is the only
// shape that survives the fence boundary.
const violations = [];
for (const name of await commandFiles()) {
const content = await readFile(resolve(COMMANDS_DIR, name), 'utf-8');
content.split('\n').forEach((raw, i) => {
const re = /(\/tmp\/[A-Za-z0-9._-]*\$\$[A-Za-z0-9._-]*)/g;
let m;
while ((m = re.exec(stripComment(raw))) !== null) {
violations.push(
`${name}:${i + 1} ${m[1]} — $$ differs per Bash call; no later step can name this file`,
);
}
});
}
assert.deepEqual(violations, [], `Unresolvable $$ temp paths:\n${violations.join('\n')}`);
});
test('Read tool: never asked to expand a glob', async () => {
const violations = [];
for (const name of await commandFiles()) {