fix(commands): stop answering questions the caller did not ask
Dogfooding `campaign` + `knowledge-refresh` against a throwaway ledger. Seven
defects, all found by running the commands as written and measuring, not by
reading them.
The headline pair only existed together. `knowledge-refresh` built
`STALE_AFTER="--stale-after 30"` and expanded it unquoted, trusting the shell to
split it in two. bash does; zsh — the macOS default, and what the Bash tool runs
here — does not. The CLI got one argv entry, matched no flag, and because it had
no unknown-flag branch, silently kept the 90-day default and reported "✓ All 14
register entries were re-verified within the last 90 days": a true-sounding
sentence about a threshold the user had just overridden. Fixing either half alone
leaves a silent wrong answer or a loud one; both are fixed, and a guard now
rejects any template that packs a flag and its value into one variable.
`knowledge-refresh` also read one register and wrote another: step 6 named an
unanchored `knowledge/best-practices.json` while the CLI reads
`${CLAUDE_PLUGIN_ROOT}/…`, which for an installed plugin is the cache. The
validation gate then ran the cached test against the cached register — green no
matter what was written. The two copies were byte-identical that day, which is
exactly why it was invisible.
`campaign` vouched for repos it could not read. `add /finnes/ikke` returned
`added` + exit 0; `refresh-tokens` then put the phantom in `swept[]` with a
0-token delta and left `skipped[]` empty, so the machine-wide bill claimed
coverage of three repos on a machine with two. Paths stay tracked — an unmounted
volume is a legitimate absence — but are reported as `addedUnverified`, and the
command names them.
Two class sweeps, both measured rather than assumed. `posture` was the single
scanner (1 of 14) whose fatal catch exited 1, which ux-rules defines as a normal
WARNING grade — a crash indistinguishable from a result. And all 13 payload
writers failed on a `--output-file` whose parent did not exist, which on a fresh
machine turned `campaign`'s first run into "the ledger may be corrupt"; they now
share `scanners/lib/write-output.mjs`.
Predicted breadth was too wide for the first time in five sessions: 6 of 8 CLIs
predicted to lack unknown-flag rejection, 4 measured. `drift` and `fix` already
reject them, via a construct the grep did not recognise — a grep matches an
implementation, the invariant is a behaviour. The sweep was rewritten to run each
CLI with a bogus flag and read the exit code.
Suite 1453 → 1469/0. Frozen snapshots untouched. `optimize-lens-cli` and
`token-hotspots-cli` share the unknown-flag defect and are deferred to the v5.14
argument-handling chunk with their positional-swallow arm; the count is recorded
in the guard rather than rounded down to zero.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012NHWjN8EnoxSqRvMTLK2NE
This commit is contained in:
parent
acd1cf1248
commit
caea8aca23
23 changed files with 742 additions and 48 deletions
91
tests/commands/command-flag-value-portability.test.mjs
Normal file
91
tests/commands/command-flag-value-portability.test.mjs
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
/**
|
||||
* Session #51 — command-template flag-value portability.
|
||||
*
|
||||
* Dogfooding `knowledge-refresh` surfaced a defect that only exists at the
|
||||
* seam between the command template and the shell that runs it:
|
||||
*
|
||||
* STALE_AFTER="--stale-after 30"
|
||||
* node …-cli.mjs --reference-date "$TODAY" $STALE_AFTER --output-file …
|
||||
*
|
||||
* The unquoted `$STALE_AFTER` is meant to split into TWO argv entries. Under
|
||||
* **bash** it does. Under **zsh** — the macOS default since Catalina, and the
|
||||
* shell the Bash tool actually runs on this machine — unquoted parameter
|
||||
* expansions are NOT word-split, so the CLI receives ONE argv entry with the
|
||||
* literal text `--stale-after 30`, matches no known flag, and (because the CLI
|
||||
* silently ignored unknown flags — see cli-unknown-flag-rejection.test.mjs)
|
||||
* falls back to the 90-day default while reporting success. Measured:
|
||||
*
|
||||
* $ STALE_AFTER="--stale-after 30"; set -- $STALE_AFTER; echo $#
|
||||
* 1 # zsh (bash prints 2)
|
||||
* → payload staleAfterDays: 90, exit 0, "✓ All 14 entries fresh"
|
||||
*
|
||||
* The user-facing knob was silently dead. Note the asymmetry that makes this
|
||||
* survivable elsewhere: an EMPTY unquoted expansion yields ZERO argv entries in
|
||||
* both shells, so the `FLAG=""` idiom used by ~25 other sites is portable. Only
|
||||
* a variable that can hold a flag AND its value is affected.
|
||||
*
|
||||
* The invariant asserted here is therefore about VALUE-carrying flags, not
|
||||
* about quoting in general: a command template must never depend on the shell
|
||||
* splitting one variable into a flag plus its argument. Pass the value through
|
||||
* its own quoted variable instead.
|
||||
*/
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { readFile, readdir } from 'node:fs/promises';
|
||||
import { resolve, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const COMMANDS_DIR = resolve(__dirname, '..', '..', 'commands');
|
||||
|
||||
async function commandFiles() {
|
||||
const entries = await readdir(COMMANDS_DIR);
|
||||
return entries.filter((e) => e.endsWith('.md')).sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Assignments whose right-hand side contains a flag followed by a value —
|
||||
* i.e. the value only reaches argv if the shell word-splits. Matches both
|
||||
* `X="--flag value"` and `X="--flag $(cmd)"`.
|
||||
*/
|
||||
const MULTIWORD_FLAG_ASSIGN = /^\s*([A-Z_][A-Z0-9_]*)=(["'])(--[a-z0-9-]+)[ \t]+\S.*\2\s*$/;
|
||||
|
||||
test('no command template builds a flag AND its value into one shell variable', async () => {
|
||||
const offenders = [];
|
||||
|
||||
for (const file of await commandFiles()) {
|
||||
const content = await readFile(resolve(COMMANDS_DIR, file), 'utf-8');
|
||||
content.split('\n').forEach((line, i) => {
|
||||
const m = line.match(MULTIWORD_FLAG_ASSIGN);
|
||||
if (m) offenders.push(`${file}:${i + 1} ${m[1]}=${m[2]}${m[3]} …${m[2]}`);
|
||||
});
|
||||
}
|
||||
|
||||
assert.deepEqual(
|
||||
offenders,
|
||||
[],
|
||||
'A variable holding "--flag value" only reaches argv correctly if the shell\n' +
|
||||
'word-splits an unquoted expansion. zsh does not. Pass the value in its own\n' +
|
||||
'quoted variable instead:\n' +
|
||||
' N=$(… extract …); [ -n "$N" ] && node cli.mjs --flag "$N"\n' +
|
||||
'Offending assignments:\n ' + offenders.join('\n '),
|
||||
);
|
||||
});
|
||||
|
||||
test('knowledge-refresh.md passes --stale-after with a quoted value', async () => {
|
||||
const content = await readFile(resolve(COMMANDS_DIR, 'knowledge-refresh.md'), 'utf-8');
|
||||
|
||||
assert.ok(
|
||||
!/\$STALE_AFTER\b(?!")/.test(content.replace(/"\$STALE_AFTER"/g, '')),
|
||||
'knowledge-refresh.md still expands a flag-carrying variable unquoted; under zsh the\n' +
|
||||
'threshold silently reverts to the 90-day default while the command reports success.',
|
||||
);
|
||||
|
||||
assert.match(
|
||||
content,
|
||||
/--stale-after "\$[A-Z_]+"/,
|
||||
'knowledge-refresh.md must pass the extracted threshold as its own quoted argument\n' +
|
||||
'(`--stale-after "$STALE_AFTER_DAYS"`), so no word-splitting is required.',
|
||||
);
|
||||
});
|
||||
|
|
@ -215,3 +215,38 @@ test('state.yaml: phase commands name all four fields the rule requires', async
|
|||
}
|
||||
assert.deepEqual(violations, [], `Incomplete state.yaml contracts:\n${violations.join('\n')}`);
|
||||
});
|
||||
|
||||
/**
|
||||
* Session #51 — the instruction that CAUSED the $TODAY defect must not outlive its fix.
|
||||
*
|
||||
* #49 fixed campaign.md by re-deriving `TODAY=$(date +%F)` inside each of the six write
|
||||
* blocks. But step 1 still carried the original prose — "Set a shared date stamp for any
|
||||
* write: `TODAY=$(date +%F)`" — which is not in a fence, cannot set anything, and directly
|
||||
* contradicts the six comments added below it. A template that argues with itself is not a
|
||||
* contract, and the next edit is the one that believes the wrong half.
|
||||
*
|
||||
* The guard above asserts fences; this one asserts that no PROSE line instructs the reader
|
||||
* to establish shell state for later blocks.
|
||||
*/
|
||||
test('no command template instructs shell state to be set outside a fence', async () => {
|
||||
const offenders = [];
|
||||
|
||||
for (const file of await commandFiles()) {
|
||||
const content = await readFile(resolve(COMMANDS_DIR, file), 'utf-8');
|
||||
const { lines, blockIndexOf } = parseFences(content);
|
||||
lines.forEach((line, i) => {
|
||||
if (blockIndexOf(i) !== -1) return; // inside a fence: that is where state belongs
|
||||
if (/`[A-Z_][A-Z0-9_]*=\$\(/.test(line) || /^\s*[A-Z_][A-Z0-9_]*=\$\(/.test(line)) {
|
||||
offenders.push(`${file}:${i + 1} ${line.trim()}`);
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
assert.deepEqual(
|
||||
offenders,
|
||||
[],
|
||||
'Prose that tells the reader to set a shell variable implies it survives to a later\n' +
|
||||
'block. It does not — every fence is its own process. Delete the instruction; the\n' +
|
||||
'blocks that need the value derive it themselves:\n ' + offenders.join('\n '),
|
||||
);
|
||||
});
|
||||
|
|
|
|||
91
tests/commands/knowledge-refresh-write-target.test.mjs
Normal file
91
tests/commands/knowledge-refresh-write-target.test.mjs
Normal file
|
|
@ -0,0 +1,91 @@
|
|||
/**
|
||||
* Session #51 — knowledge-refresh must write the register it actually read.
|
||||
*
|
||||
* The command reads the register through `knowledge-refresh-cli.mjs`, whose `REGISTER_PATH`
|
||||
* is anchored to the CLI module itself — i.e. `${CLAUDE_PLUGIN_ROOT}/knowledge/
|
||||
* best-practices.json`. For any installed plugin that is the marketplace cache; measured
|
||||
* live during this chunk:
|
||||
*
|
||||
* registerPath: …/.claude/plugins/cache/ktg-plugin-marketplace/config-audit/5.13.0/
|
||||
* knowledge/best-practices.json
|
||||
*
|
||||
* Step 6 then said: Edit `knowledge/best-practices.json` — an UNANCHORED relative path,
|
||||
* which resolves against whatever repo the user happens to be sitting in. Three consequences,
|
||||
* none of them visible at the time:
|
||||
*
|
||||
* 1. For a normal user, that path does not exist in their repo at all, so the approved
|
||||
* change either fails or drops a stray file into their project.
|
||||
* 2. In the plugin's own checkout it resolves to the working tree, so the command reads
|
||||
* one file and writes a different one — the refresh appears to do nothing, because the
|
||||
* CLI keeps reporting the cached copy's dates.
|
||||
* 3. Step 6.3's validation gate runs the plugin's own test file, which loads the register
|
||||
* via the same anchored `REGISTER_PATH`. So the gate validates the copy that was NOT
|
||||
* edited and passes no matter what was written — a guard that cannot see the file it
|
||||
* guards.
|
||||
*
|
||||
* The two register copies were byte-identical on the day this was found, which is exactly
|
||||
* why the defect was invisible to a casual dogfood run.
|
||||
*/
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { readFile } from 'node:fs/promises';
|
||||
import { resolve, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const FILE = resolve(__dirname, '..', '..', 'commands', 'knowledge-refresh.md');
|
||||
|
||||
/**
|
||||
* Scoped to lines that tell the model to MODIFY the register. Descriptive prose ("this
|
||||
* command keeps knowledge/best-practices.json current") and the closing `git commit`
|
||||
* suggestion name the file without acting on it, and a git path is correctly repo-relative.
|
||||
* The defect was specifically an unanchored path in an imperative write step.
|
||||
*/
|
||||
const WRITE_VERB = /\b(edit|write|append|overwrite|save)\b/i;
|
||||
|
||||
function unanchoredWriteTargets(content) {
|
||||
const out = [];
|
||||
content.split('\n').forEach((line, i) => {
|
||||
if (!WRITE_VERB.test(line)) return;
|
||||
for (const m of line.matchAll(/(\S*)knowledge\/best-practices\.json/g)) {
|
||||
if (!m[1].endsWith('${CLAUDE_PLUGIN_ROOT}/')) out.push(`${i + 1}: ${line.trim()}`);
|
||||
}
|
||||
});
|
||||
return out;
|
||||
}
|
||||
|
||||
test('the guard still catches the original defect', () => {
|
||||
const original = '1. Edit `knowledge/best-practices.json` — bump `source.verified`, update the `claim`/';
|
||||
assert.equal(
|
||||
unanchoredWriteTargets(original).length,
|
||||
1,
|
||||
'A narrowed guard must still fail on the text it was written for.',
|
||||
);
|
||||
});
|
||||
|
||||
test('every register path in knowledge-refresh.md is anchored to the plugin root', async () => {
|
||||
const content = await readFile(FILE, 'utf-8');
|
||||
const unanchored = unanchoredWriteTargets(content);
|
||||
|
||||
assert.deepEqual(
|
||||
unanchored,
|
||||
[],
|
||||
'A bare `knowledge/best-practices.json` resolves against the user\'s current repo, not\n' +
|
||||
'against the register the CLI actually read. Anchor every reference to\n' +
|
||||
'${CLAUDE_PLUGIN_ROOT}/ so the file that is read, written, and validated is one file:\n ' +
|
||||
unanchored.join('\n '),
|
||||
);
|
||||
});
|
||||
|
||||
test('the write step says where the register really lives', async () => {
|
||||
const content = await readFile(FILE, 'utf-8');
|
||||
assert.match(
|
||||
content,
|
||||
/marketplace|plugin cache|replaced on upgrade|plugin's own checkout/i,
|
||||
'Step 6 must state that the register is part of the installed plugin, so an approved\n' +
|
||||
'edit to a marketplace-installed copy is discarded by the next plugin upgrade and the\n' +
|
||||
'durable change belongs in the plugin\'s own checkout. Silently editing a cache\n' +
|
||||
'directory is the kind of write that looks successful and evaporates.',
|
||||
);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue