Dogfooding `campaign` + `knowledge-refresh` against a throwaway ledger. Seven
defects, all found by running the commands as written and measuring, not by
reading them.
The headline pair only existed together. `knowledge-refresh` built
`STALE_AFTER="--stale-after 30"` and expanded it unquoted, trusting the shell to
split it in two. bash does; zsh — the macOS default, and what the Bash tool runs
here — does not. The CLI got one argv entry, matched no flag, and because it had
no unknown-flag branch, silently kept the 90-day default and reported "✓ All 14
register entries were re-verified within the last 90 days": a true-sounding
sentence about a threshold the user had just overridden. Fixing either half alone
leaves a silent wrong answer or a loud one; both are fixed, and a guard now
rejects any template that packs a flag and its value into one variable.
`knowledge-refresh` also read one register and wrote another: step 6 named an
unanchored `knowledge/best-practices.json` while the CLI reads
`${CLAUDE_PLUGIN_ROOT}/…`, which for an installed plugin is the cache. The
validation gate then ran the cached test against the cached register — green no
matter what was written. The two copies were byte-identical that day, which is
exactly why it was invisible.
`campaign` vouched for repos it could not read. `add /finnes/ikke` returned
`added` + exit 0; `refresh-tokens` then put the phantom in `swept[]` with a
0-token delta and left `skipped[]` empty, so the machine-wide bill claimed
coverage of three repos on a machine with two. Paths stay tracked — an unmounted
volume is a legitimate absence — but are reported as `addedUnverified`, and the
command names them.
Two class sweeps, both measured rather than assumed. `posture` was the single
scanner (1 of 14) whose fatal catch exited 1, which ux-rules defines as a normal
WARNING grade — a crash indistinguishable from a result. And all 13 payload
writers failed on a `--output-file` whose parent did not exist, which on a fresh
machine turned `campaign`'s first run into "the ledger may be corrupt"; they now
share `scanners/lib/write-output.mjs`.
Predicted breadth was too wide for the first time in five sessions: 6 of 8 CLIs
predicted to lack unknown-flag rejection, 4 measured. `drift` and `fix` already
reject them, via a construct the grep did not recognise — a grep matches an
implementation, the invariant is a behaviour. The sweep was rewritten to run each
CLI with a bogus flag and read the exit code.
Suite 1453 → 1469/0. Frozen snapshots untouched. `optimize-lens-cli` and
`token-hotspots-cli` share the unknown-flag defect and are deferred to the v5.14
argument-handling chunk with their positional-swallow arm; the count is recorded
in the guard rather than rounded down to zero.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012NHWjN8EnoxSqRvMTLK2NE
Dogfooding `plan` + `implement` against a throwaway config surfaced one root
defect with many arms: the command templates treat consecutive fenced blocks as
one shell. They are not. Every ```bash fence runs as its own Bash call in its own
process, so a variable set in one block is empty in the next, and `$$` is a
different PID (measured: 21710 vs 22109).
The planner agent confirmed the sharpest arm at runtime, reporting that
`Mode: $RAW_FLAG` "arrived literally unsubstituted" — `--raw` was documented in
three command files while being functionally dead. A machine sweep found the same
root in 20 places across 9 files, well past the two the written fasit predicted:
- `$RAW_FLAG` read from non-shell agent prompts (analyze, plan, implement)
- `$TMPFILE` read across blocks (tokens, manifest, whats-active,
plugin-health) — each command could not read the file it had just written
- `$GLOBAL_FLAG` across blocks (fix)
- `$TODAY` never assigned in any block (campaign), passing
`--reference-date ""` to a write CLI in six places
- three `$$` temp paths handed to the Read tool (fix), which expands neither
All now follow the hardened drift.md pattern: a fixed literal path, or a
re-derivation inside each block that needs it.
Also fixed, all confirmed against ground truth rather than inferred:
- `implement` printed a rollback ID it never captured (the timestamp lived only
inside a command substitution) — the one message a user reads after a bad run
- `plan` reported "No analysis results found" for valid sessions, because Read
was pointed at a glob it cannot expand; now uses Glob and verifies the
analysis report exists before spawning the agent
- five phase commands wrote state.yaml with two of four required fields; since
the agent writes all four, a follow-up write silently deleted the rest
- `implement` promised rollback deletes created files; rollback deliberately
leaves them (M-BUG-26 still open) — the doc, not the engine, was wrong
- `implement` claimed a score delta with no pre-change measurement
- `verifier-agent` was told to write a report it has no tool to write
- dead `Task` tool name in always-loaded rule context; planner-agent template
demonstrated the inline file content its own line 110 forbids
The sweeps land as tests/commands/command-shell-state-shape.test.mjs, verified
red before the fix and proven able to fail by reintroducing the defect. Two
existing tests asserted the old bash-block mechanism rather than the intent and
were updated. Suite 1449/0; frozen v5.0.0 snapshots and all scanner code
untouched.
Not fixed, deliberately: neither command scope-gates its actions to the audit
target. The generated plan included an edit to a real file under ~/.claude,
outside the throwaway target, because the skill/agent scanners are machine-wide.
That is a design change, not a side fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0195udHgCcFegzm7ecKku2Yc
Wires the user-facing surface of the machine-wide token roll-up (executable CLI
shipped in B2b-2, d664b70). The campaign report (Step 2) now renders rollUp.tokens:
the headline machine-wide always-loaded total split into the shared global layer
(paid once, every repo) + per-repo deltas, plus a ranked 'most expensive repos'
table. New mode 'refresh-tokens' (Step 6) documents the human-approved live sweep —
idempotent, skips unreadable repos, names any skipped so the bill's coverage stays
honest (Verifiseringsplikt).
Shapes documented carefully: rollUp.tokens.{sharedGlobal,perRepoDelta,machineWide}
are flat number maps; byRepo[] is the ranked {name,path,always,...} list. campaign-cli
already emitted rollUp.tokens (B2a) — no reader change; this is rendering only.
commands/campaign.md (73 lines) is the primary doc; README + CLAUDE.md campaign rows
get the matching one-line summary. Markdown-only → suite unchanged at 1198 green;
campaign command stays judgment-driven (not byte-stable).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes Block 4 (4b backlog + 4c export/execution). Asymmetric: plan
export is new testable code; execution is pure reuse of the existing
per-repo implement/rollback (no new execution machinery), per the plan's
"reuse existing backup/rollback".
Plan export ("planer følger arbeidsstedet"):
- scanners/lib/campaign-export.mjs (pure, now injected, 8 tests):
planExportPath(repo,sessionId) -> <repo>/docs/config-audit-plan-<sessionId>.md
(sessionId-keyed so same-day re-audits never collide);
buildPlanExportDocument({...,now}) -> provenance header + verbatim plan.
- scanners/campaign-export-cli.mjs (-cli, read-only by default, 10 tests):
--repo resolves the repo's linked session, reads its action-plan.md,
assembles the doc, emits {exportable,problems,targetPath,document}. Two
gates -> exit 1 advisory: no-session-linked / no-action-plan. Writes the
file ONLY under opt-in --write (byte-faithful copy; the LLM never re-types
a 200-line plan). --sessions-dir override for hermetic tests; exit 0/1/3.
Command: commands/campaign.md gains an `export <path>` mode (Step 6:
preview -> approve -> --write), then routes the user to the existing
/config-audit implement (backup + verify) + rollback + set-status
implemented. Nothing auto-written (Verifiseringsplikt).
Byte-stable: lib + -cli + command-doc only -> scanner count stays 15,
agents 7, commands 21 (export is a mode, not a new command), SC-5 +
backcompat suite untouched. suite 1150->1168. Block 4a (migrateLedger)
still deferred to the first breaking schema change.
Docs: CLAUDE.md section + badge 1150->1168/65->67 files; README badge +
campaign row + Testing prose (fixed stale 1055/59 -> true 1168/67).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the THIN machine-wide campaign surface (ledger + roll-up + status).
The write half of the campaign motor, mirroring how knowledge-refresh gates
register writes:
- scanners/campaign-write-cli.mjs (-cli → NOT a scanner; 11 tests): init/add/
set-status, each a thin wrapper over the invariant-enforcing lib transforms
(createLedger/addRepo/setRepoStatus) + saveLedger — path-normalization/dedup,
idempotent add, status-lifecycle guard and updatedDate bump never hand-rolled.
init refuses to clobber an existing/corrupt ledger (exit 1, file untouched);
add auto-inits + reports added vs skipped; set-status takes --findings/--session.
Deterministic: --reference-date is the only clock read, injected as `now`.
Exit 0=write, 1=advisory no-op, 3=error.
- commands/campaign.md (opus, no Web): thin orchestrator — always reports first
(read-only campaign-cli), then proposes init/add/set-status and invokes ONE
write-CLI subcommand only on explicit human approval (Verifiseringsplikt; never
hand-edits the JSON). add --discover finds git repos under a root to pick from.
Not byte-stable (own command, outside snapshot suite) like /config-audit optimize
+ knowledge-refresh. Both CLIs are -cli → scanner count stays 15, snapshot suite
untouched. commands 20→21, suite 1127→1138 (+11), test files 64→65.
Docs: CLAUDE.md §campaign-write-cli + command table + Testing badge; README
commands badge 20→21 + table row; help.md + router wiring.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
