Kjell Tore Guttormsen
09f817977c
Dogfooding `plan` + `implement` against a throwaway config surfaced one root
defect with many arms: the command templates treat consecutive fenced blocks as
one shell. They are not. Every ```bash fence runs as its own Bash call in its own
process, so a variable set in one block is empty in the next, and `$$` is a
different PID (measured: 21710 vs 22109).
The planner agent confirmed the sharpest arm at runtime, reporting that
`Mode: $RAW_FLAG` "arrived literally unsubstituted" — `--raw` was documented in
three command files while being functionally dead. A machine sweep found the same
root in 20 places across 9 files, well past the two the written fasit predicted:
- `$RAW_FLAG` read from non-shell agent prompts (analyze, plan, implement)
- `$TMPFILE` read across blocks (tokens, manifest, whats-active,
plugin-health) — each command could not read the file it had just written
- `$GLOBAL_FLAG` across blocks (fix)
- `$TODAY` never assigned in any block (campaign), passing
`--reference-date ""` to a write CLI in six places
- three `$$` temp paths handed to the Read tool (fix), which expands neither
All now follow the hardened drift.md pattern: a fixed literal path, or a
re-derivation inside each block that needs it.
Also fixed, all confirmed against ground truth rather than inferred:
- `implement` printed a rollback ID it never captured (the timestamp lived only
inside a command substitution) — the one message a user reads after a bad run
- `plan` reported "No analysis results found" for valid sessions, because Read
was pointed at a glob it cannot expand; now uses Glob and verifies the
analysis report exists before spawning the agent
- five phase commands wrote state.yaml with two of four required fields; since
the agent writes all four, a follow-up write silently deleted the rest
- `implement` promised rollback deletes created files; rollback deliberately
leaves them (M-BUG-26 still open) — the doc, not the engine, was wrong
- `implement` claimed a score delta with no pre-change measurement
- `verifier-agent` was told to write a report it has no tool to write
- dead `Task` tool name in always-loaded rule context; planner-agent template
demonstrated the inline file content its own line 110 forbids
The sweeps land as tests/commands/command-shell-state-shape.test.mjs, verified
red before the fix and proven able to fail by reintroducing the defect. Two
existing tests asserted the old bash-block mechanism rather than the intent and
were updated. Suite 1449/0; frozen v5.0.0 snapshots and all scanner code
untouched.
Not fixed, deliberately: neither command scope-gates its actions to the audit
target. The generated plan included an edit to a real file under ~/.claude,
outside the throwaway target, because the skill/agent scanners are machine-wide.
That is a design change, not a side fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0195udHgCcFegzm7ecKku2Yc
85 lines
3.4 KiB
Markdown
85 lines
3.4 KiB
Markdown
---
|
|
name: config-audit:plugin-health
|
|
description: Audit plugin configuration quality — validates structure, frontmatter, and cross-plugin coherence
|
|
argument-hint: "[plugin-path]"
|
|
allowed-tools: Read, Glob, Grep, Bash
|
|
model: sonnet
|
|
---
|
|
|
|
# Config-Audit: Plugin Health
|
|
|
|
Audit Claude Code plugin structure and quality — validates plugin.json, CLAUDE.md, command/agent frontmatter, and detects cross-plugin conflicts.
|
|
|
|
## Arguments
|
|
|
|
- `$ARGUMENTS` may contain a path to a specific plugin directory
|
|
- If omitted: scans all plugins in the marketplace root
|
|
- `--raw`: pass-through to the scanner; produces v5.0.0 verbatim envelope (bypasses the humanizer) for byte-stable diff tooling
|
|
|
|
## Implementation
|
|
|
|
### Step 1: Discover plugins and greet
|
|
|
|
If a specific path is given, scan only that plugin. Otherwise, find all plugins using Glob for `**/.claude-plugin/plugin.json`.
|
|
|
|
Tell the user:
|
|
|
|
```
|
|
## Plugin Health Check
|
|
|
|
Auditing {N} plugin(s) for structure, frontmatter quality, and cross-plugin conflicts...
|
|
```
|
|
|
|
### Step 2: Run scanner
|
|
|
|
Run silently for each plugin. Default mode writes a humanized JSON payload to `--output-file` where each PLH finding carries `userImpactCategory`, `userActionLanguage`, and `relevanceContext` alongside the v5.0.0 fields. `--raw` is passed through verbatim when present, and prints the byte-stable v5.0.0 envelope on stdout instead.
|
|
|
|
```bash
|
|
RAW_FLAG=""
|
|
if echo "$ARGUMENTS" | grep -q -- "--raw"; then RAW_FLAG="--raw"; fi
|
|
node ${CLAUDE_PLUGIN_ROOT}/scanners/plugin-health-scanner.mjs <path> --output-file /tmp/config-audit-plugin-health.json $RAW_FLAG 2>/dev/null; echo $?
|
|
```
|
|
|
|
Read `/tmp/config-audit-plugin-health.json` with the Read tool. Exit codes 0, 1 and 2 are normal; only 3 is a real error.
|
|
|
|
The payload carries three things the report needs:
|
|
|
|
- `plugins[]` — one row per plugin: `name`, `declaredName`, `commandCount`, `agentCount`, `findingCount`, `score`, `grade`. Use these for the table; never estimate a grade yourself.
|
|
- `cross_plugin_findings[]` — the namespace-collision and shared-command-name findings, already separated from the per-plugin ones (they also carry `crossPlugin: true` in `findings`).
|
|
- `findings[]` — every finding, humanized.
|
|
|
|
### Step 3: Present results
|
|
|
|
```markdown
|
|
### Plugin Health Report
|
|
|
|
| Plugin | Grade | Commands | Agents | Status |
|
|
|--------|-------|----------|--------|--------|
|
|
| {plugins[].name} | {plugins[].grade} ({plugins[].score}) | {plugins[].commandCount} | {plugins[].agentCount} | {Good/Issues found} |
|
|
| ... | ... | ... | ... | ... |
|
|
|
|
{If cross-plugin issues:}
|
|
#### Cross-Plugin Issues ({count})
|
|
| Issue | Plugins | Recommendation |
|
|
|-------|---------|----------------|
|
|
| ... | ... | ... |
|
|
|
|
{If findings:}
|
|
#### Findings by Plugin
|
|
|
|
**{plugin-name}** ({finding_count} findings):
|
|
1. [{userActionLanguage}] {humanized title} ({id}) — {humanized recommendation}
|
|
2. ...
|
|
```
|
|
|
|
Group findings within each plugin by `userImpactCategory` (e.g., "Configuration mistake", "Conflict") and lead each line with `userActionLanguage` ("Fix this now", "Fix soon", "Optional cleanup"). The humanizer already produced the plain-language `title`/`recommendation` strings — render them verbatim, do not paraphrase.
|
|
|
|
### Step 4: Suggest next steps
|
|
|
|
```
|
|
### What's next
|
|
|
|
- Fix structural issues based on recommendations above
|
|
- `/config-audit posture` — Full configuration posture assessment
|
|
- `/config-audit fix` — Auto-fix deterministic issues
|
|
```
|