Kjell Tore Guttormsen
09f817977c
Dogfooding `plan` + `implement` against a throwaway config surfaced one root
defect with many arms: the command templates treat consecutive fenced blocks as
one shell. They are not. Every ```bash fence runs as its own Bash call in its own
process, so a variable set in one block is empty in the next, and `$$` is a
different PID (measured: 21710 vs 22109).
The planner agent confirmed the sharpest arm at runtime, reporting that
`Mode: $RAW_FLAG` "arrived literally unsubstituted" — `--raw` was documented in
three command files while being functionally dead. A machine sweep found the same
root in 20 places across 9 files, well past the two the written fasit predicted:
- `$RAW_FLAG` read from non-shell agent prompts (analyze, plan, implement)
- `$TMPFILE` read across blocks (tokens, manifest, whats-active,
plugin-health) — each command could not read the file it had just written
- `$GLOBAL_FLAG` across blocks (fix)
- `$TODAY` never assigned in any block (campaign), passing
`--reference-date ""` to a write CLI in six places
- three `$$` temp paths handed to the Read tool (fix), which expands neither
All now follow the hardened drift.md pattern: a fixed literal path, or a
re-derivation inside each block that needs it.
Also fixed, all confirmed against ground truth rather than inferred:
- `implement` printed a rollback ID it never captured (the timestamp lived only
inside a command substitution) — the one message a user reads after a bad run
- `plan` reported "No analysis results found" for valid sessions, because Read
was pointed at a glob it cannot expand; now uses Glob and verifies the
analysis report exists before spawning the agent
- five phase commands wrote state.yaml with two of four required fields; since
the agent writes all four, a follow-up write silently deleted the rest
- `implement` promised rollback deletes created files; rollback deliberately
leaves them (M-BUG-26 still open) — the doc, not the engine, was wrong
- `implement` claimed a score delta with no pre-change measurement
- `verifier-agent` was told to write a report it has no tool to write
- dead `Task` tool name in always-loaded rule context; planner-agent template
demonstrated the inline file content its own line 110 forbids
The sweeps land as tests/commands/command-shell-state-shape.test.mjs, verified
red before the fix and proven able to fail by reintroducing the defect. Two
existing tests asserted the old bash-block mechanism rather than the intent and
were updated. Suite 1449/0; frozen v5.0.0 snapshots and all scanner code
untouched.
Not fixed, deliberately: neither command scope-gates its actions to the audit
target. The generated plan included an edit to a real file under ~/.claude,
outside the throwaway target, because the skill/agent scanners are machine-wide.
That is a design change, not a side fix.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0195udHgCcFegzm7ecKku2Yc
104 lines
4.1 KiB
Markdown
104 lines
4.1 KiB
Markdown
---
|
|
name: config-audit:analyze
|
|
description: Phase 2 - Generate analysis report with hierarchy map and issue detection
|
|
allowed-tools: Read, Write, Edit, Glob, Grep, Agent
|
|
model: opus
|
|
---
|
|
|
|
# Config-Audit: Analysis (Phase 2)
|
|
|
|
Generate comprehensive analysis report from discovery findings.
|
|
|
|
## Prerequisites
|
|
|
|
- Must have completed Phase 1 (discovery)
|
|
- Findings must exist in `~/.claude/config-audit/sessions/{session-id}/findings/`
|
|
|
|
## Arguments
|
|
|
|
- `$ARGUMENTS` may contain `--raw` to forward to the analyzer agent's instructions; in `--raw` mode the agent renders v5.0.0 verbatim severity prefiks instead of humanized `userActionLanguage` urgency phrasing.
|
|
|
|
## Implementation
|
|
|
|
### Step 1: Resolve the session and verify its state
|
|
|
|
Find the session first — never guess which one `{session-id}` refers to:
|
|
|
|
```
|
|
Glob: ~/.claude/config-audit/sessions/*/state.yaml
|
|
Sort by modification time — the most recently modified session wins
|
|
```
|
|
|
|
Every `{session-id}` below is that session's id. Read its `state.yaml` using the Read tool and verify discovery phase completed. If the Glob returns nothing, or discovery hasn't completed, tell the user: "Discovery hasn't been run yet. Start with `/config-audit discover` or just run `/config-audit` for a full audit."
|
|
|
|
### Step 2: Tell the user what's happening
|
|
|
|
```
|
|
## Analyzing Configuration
|
|
|
|
Reading your scan findings and generating a detailed analysis report...
|
|
This includes hierarchy mapping, conflict detection, and prioritized recommendations.
|
|
```
|
|
|
|
### Step 3: Spawn analyzer agent
|
|
|
|
Tell the user: **"Generating analysis (this takes about 30 seconds)..."**
|
|
|
|
Check whether `$ARGUMENTS` contains `--raw`. Carry the answer yourself: the agent
|
|
prompt below is **not** a shell, so a variable assigned in a bash block cannot be
|
|
referenced from it. Substitute `{mode}` literally with `--raw` or `humanized`.
|
|
|
|
```
|
|
Agent(subagent_type: "config-audit:analyzer-agent")
|
|
model: sonnet
|
|
prompt: |
|
|
Analyze all findings in: ~/.claude/config-audit/sessions/{session-id}/findings/
|
|
Mode: {mode} ("humanized" = humanized; "--raw" = v5.0.0 verbatim severity prefiks)
|
|
Generate comprehensive report covering:
|
|
1. Executive summary with key metrics, grouped by userImpactCategory
|
|
2. Hierarchy map visualization
|
|
3. Conflict detection across config layers
|
|
4. CLAUDE.md quality assessment
|
|
5. Security issues (secrets, permissions)
|
|
6. Top 10 prioritized recommendations — lead each item with the
|
|
finding's userActionLanguage ("Fix this now," "Fix soon,"
|
|
"Fix when convenient," "Optional cleanup," "FYI") rather than
|
|
raw severity. The humanizer already replaced jargon-heavy
|
|
title/description/recommendation strings with plain-language
|
|
equivalents — render them verbatim, do not paraphrase.
|
|
Return the complete report as your final message. Do not write it
|
|
to a file — the orchestrating command saves it to the session directory.
|
|
```
|
|
|
|
### Step 4: Save the report
|
|
|
|
The agent returns the complete report as its final message — the Claude Code
|
|
subagent harness instructs agents not to write report/analysis files themselves,
|
|
so the command must persist it. Write the returned report verbatim (no edits,
|
|
no truncation) to `~/.claude/config-audit/sessions/{session-id}/analysis-report.md`
|
|
using the Write tool. Downstream phases (`plan`, `interview`, `status`) read this file.
|
|
|
|
### Step 5: Present summary
|
|
|
|
After saving the report, show a brief summary:
|
|
|
|
```markdown
|
|
### Analysis Complete
|
|
|
|
Report generated with:
|
|
- {N} conflicts detected
|
|
- {N} optimization opportunities
|
|
- {N} security notes
|
|
- Top recommendation: {first recommendation}
|
|
|
|
Full report: `~/.claude/config-audit/sessions/{session-id}/analysis-report.md`
|
|
|
|
### What's next
|
|
|
|
- **`/config-audit plan`** — Turn findings into a prioritized action plan
|
|
- **`/config-audit fix`** — Auto-fix deterministic issues right away
|
|
```
|
|
|
|
### Step 6: Update state
|
|
|
|
Update `state.yaml` with all four fields `.claude/rules/state-management.md` requires: `current_phase: "analyze"`, `completed_phases` (append `analyze` to the existing array — read it first), `next_phase: "plan"`, and `updated_at`. A write that names only two of the four silently deletes the other two.
|