open/config-audit
1
0
Fork 0
Code Issues Releases 1 Activity

fix(agents): an agent cannot promise what its tools forbid

Browse source 
R4 — `verifier-agent.md` carried two contracts at once: §Output Format said
"Append to: implementation-log.md", §Read-Only Guarantee said "never modifies
any files", and `tools:` granted only Read/Glob/Grep. Which one wins is
nondeterministic, and the loss is not a blocked write but a full-file Write on
the log the agent SHARES with the parallel implementer agents — the defect
`implement-log-append.test.mjs` exists to prevent, entering through the one file
that test does not read. The orchestrator half was already right
(`implement.md` Step 5 appends with Bash `>>` and tells the agent not to write),
so the fix is one-way: the agent file now returns its report inline and names
who appends it, and why a Write there would clobber.

The guard is the blanket invariant over the catalogue, not a fact about one
file: any agent whose tools grant no write capability must instruct no write AND
say positively that it returns findings inline. Tools and body are both read, so
stripping `Write` from any agent whose body still writes turns it red. Measured
1 of 7 agents carried the defect; the sweep asserts a write-tool-less agent
exists so the invariant cannot pass vacuously.

R6 — both "Required Frontmatter" rules were enforced by nothing, and the only
test reading agent frontmatter checked `name:` against a hand-written 3-of-7
list. The new guard takes nothing by hand: required keys are parsed from each
rule's own yaml block, the swept files from each rule's own `paths:`, the plugin
name from plugin.json — add a key to a rule and it is enforced next run. The
repo was already 7/7 and 21/21 compliant, so a green first run proves nothing:
all seven arms were seen red against a temporarily introduced defect one at a
time, including emptying a rule's yaml block to show the derivation is not
vacuous.

The color enum is deliberately NOT guarded: the official subagent docs list
red/blue/green/yellow/purple/orange/pink/cyan (no magenta) while issue 19292
lists magenta but neither purple nor orange, and this plugin ships both. Pinning
an unsettled set would encode an unverified premise rather than measure one.

Suite 1777 -> 1785, 0 fail. Frozen tests/snapshots/v5.0.0/ untouched.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MEMKCAyVzYTMzLaqPcULVr
This commit is contained in:
Kjell Tore Guttormsen 2026-08-20 22:59:51 +02:00
commit d5714261d1
4 changed files with 289 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

31
CLAUDE.md
View file

@ -83,6 +83,37 @@ Finding ID format: `CA-{SCANNER}-{NNN}` — e.g. `CA-CML-001`, `CA-SET-003`, `CA
**`{NNN}` names the CHECK, never the emission position (invariant).** `scanners/lib/finding-codes.mjs` is the single authority: every `finding()` call passes a `code`, and an undeclared or missing one **throws** — there is no counter fallback, because a fallback lets a half-converted scanner ship IDs that look valid. Adding a check takes the next free number for that scanner, never the next source-order position; removing one moves its key to `RETIRED_CODES` and its number is never reissued. IDs are therefore **not unique per finding** — one check failing in three files emits three findings sharing an ID, and `(id, file, line)` is the instance key that `fix-engine` verification uses. Frozen `v5.0.0` baselines mask IDs (`tests/helpers/mask-finding-ids.mjs`) instead of re-deriving them; the check→number pairs are pinned exhaustively in `tests/lib/finding-codes.test.mjs`.
**A file's contract cannot exceed its tools (invariant).** `agents/verifier-agent.md` said both
"Append to: implementation-log.md" (§Output Format) and "never modifies any files" (§Read-Only
Guarantee) while granting only `Read, Glob, Grep`. The failure mode is not a blocked write — it is
the agent improvising a full-file `Write` on the log it *shares* with the parallel implementer
agents, which is the exact defect `implement-log-append.test.mjs` exists to prevent, entering
through the one file that test does not read. `tests/agents/agent-write-contract.test.mjs` asserts
the blanket invariant over the whole catalogue rather than a fact about one file: an agent whose
`tools:` grant no write capability (`Write`/`Edit`/`NotebookEdit`/`Bash`) must instruct no file
write **and** state positively that it returns its findings inline. Both sides are read from the
file, so stripping `Write` from any agent whose body still instructs a write turns it red, and the
sweep asserts a write-tool-less agent exists so the invariant cannot pass vacuously. Measured
2026-08-20: **1 of 7** agents carried the defect; the other six all hold `Write`. The orchestrator
half was already correct — `implement.md` Step 5 appends with Bash `>>` and tells the agent not to
write — so this was a one-way fix in the agent file, not a two-sided one.
**Frontmatter contracts are derived, never listed (invariant).** `.claude/rules/agent-development.md`
and `.claude/rules/command-development.md` state which keys an agent/command MUST carry and that
agent colors are unique; nothing enforced any of it, and the only test that read agent frontmatter
checked `name:` against a hand-written 3-of-7 list. `tests/agents/frontmatter-contract.test.mjs`
takes nothing by hand: required keys are parsed from each rule's own fenced `yaml` block, the files swept
are resolved from each rule's own `paths:`, and the plugin name is read from
`.claude-plugin/plugin.json` — add a key to a rule and it is enforced on the next run with no test
edit, because a hand-kept list of what to sweep is a premise, not a measurement. The repo was
already compliant (7/7 agents, 21/21 commands, 0 duplicate colors), so a green first run proves
nothing: every arm was seen red against a temporarily introduced defect **one at a time**,
including emptying a rule's yaml block to show the derivation is not vacuous. The color **enum** is
deliberately *not* guarded — the official subagent docs list `red/blue/green/yellow/purple/orange/
pink/cyan` (no `magenta`) while issue #19292 lists `magenta` but neither `purple` nor `orange`, and
this plugin ships both `magenta` and `orange`; pinning an unsettled set in a guard would encode an
unverified premise rather than measure one.
## Conventions
Enforced conventions live in `.claude/rules/` (auto-loaded as project instructions):