Kjell Tore Guttormsen
9eba0f6169
A (RUL+CML durability) and E (PLH plugin-agent dead-config) implemented and committed. Foundation moved to v5.6 (A/E are additive and don't consume it). STATE + plan updated; v5.5.0 release-cut is a separate GO. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ter3E2JSi1Khgmuf2kady8
245 lines
16 KiB
Markdown
245 lines
16 KiB
Markdown
# v5.5+ Plan — Steering-Model Coverage
|
||
|
||
_Status: PLAN (awaiting GO per feature). Created 2026-06-20. Plans live next to the work (continuity rule)._
|
||
|
||
## Why this plan exists
|
||
|
||
A "seven ways to steer Claude Code" framing (CLAUDE.md / rules / skills / sub-agents /
|
||
hooks / output styles / mechanism-fit) was used as a **check-against** reference — not a
|
||
spec. Checking it against the live docs (`code.claude.com/docs`, map updated 2026-06-19)
|
||
both corrected the video **and** surfaced gaps in config-audit's own coverage.
|
||
|
||
The unifying insight: every steering mechanism has a different **loading model** and
|
||
**compaction-survival** profile, and the docs now publish both explicitly. config-audit
|
||
already audits *structure* well; it does **not** yet model *when a thing is loaded* or
|
||
*whether it survives compaction* — which is exactly the "wrong mechanism = wasted tokens /
|
||
unloaded instruction" axis. This plan closes that.
|
||
|
||
The small correctness items the check surfaced (HKV hook-events, RUL globs wording) are
|
||
**already landed** on `main` (`b6a62d7`) and bound for the pending **v5.4.1** patch. This
|
||
document covers only the **new functionality** (v5.5+), greenlit by the operator: A, B, C,
|
||
D, E.
|
||
|
||
---
|
||
|
||
## Verification log (Verifiseringsplikt)
|
||
|
||
Every claim this plan builds on, with source and status. `CONFIRMED` = stated in primary
|
||
docs; `REFUTED` = docs contradict; `UNVERIFIED` = not found in primary docs (do not assert).
|
||
|
||
| # | Claim | Status | Source |
|
||
|---|-------|--------|--------|
|
||
| V1 | Project-root CLAUDE.md + unscoped rules are **re-injected from disk after compaction** | CONFIRMED | `context-window.md#what-survives-compaction` |
|
||
| V2 | Rules with `paths:` frontmatter are **lost after compaction** until a matching file is read again | CONFIRMED | `context-window.md#what-survives-compaction` |
|
||
| V3 | Nested (subdir) CLAUDE.md is **lost after compaction** until a file in that dir is read again | CONFIRMED | `context-window.md`, `memory.md` |
|
||
| V4 | Path-scoped rules trigger on **Read** of a matching file (not every tool use) | CONFIRMED | `memory.md#path-specific-rules` |
|
||
| V5 | The only documented rule-scoping frontmatter field is **`paths`** (not `globs`) | CONFIRMED | `memory.md#path-specific-rules` |
|
||
| V6 | `.claude/rules/` is an **official** feature; all `.md` auto-discovered recursively; unscoped = always-on | CONFIRMED | `memory.md#organize-rules-with-claude/rules/` |
|
||
| V7 | Skill **name+description load every turn**; body loads on invoke | CONFIRMED | `skills.md`, `features-overview.md` |
|
||
| V8 | Skill listing description cap = **1,536 chars** default of `maxSkillDescriptionChars` (configurable, v2.1.105+) | CONFIRMED | `skills.md`, `settings.md` |
|
||
| V9 | Output styles **still exist** (not deprecated); only the standalone `/output-style` command was removed (v2.1.91) → use `/config` | CONFIRMED | `output-styles.md`, changelog |
|
||
| V10 | Output styles **modify the system prompt** (add to end); a **custom** style without `keep-coding-instructions: true` **removes built-in software-engineering instructions** | CONFIRMED | `output-styles.md` |
|
||
| V11 | `force-for-plugin: true` auto-applies a plugin's style, **overriding the user's `outputStyle`** | CONFIRMED | `output-styles.md` |
|
||
| V12 | CLAUDE.md is "**a user message after the system prompt**", whereas output styles are a system-prompt mechanism | CONFIRMED | `output-styles.md` (comparison table) |
|
||
| V13 | Output styles are "the most expensive" steering mechanism | **UNVERIFIED** — docs say cost rises but prompt-cache mitigates; no ranking | `output-styles.md` |
|
||
| V14 | Sub-agent runs in isolated context; only a summary returns | CONFIRMED | `sub-agents.md` |
|
||
| V15 | For **plugin** subagents, `hooks` / `mcpServers` / `permissionMode` frontmatter are **silently ignored** | CONFIRMED | `sub-agents.md` |
|
||
| V16 | Agent `skills:` frontmatter **preloads full skill content at startup** | CONFIRMED | `sub-agents.md` |
|
||
| V17 | Required agent frontmatter = **only `name` + `description`** | CONFIRMED | `sub-agents.md` |
|
||
| V18 | Hooks: **~30 events** (not "five"); hook scripts run outside context, but injected `additionalContext` IS saved to transcript (subject to compaction) | CONFIRMED | `hooks.md` |
|
||
| V19 | An official **mechanism-fit comparison table** exists (output style vs CLAUDE.md vs `--append-system-prompt` vs agents vs skills) | CONFIRMED | `output-styles.md` |
|
||
| U1 | hook event `post-session` (kebab) | **RESOLVED → REFUTED** (2026-06-20). The 2.1.169 changelog `post-session` is a **self-hosted-runner** workspace-lifecycle hook, NOT a settings.json hook event; absent from `hooks.md` (all 30 events PascalCase). **Removed** from HKV `VALID_EVENTS` in v5.4.1. | `hooks.md`, changelog 2.1.169 |
|
||
| U2 | A plugin.json `outputStyles` path-override key (PLH `SHADOWING_PATH_FIELDS`) | **RESOLVED → CONFIRMED** (2026-06-20). `outputStyles` (camelCase) is a documented plugin.json key in the **replaces** category (default `output-styles/` ignored when set). PLH is correct — no change. See `[[plugin-json-path-behavior]]`. | `plugins-reference.md` |
|
||
|
||
---
|
||
|
||
## The five features
|
||
|
||
Design constraints throughout: deterministic where possible; additive findings that do
|
||
**not** fire on the SC-5 snapshot fixture stay byte-stable (the v5.3/v5.4 pattern); every
|
||
new finding gets a stable title (fix-engine + humanizer couple on title — see `b6a62d7`).
|
||
|
||
### Foundation (prerequisite for B/A/C/E): enumeration
|
||
|
||
`active-config-reader.mjs` today enumerates CLAUDE.md cascade, hooks, MCP, skills, plugins —
|
||
but **not** rules, agents, or output styles (confirmed by the coverage scan). B, and parts
|
||
of A/C/E, need these enumerated with a `loadPattern` tag. This is the first build step.
|
||
|
||
- Add enumeration for: `.claude/rules/*.md` (+ `~/.claude/rules/`), agents (`.claude/agents/`,
|
||
`~/.claude/agents/`, plugin agents), output styles (`.claude/output-styles/`,
|
||
`~/.claude/output-styles/`).
|
||
- Tag each source kind with `loadPattern ∈ { always, on-demand, external }` and
|
||
`survivesCompaction ∈ { yes, no, n/a }` derived from V1–V3, V6, V7, V18.
|
||
- **Test:** fixture with root+subdir CLAUDE.md, scoped+unscoped rule, project agent, output
|
||
style → assert each is enumerated with the correct `loadPattern`/`survivesCompaction`.
|
||
|
||
### A — Durability / compaction-survival findings
|
||
|
||
**Problem.** A must-always-hold instruction placed where it does **not** survive compaction
|
||
(nested CLAUDE.md per V3, path-scoped rule per V2) silently disappears mid-session after a
|
||
`/compact`. The docs publish exactly which mechanisms survive (V1–V3) — so this is
|
||
doc-grounded, not a guess.
|
||
|
||
**Shape (additive, no new scanner — count stays 13).**
|
||
- **RUL** new finding `CA-RUL-NNN` — a **large** path-scoped rule (e.g. > 50 lines, reuse the
|
||
existing unscoped-size heuristic) carries an informational note: its content is **lost after
|
||
compaction** until a matching file is re-read (V2/V4). Severity **low** (awareness, not a bug).
|
||
- **CML** new finding `CA-CML-NNN` — a **nested** (non-root) CLAUDE.md of meaningful size
|
||
notes it is **not re-injected after compaction** (V3). Severity **low**.
|
||
- Keep it strictly **structural** (size + location), never semantic ("is this instruction
|
||
critical?") — that stays deterministic.
|
||
|
||
**Key assumption + test.** Discovery distinguishes root vs nested CLAUDE.md and scoped vs
|
||
unscoped rules. → Test with a fixture asserting the finding fires for nested/scoped+large and
|
||
**not** for root/unscoped.
|
||
|
||
**Byte-stability.** Additive; snapshot fixture has no nested CLAUDE.md / large scoped rule →
|
||
SC-5 byte-stable (re-verify).
|
||
|
||
### B — Load-pattern accounting in `tokens` + `manifest`
|
||
|
||
**Problem.** `manifest.mjs` and TOK rank all sources uniformly by `estimated_tokens`. A skill
|
||
listing entry (paid **every turn**, V7) ranks identically to a skill body (paid **on invoke**).
|
||
The docs give a precise per-mechanism loading + survival model (V1–V3, V7, V18) that we can
|
||
surface. This is the **core** of the "wrong mechanism = wasted tokens" thesis.
|
||
|
||
**Shape.**
|
||
- Add a `loadPattern` (and `survivesCompaction`) column to `manifest` output and to TOK
|
||
source records, derived from the foundation enumeration.
|
||
- Add an **always-loaded subtotal** to the manifest summary: "≈X tokens enter context every
|
||
turn before you type" (project-root CLAUDE.md + unscoped rules + skill/agent listing +
|
||
output style + MCP schemas), vs an on-demand subtotal.
|
||
- Optional follow-up: cost-benefit hint (a large `always` source dwarfing the on-demand pool).
|
||
|
||
**Key assumption + test.** Source-kind → `loadPattern` is a deterministic mapping. → Test:
|
||
manifest on a mixed fixture yields the correct always-subtotal; `--json` includes the new field.
|
||
|
||
**Byte-stability — RISK.** B changes the **manifest/tokens output format** → **not**
|
||
byte-stable. Requires snapshot regen (SC-5) and a decision on `--json` back-compat (add field
|
||
vs version the schema). This is the highest-format-risk item; sequence it deliberately.
|
||
|
||
### C — Output-style scanner (new `CA-OST`, count 13 → 14)
|
||
|
||
**Problem.** Output styles are live (V9) and the **most surprising** surface: a custom style
|
||
silently strips built-in software-engineering instructions (V10), and `force-for-plugin`
|
||
overrides the user's choice (V11). config-audit does not scan them at all today (GAP/SET/PLH
|
||
only touch adoption / a settings key / plugin folder-shadow).
|
||
|
||
**Shape (new scanner — output styles are a genuinely new file surface, so a new scanner is
|
||
warranted, unlike the additive v5.3/v5.4 work).**
|
||
- `CA-OST-001` — **custom** output style missing `keep-coding-instructions: true` → flags that
|
||
built-in SWE instructions are removed (V10). Severity **medium** (silently changes coding
|
||
behavior). The headline footgun.
|
||
- `CA-OST-002` — `force-for-plugin: true` in a project/user style → it overrides the user's
|
||
`outputStyle` (V11). Severity **low**.
|
||
- `CA-OST-003` — settings `outputStyle` value resolving to a non-existent style → dead config.
|
||
Severity **low/medium**. (Could live in SET instead; decide during build.)
|
||
- Active output-style token cost → feed B's `always` accounting (V10: system-prompt, every turn).
|
||
- **Not** asserting V13 ("most expensive") anywhere — unverified.
|
||
|
||
**Key assumption + test.** Discovery can find `.claude/output-styles/*.md` and
|
||
`~/.claude/output-styles/*.md` (new discovery type). → Test: fixture with a custom style
|
||
missing the flag fires `CA-OST-001`; a built-in style reference does not.
|
||
|
||
**Byte-stability.** New scanner; fixture-gated → SC-5 byte-stable if it does not fire on the
|
||
snapshot project (re-verify). Self-audit scanner-count badge moves 13 → 14 (update README +
|
||
CLAUDE.md inventory + the "count stays 13" lore).
|
||
|
||
### D — Mechanism-fit detector (heuristic — lowest precision, sequence last)
|
||
|
||
**Problem.** The docs publish a mechanism-fit table (V19) and the architectural distinction
|
||
that CLAUDE.md is a per-turn user message while output styles are system-prompt (V12). Common
|
||
mismatch: an "every time / before each" instruction in CLAUDE.md that should be a **hook**; a
|
||
path-specific instruction in root CLAUDE.md that should be a **path-scoped rule** (V4/V6).
|
||
|
||
**Shape (heuristic; additive to CML or a small new check).**
|
||
- `CA-???-001` — imperative lifecycle phrasing in CLAUDE.md (`every time`, `before each`,
|
||
`always run`, `after you …`, `whenever you …`) describing a tool/lifecycle action → suggest
|
||
a hook. Framed as **Missed opportunity**, severity **low**, never "Fix this now".
|
||
- `CA-???-002` — path-specific phrasing in **root** CLAUDE.md (`in src/**`, `for *.ts files`)
|
||
→ suggest a path-scoped rule.
|
||
|
||
**Key assumption + test — THE RISK.** Phrase heuristics must have low false-positive rate. →
|
||
Test with a positive corpus (clear mismatches → flagged) **and** a negative corpus (normal
|
||
project conventions → silent). Must be suppressible (`.config-audit-ignore`). If precision is
|
||
poor in testing, **ship behind a flag or defer** — do not ship a noisy heuristic.
|
||
|
||
**Byte-stability.** Additive; risk it fires on the snapshot's CLAUDE.md → check carefully,
|
||
may need snapshot regen.
|
||
|
||
### E — Agent-listing cost + plugin-agent dead-config (additive to PLH; feeds B)
|
||
|
||
**Problem.** Agents' name+description load for delegation (every turn), like skills — but
|
||
unlike skills they're unmetered. And `hooks`/`mcpServers`/`permissionMode` in a **plugin**
|
||
agent's frontmatter are silently ignored (V15) — dead config, and a false sense of
|
||
isolation/security if `permissionMode` is among them.
|
||
|
||
**Shape (additive to PLH).**
|
||
- `CA-PLH-NNN` — plugin agent declaring `hooks` / `mcpServers` / `permissionMode` → dead config
|
||
(V15). Severity **low** generally; **medium** for `permissionMode` (false security).
|
||
- Agent description cost → feed B's `always` accounting. **No hard cap claimed** — research
|
||
found no documented agent-description limit (unlike skills' 1,536, V8). Treat as cost signal,
|
||
not a limit.
|
||
- `skills:` preload in agent frontmatter → informational note: full skill content injected at
|
||
startup (V16).
|
||
|
||
**Key assumption + test.** PLH can tell a **plugin** agent from a user/project agent (V15
|
||
only applies to plugin subagents). → Test: plugin-agent fixture with `permissionMode` fires
|
||
medium dead-config; identical user-level agent does **not**.
|
||
|
||
---
|
||
|
||
## Dependencies & phased rollout
|
||
|
||
```
|
||
Foundation (enumeration: rules, agents, output styles + loadPattern/survivesCompaction)
|
||
├── A (durability) additive RUL+CML doc-grounded, low risk
|
||
├── E (plugin-agent dead) additive PLH doc-grounded, low risk
|
||
├── C (output-style) new CA-OST (13→14) doc-grounded, new surface
|
||
└── B (load-pattern) manifest/tokens format doc-grounded, FORMAT risk
|
||
D (mechanism-fit) heuristic independent, precision risk
|
||
```
|
||
|
||
**Recommended phasing** (chunk-work rule — session-sized, checkpoint STATE between):
|
||
|
||
- **v5.5.0 "steering-model I"** — A + E only. **DONE on main** (E `f75ed56`, A `f3aadb5`,
|
||
2026-06-20). Additive to RUL/CML/PLH, doc-grounded, byte-stable, count stays 13. **Foundation
|
||
was dropped from v5.5.0**: on inspection A/E are additive to scanners that already read the
|
||
files and do NOT consume the `active-config-reader` enumeration — that serves B, so it moves to
|
||
v5.6. (Release-cut of v5.5.0 is a separate GO step.)
|
||
- **v5.6.0 "steering-model II"** — **Foundation** (active-config-reader enumeration +
|
||
`loadPattern`/`survivesCompaction`) + B (load-pattern accounting, manifest format change +
|
||
snapshot regen + `--json` back-compat decision) + C (new `CA-OST`, count → 14). Also fix the
|
||
frontmatter-parser block-sequence limitation (inline `paths:` only today) as part of Foundation.
|
||
- **v5.7.0 (optional)** — D, only if the negative-corpus test shows acceptable precision;
|
||
otherwise behind a flag or dropped.
|
||
|
||
(Operator may prefer a single larger v5.5 — but B's format change and D's precision risk argue
|
||
for separating them from the low-risk additive batch.)
|
||
|
||
---
|
||
|
||
## Acceptance criteria (whole programme)
|
||
|
||
- [ ] Every new finding's primary claim traces to a `CONFIRMED` row above (no `UNVERIFIED`
|
||
assertions in user-facing text — V13 and U1/U2 must be resolved or omitted).
|
||
- [ ] `node --test 'tests/**/*.test.mjs'` green; README test badge == suite count
|
||
(`self-audit --check-readme` `passed: true`).
|
||
- [ ] `self-audit` stays **A / A**, no critical/high on the plugin itself.
|
||
- [ ] Each new finding has: a positive fixture (fires) **and** a negative fixture (silent).
|
||
- [ ] fix-engine + humanizer-data entries added/updated for any new finding title (title
|
||
coupling — see `b6a62d7`).
|
||
- [ ] SC-5 snapshot: byte-stable for additive findings (A/E); regenerated + reviewed for B
|
||
(and D if it touches the snapshot project).
|
||
- [ ] Scanner-count lore updated everywhere if C lands (README badge, CLAUDE.md inventory,
|
||
`docs/scanner-internals.md`, the "count stays 13" notes).
|
||
- [x] U1 (`post-session`) and U2 (plugin `outputStyles` key) resolved (2026-06-20): U1 refuted
|
||
→ removed from HKV in v5.4.1; U2 confirmed → PLH unchanged.
|
||
|
||
## Open decisions for the operator
|
||
|
||
1. **Phasing**: three releases (recommended) vs one v5.5 with everything?
|
||
2. **C placement**: new `CA-OST` scanner (count → 14) vs folding output-style checks into
|
||
SET+a file check (keeps 13)? (Recommend new scanner — distinct surface.)
|
||
3. **D**: build now (heuristic, precision-gated) vs defer until A/B/C/E prove the theme?
|
||
4. **B `--json`**: add `loadPattern` field in place (mild back-compat risk) vs schema version bump?
|