From 69a4654dd7533774ac6171343d1366d5b72bb917 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Tue, 14 Jul 2026 10:45:57 +0200 Subject: [PATCH] =?UTF-8?q?docs(plan):=20v5.13=20plan=20=E2=80=94=20model?= =?UTF-8?q?=20routing,=20effort=20awareness,=20dead=20references?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Video-derived audit ('The Model Isn't the Moat') cross-checked against primary sources. Verified: orchestrator+cheap-worker pattern and 5-level per-agent effort tuning (official docs); rejected: the 'Fable low ≈ Opus high' chart claim (contradicted by Anthropic's own pages). Five chunks: register entries BP-MODEL-001/002, fix-engine xhigh hygiene, CA-CML dead prose references, feature-gap model/effort opportunity, planner-agent adversarial gate. Sequenced AFTER DEL B pipeline dogfood + batch release. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01CTontYwY5JGS4nL2AuiASy --- ...v5.13-model-routing-effort-deadref-plan.md | 157 ++++++++++++++++++ 1 file changed, 157 insertions(+) create mode 100644 docs/v5.13-model-routing-effort-deadref-plan.md diff --git a/docs/v5.13-model-routing-effort-deadref-plan.md b/docs/v5.13-model-routing-effort-deadref-plan.md new file mode 100644 index 0000000..844c73e --- /dev/null +++ b/docs/v5.13-model-routing-effort-deadref-plan.md @@ -0,0 +1,157 @@ +# v5.13 Plan — Model Routing, Effort Awareness, Dead References + +Derived from an external video analysis ("The Model Isn't the Moat", 2026-07) cross-checked +against primary sources and against what config-audit already encodes. Every claim acted on +here was verified against Anthropic's own docs; video-only claims are explicitly rejected below. + +## Source verification (done 2026-07-14) + +| Claim from video | Verdict | Source | +|---|---|---| +| Orchestrator + cheaper worker models is a supported, recommended pattern | VERIFIED | code.claude.com/docs/en/sub-agents ("Control costs by routing tasks to faster, cheaper models like Haiku"), code.claude.com/docs/en/workflows | +| Reasoning effort is tunable per settings / session / launch / **per-agent frontmatter** / SDK; levels `low, medium, high, xhigh, max` | VERIFIED | code.claude.com/docs/en/model-config#adjust-effort-level, sub-agents doc | +| Leaked Fable 5 system prompt principles ("partial recognition ≠ current knowledge"; "a prompt implying a file is present doesn't mean one is"; answer-first-then-one-question; tool-call scaling 1 / 3–5 / 5–10) | VERIFIED near-verbatim, **provenance unconfirmed** (third-party leak repo, not Anthropic-confirmed) | github.com/asgeirtj/system_prompts_leaks `Anthropic/claude-fable-5.md` | +| "Fable 5 on low ≈ Opus 4.8 on high, slightly higher cost/quality" score-vs-cost chart | **CONTRADICTED** — no such chart/statement on Anthropic's pages; GPT-5.5 appears only in a testimonial | anthropic.com/news/claude-fable-5-mythos-5 | + +## Already covered — no action + +| Video idea | Existing coverage | +|---|---| +| "Process is the moat" (config/harness > raw model) | The plugin's entire thesis | +| Extract repeated procedure into a skill | BP-MECH-003 + CA-OPT-001 (`optimization-lens-scanner.mjs:121`) | +| CLAUDE.md size/ownership discipline | BP-SIZE-001 + CA-CML line/size checks (`claude-md-linter.mjs:109/:120/:140`) | +| Check that referenced imports exist | CA-IMP broken `@import` (`lib/import-resolver.mjs:88`) — but **only** `@import`, see Chunk 3 | +| Plugin's own agents are model-routed | Agents table already pins sonnet for mechanical, opus for judgment | + +## Gaps → chunks + +Verified gap summary (register-mapper sweep, 2026-07-14): no scanner audits per-agent +`model:`/`effort:` frontmatter; effort has validity-check only (`settings-validator.mjs:195`), +no recommendation; no dead-reference check for prose file mentions in CLAUDE.md; no +adversarial/failure-mode requirement in planner-agent; `fix-engine.mjs:26` effort list +omits `xhigh` (settings-validator has all five). + +### Chunk 1 — Register entries: model routing + effort (dogfoods `knowledge-refresh`) + +Add to `knowledge/best-practices.json` via the knowledge-refresh flow (human-approved write): + +- **BP-MODEL-001** (`category: model-fit`): subagents doing mechanical/read-only work can pin a + cheaper model via `model:` frontmatter; orchestrator keeps the strong model. Source: + code.claude.com/docs/en/sub-agents → `confidence: confirmed`. +- **BP-MODEL-002** (`category: model-fit`): reasoning effort is tunable at five levels in five + places (settings `effortLevel`, `/effort`, `--effort`, per-agent `effort` frontmatter, SDK); + default `high`; higher effort is not universally better for simple tasks. Source: + code.claude.com/docs/en/model-config → `confidence: confirmed`. + +Schema per `scanners/lib/best-practices-register.mjs:42-102` (id/claim/confidence/source.url/ +source.verified required). This chunk doubles as the DEL B dogfood of `/config-audit +knowledge-refresh` (each chunk is also a plugin test). + +### Chunk 2 — fix-engine effort hygiene (tiny, TDD) + +`fix-engine.mjs:26` `VALID_EFFORT_LEVELS = ['low','medium','high','max']` — missing `xhigh`. +Consequence: nearest-match "fix" for a typo like `xhig` corrects to `high`, not `xhigh`. +Red test first: `findNearestEffortLevel('xhig') === 'xhigh'`. Align list with +`settings-validator.mjs:75`. + +### Chunk 3 — CA-CML dead prose references (new deterministic check) + +The strongest video-derived principle ("a prompt implying a file is present doesn't mean one +is") applied to CLAUDE.md quality: flag file paths mentioned in CLAUDE.md **prose** that do not +exist on disk. Today only `@import` targets are existence-checked; stale pointers like +`docs/foo.md` or `scripts/bar.sh` rot silently and burn always-loaded tokens on misdirection. + +Conservative v1 to control false positives: +- Only backtick-quoted tokens that look like relative file paths (contain `/` or a known + extension), resolved against the CLAUDE.md's own directory. +- Skip URLs, globs (`*`), placeholders (`{...}`, `<...>`, `$VAR`, `${...}`), absolute and + `~/` paths (machine-specific), and paths under `.gitignore`d dirs if cheap to determine. +- Severity: low. New CA-CML-NNN (verify next free NNN at implementation — IDs are dynamic). + +Byte-stability: follow [[adding-scanner-byte-stability]] steps for a new finding type in an +EXISTING scanner — frozen `tests/snapshots/v5.0.0/` must stay untouched; default-output +snapshots regenerate (`UPDATE_SNAPSHOT=1`) only if a fixture actually carries the new type; +humanizer step 7 (M-16/M-17 lessons): `TRANSLATIONS`-static entry for the new RAW title +(CML category mapping already exists). + +### Chunk 4 — feature-gap + inventory: model/effort awareness + +- New T3 opportunity check in `feature-gap-scanner.mjs`: authored agents + (`isAuthoredConfig`, M-BUG-13 lesson) where **no** agent sets `model:` or `effort:` → + "all agents inherit the session model/effort — mechanical agents can be routed cheaper / + effort-calibrated" citing BP-MODEL-001/002. Fires only when authored agents exist + (M-BUG-15 lesson: no enhancement-check on empty collections). Opportunity framing, never + failure — deliberate max-model setups are a valid choice; finding is suppressable + (`.config-audit-ignore`). +- `whats-active` / `manifest`: surface `model`/`effort` per agent in the inventory tables. +- Humanizer wiring step 7 for the new GAP finding; verify via direct `scan()` output, not the + self-suppressed default output ([[agent-commands-need-scanner-scoping]]). +- feat commit → docs-gate: README + CLAUDE.md diffs required. + +### Chunk 5 — planner-agent adversarial gate (do AFTER DEL B pipeline dogfood) + +Add a required "Failure modes" section to `agents/planner-agent.md`'s action-plan contract: +before an action plan is emitted, list what could go wrong per change + rollback trigger. +Mirrors the video's scoping-vs-devil's-advocate distinction; currently absent (zero +adversarial requirements in agents/). **Sequencing constraint:** DEL B step 3.2 judges +planner-agent against a fasit — change the agent only after that dogfood pass, or the +fasit target moves mid-evaluation. + +## Explicitly rejected (do not revisit without new evidence) + +1. **"Fable low ≈ Opus high" cost/score framing** — contradicted by Anthropic's own pages. + Never encode in register, copy, or recommendations. +2. **Tool-call-count effort scaling (1 / 3–5 / 5–10) as a register entry** — source is an + unconfirmed third-party leak → would be `confidence: inferred`, never surfaced. Not worth + carrying. +3. **Cost/intelligence/"taste" model-routing table generator** — subjective scores don't fit + the deterministic, provenance-gated design. BP-MODEL-001 covers the actionable core. +4. **"Fable mode" skill** — a user-level skill, not configuration auditing. Out of plugin scope. +5. **CLAUDE.md prose contradiction detection** — real gap (CA-CNF only covers + settings/permissions/hooks) but not video-driven; keep this plan surgical. + +## Known tension (named, not resolved here) + +The operator's own global policy is Opus/max-effort for ALL subagents, never Haiku — the +opposite of Chunk 4's recommendation. Both are legitimate: the docs-backed routing advice +optimizes cost at equal quality for the general user; the operator deliberately buys maximum +quality. Chunk 4's copy must respect that (opportunity framing + suppressability), and on this +machine the finding will simply be suppressed or ignored. The plugin serves general users; +the operator's setup is not the target of the check. + +## Verification + +Global, after every chunk: +- `node --test 'tests/**/*.test.mjs'` → green (baseline 1359/0; count grows with new tests) +- `git status --porcelain tests/snapshots/v5.0.0/` → empty (frozen untouched) +- TDD: red test exists and fails BEFORE each production change + +Per chunk: +- **C1:** `node --test tests/lib/best-practices-register.test.mjs` green; + `node scanners/knowledge-refresh-cli.mjs` classifies BP-MODEL-001/002 as fresh +- **C2:** `findNearestEffortLevel('xhig')` → `xhigh` (red first); `grep xhigh scanners/fix-engine.mjs` non-empty +- **C3:** fixture CLAUDE.md referencing `docs/missing.md` → finding fires; existing file / + URL / glob / placeholder / `~/` path → silent; humanized output has non-contradictory copy +- **C4:** authored-agent fixture without model/effort → opportunity fires; with either set → + silent; zero authored agents → silent; `userImpactCategory` ≠ `Other` end-to-end via direct scan() +- **C5:** dogfood plan run produces a Failure-modes section; DEL B 3.2 fasit judged BEFORE the change + +## Key assumptions (test before/at implementation) + +1. **Per-agent `effort` frontmatter is official** — verified 2026-07-14 against + code.claude.com/docs/en/sub-agents + /model-config; re-fetch both pages at implementation + (docs move). +2. **New finding type in existing scanner leaves frozen snapshots untouched** — M-17 precedent + says yes when no v5.0.0 fixture carries the type; verify by running the suite and inspecting + which snapshots differ before committing. +3. **Next free CA-CML/CA-GAP finding numbers** — IDs are built dynamically; grep tests + + snapshots for the highest used NNN before assigning. + +## Sequencing vs DEL B (one plan, no relitigation) + +This plan does NOT preempt the active DEL B sequence. Recommended order: +1. DEL B step 3 pipeline dogfood (`analyze → plan → implement → rollback`) — unchanged, next. +2. Batch patch release M-11→M-17 — unchanged. +3. v5.13 chunks 1→5 (chunk 1 doubles as the `knowledge-refresh` dogfood already queued in + DEL B "Resten"; chunk 5 explicitly waits for step 3.2). Release as minor v5.13.0 via + `release-plugin.mjs` when all chunks land.