docs(plan): v5.13 plan — model routing, effort awareness, dead references
Video-derived audit ('The Model Isn't the Moat') cross-checked against
primary sources. Verified: orchestrator+cheap-worker pattern and 5-level
per-agent effort tuning (official docs); rejected: the 'Fable low ≈ Opus
high' chart claim (contradicted by Anthropic's own pages). Five chunks:
register entries BP-MODEL-001/002, fix-engine xhigh hygiene, CA-CML dead
prose references, feature-gap model/effort opportunity, planner-agent
adversarial gate. Sequenced AFTER DEL B pipeline dogfood + batch release.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CTontYwY5JGS4nL2AuiASy
This commit is contained in:
parent
97867dbf37
commit
69a4654dd7
1 changed files with 157 additions and 0 deletions
157
docs/v5.13-model-routing-effort-deadref-plan.md
Normal file
157
docs/v5.13-model-routing-effort-deadref-plan.md
Normal file
|
|
@ -0,0 +1,157 @@
|
|||
# v5.13 Plan — Model Routing, Effort Awareness, Dead References
|
||||
|
||||
Derived from an external video analysis ("The Model Isn't the Moat", 2026-07) cross-checked
|
||||
against primary sources and against what config-audit already encodes. Every claim acted on
|
||||
here was verified against Anthropic's own docs; video-only claims are explicitly rejected below.
|
||||
|
||||
## Source verification (done 2026-07-14)
|
||||
|
||||
| Claim from video | Verdict | Source |
|
||||
|---|---|---|
|
||||
| Orchestrator + cheaper worker models is a supported, recommended pattern | VERIFIED | code.claude.com/docs/en/sub-agents ("Control costs by routing tasks to faster, cheaper models like Haiku"), code.claude.com/docs/en/workflows |
|
||||
| Reasoning effort is tunable per settings / session / launch / **per-agent frontmatter** / SDK; levels `low, medium, high, xhigh, max` | VERIFIED | code.claude.com/docs/en/model-config#adjust-effort-level, sub-agents doc |
|
||||
| Leaked Fable 5 system prompt principles ("partial recognition ≠ current knowledge"; "a prompt implying a file is present doesn't mean one is"; answer-first-then-one-question; tool-call scaling 1 / 3–5 / 5–10) | VERIFIED near-verbatim, **provenance unconfirmed** (third-party leak repo, not Anthropic-confirmed) | github.com/asgeirtj/system_prompts_leaks `Anthropic/claude-fable-5.md` |
|
||||
| "Fable 5 on low ≈ Opus 4.8 on high, slightly higher cost/quality" score-vs-cost chart | **CONTRADICTED** — no such chart/statement on Anthropic's pages; GPT-5.5 appears only in a testimonial | anthropic.com/news/claude-fable-5-mythos-5 |
|
||||
|
||||
## Already covered — no action
|
||||
|
||||
| Video idea | Existing coverage |
|
||||
|---|---|
|
||||
| "Process is the moat" (config/harness > raw model) | The plugin's entire thesis |
|
||||
| Extract repeated procedure into a skill | BP-MECH-003 + CA-OPT-001 (`optimization-lens-scanner.mjs:121`) |
|
||||
| CLAUDE.md size/ownership discipline | BP-SIZE-001 + CA-CML line/size checks (`claude-md-linter.mjs:109/:120/:140`) |
|
||||
| Check that referenced imports exist | CA-IMP broken `@import` (`lib/import-resolver.mjs:88`) — but **only** `@import`, see Chunk 3 |
|
||||
| Plugin's own agents are model-routed | Agents table already pins sonnet for mechanical, opus for judgment |
|
||||
|
||||
## Gaps → chunks
|
||||
|
||||
Verified gap summary (register-mapper sweep, 2026-07-14): no scanner audits per-agent
|
||||
`model:`/`effort:` frontmatter; effort has validity-check only (`settings-validator.mjs:195`),
|
||||
no recommendation; no dead-reference check for prose file mentions in CLAUDE.md; no
|
||||
adversarial/failure-mode requirement in planner-agent; `fix-engine.mjs:26` effort list
|
||||
omits `xhigh` (settings-validator has all five).
|
||||
|
||||
### Chunk 1 — Register entries: model routing + effort (dogfoods `knowledge-refresh`)
|
||||
|
||||
Add to `knowledge/best-practices.json` via the knowledge-refresh flow (human-approved write):
|
||||
|
||||
- **BP-MODEL-001** (`category: model-fit`): subagents doing mechanical/read-only work can pin a
|
||||
cheaper model via `model:` frontmatter; orchestrator keeps the strong model. Source:
|
||||
code.claude.com/docs/en/sub-agents → `confidence: confirmed`.
|
||||
- **BP-MODEL-002** (`category: model-fit`): reasoning effort is tunable at five levels in five
|
||||
places (settings `effortLevel`, `/effort`, `--effort`, per-agent `effort` frontmatter, SDK);
|
||||
default `high`; higher effort is not universally better for simple tasks. Source:
|
||||
code.claude.com/docs/en/model-config → `confidence: confirmed`.
|
||||
|
||||
Schema per `scanners/lib/best-practices-register.mjs:42-102` (id/claim/confidence/source.url/
|
||||
source.verified required). This chunk doubles as the DEL B dogfood of `/config-audit
|
||||
knowledge-refresh` (each chunk is also a plugin test).
|
||||
|
||||
### Chunk 2 — fix-engine effort hygiene (tiny, TDD)
|
||||
|
||||
`fix-engine.mjs:26` `VALID_EFFORT_LEVELS = ['low','medium','high','max']` — missing `xhigh`.
|
||||
Consequence: nearest-match "fix" for a typo like `xhig` corrects to `high`, not `xhigh`.
|
||||
Red test first: `findNearestEffortLevel('xhig') === 'xhigh'`. Align list with
|
||||
`settings-validator.mjs:75`.
|
||||
|
||||
### Chunk 3 — CA-CML dead prose references (new deterministic check)
|
||||
|
||||
The strongest video-derived principle ("a prompt implying a file is present doesn't mean one
|
||||
is") applied to CLAUDE.md quality: flag file paths mentioned in CLAUDE.md **prose** that do not
|
||||
exist on disk. Today only `@import` targets are existence-checked; stale pointers like
|
||||
`docs/foo.md` or `scripts/bar.sh` rot silently and burn always-loaded tokens on misdirection.
|
||||
|
||||
Conservative v1 to control false positives:
|
||||
- Only backtick-quoted tokens that look like relative file paths (contain `/` or a known
|
||||
extension), resolved against the CLAUDE.md's own directory.
|
||||
- Skip URLs, globs (`*`), placeholders (`{...}`, `<...>`, `$VAR`, `${...}`), absolute and
|
||||
`~/` paths (machine-specific), and paths under `.gitignore`d dirs if cheap to determine.
|
||||
- Severity: low. New CA-CML-NNN (verify next free NNN at implementation — IDs are dynamic).
|
||||
|
||||
Byte-stability: follow [[adding-scanner-byte-stability]] steps for a new finding type in an
|
||||
EXISTING scanner — frozen `tests/snapshots/v5.0.0/` must stay untouched; default-output
|
||||
snapshots regenerate (`UPDATE_SNAPSHOT=1`) only if a fixture actually carries the new type;
|
||||
humanizer step 7 (M-16/M-17 lessons): `TRANSLATIONS`-static entry for the new RAW title
|
||||
(CML category mapping already exists).
|
||||
|
||||
### Chunk 4 — feature-gap + inventory: model/effort awareness
|
||||
|
||||
- New T3 opportunity check in `feature-gap-scanner.mjs`: authored agents
|
||||
(`isAuthoredConfig`, M-BUG-13 lesson) where **no** agent sets `model:` or `effort:` →
|
||||
"all agents inherit the session model/effort — mechanical agents can be routed cheaper /
|
||||
effort-calibrated" citing BP-MODEL-001/002. Fires only when authored agents exist
|
||||
(M-BUG-15 lesson: no enhancement-check on empty collections). Opportunity framing, never
|
||||
failure — deliberate max-model setups are a valid choice; finding is suppressable
|
||||
(`.config-audit-ignore`).
|
||||
- `whats-active` / `manifest`: surface `model`/`effort` per agent in the inventory tables.
|
||||
- Humanizer wiring step 7 for the new GAP finding; verify via direct `scan()` output, not the
|
||||
self-suppressed default output ([[agent-commands-need-scanner-scoping]]).
|
||||
- feat commit → docs-gate: README + CLAUDE.md diffs required.
|
||||
|
||||
### Chunk 5 — planner-agent adversarial gate (do AFTER DEL B pipeline dogfood)
|
||||
|
||||
Add a required "Failure modes" section to `agents/planner-agent.md`'s action-plan contract:
|
||||
before an action plan is emitted, list what could go wrong per change + rollback trigger.
|
||||
Mirrors the video's scoping-vs-devil's-advocate distinction; currently absent (zero
|
||||
adversarial requirements in agents/). **Sequencing constraint:** DEL B step 3.2 judges
|
||||
planner-agent against a fasit — change the agent only after that dogfood pass, or the
|
||||
fasit target moves mid-evaluation.
|
||||
|
||||
## Explicitly rejected (do not revisit without new evidence)
|
||||
|
||||
1. **"Fable low ≈ Opus high" cost/score framing** — contradicted by Anthropic's own pages.
|
||||
Never encode in register, copy, or recommendations.
|
||||
2. **Tool-call-count effort scaling (1 / 3–5 / 5–10) as a register entry** — source is an
|
||||
unconfirmed third-party leak → would be `confidence: inferred`, never surfaced. Not worth
|
||||
carrying.
|
||||
3. **Cost/intelligence/"taste" model-routing table generator** — subjective scores don't fit
|
||||
the deterministic, provenance-gated design. BP-MODEL-001 covers the actionable core.
|
||||
4. **"Fable mode" skill** — a user-level skill, not configuration auditing. Out of plugin scope.
|
||||
5. **CLAUDE.md prose contradiction detection** — real gap (CA-CNF only covers
|
||||
settings/permissions/hooks) but not video-driven; keep this plan surgical.
|
||||
|
||||
## Known tension (named, not resolved here)
|
||||
|
||||
The operator's own global policy is Opus/max-effort for ALL subagents, never Haiku — the
|
||||
opposite of Chunk 4's recommendation. Both are legitimate: the docs-backed routing advice
|
||||
optimizes cost at equal quality for the general user; the operator deliberately buys maximum
|
||||
quality. Chunk 4's copy must respect that (opportunity framing + suppressability), and on this
|
||||
machine the finding will simply be suppressed or ignored. The plugin serves general users;
|
||||
the operator's setup is not the target of the check.
|
||||
|
||||
## Verification
|
||||
|
||||
Global, after every chunk:
|
||||
- `node --test 'tests/**/*.test.mjs'` → green (baseline 1359/0; count grows with new tests)
|
||||
- `git status --porcelain tests/snapshots/v5.0.0/` → empty (frozen untouched)
|
||||
- TDD: red test exists and fails BEFORE each production change
|
||||
|
||||
Per chunk:
|
||||
- **C1:** `node --test tests/lib/best-practices-register.test.mjs` green;
|
||||
`node scanners/knowledge-refresh-cli.mjs` classifies BP-MODEL-001/002 as fresh
|
||||
- **C2:** `findNearestEffortLevel('xhig')` → `xhigh` (red first); `grep xhigh scanners/fix-engine.mjs` non-empty
|
||||
- **C3:** fixture CLAUDE.md referencing `docs/missing.md` → finding fires; existing file /
|
||||
URL / glob / placeholder / `~/` path → silent; humanized output has non-contradictory copy
|
||||
- **C4:** authored-agent fixture without model/effort → opportunity fires; with either set →
|
||||
silent; zero authored agents → silent; `userImpactCategory` ≠ `Other` end-to-end via direct scan()
|
||||
- **C5:** dogfood plan run produces a Failure-modes section; DEL B 3.2 fasit judged BEFORE the change
|
||||
|
||||
## Key assumptions (test before/at implementation)
|
||||
|
||||
1. **Per-agent `effort` frontmatter is official** — verified 2026-07-14 against
|
||||
code.claude.com/docs/en/sub-agents + /model-config; re-fetch both pages at implementation
|
||||
(docs move).
|
||||
2. **New finding type in existing scanner leaves frozen snapshots untouched** — M-17 precedent
|
||||
says yes when no v5.0.0 fixture carries the type; verify by running the suite and inspecting
|
||||
which snapshots differ before committing.
|
||||
3. **Next free CA-CML/CA-GAP finding numbers** — IDs are built dynamically; grep tests +
|
||||
snapshots for the highest used NNN before assigning.
|
||||
|
||||
## Sequencing vs DEL B (one plan, no relitigation)
|
||||
|
||||
This plan does NOT preempt the active DEL B sequence. Recommended order:
|
||||
1. DEL B step 3 pipeline dogfood (`analyze → plan → implement → rollback`) — unchanged, next.
|
||||
2. Batch patch release M-11→M-17 — unchanged.
|
||||
3. v5.13 chunks 1→5 (chunk 1 doubles as the `knowledge-refresh` dogfood already queued in
|
||||
DEL B "Resten"; chunk 5 explicitly waits for step 3.2). Release as minor v5.13.0 via
|
||||
`release-plugin.mjs` when all chunks land.
|
||||
Loading…
Add table
Add a link
Reference in a new issue