Kjell Tore Guttormsen
7df8e0d65b
Anthropic documents that Claude Opus 5 verifies its own work, and that telling it to double-check or to delegate verification to a subagent causes over-verification -- token cost with no quality gain. The general subtraction detector (BP-SUB-001) already surfaces those blocks for every user, with no model-awareness at all. `optimize --subtract --for-model <name>` adds the missing half. It ANNOTATES a subset of the candidates --subtract already produced; it is not a second detector and can never widen the candidate set. A second SUBTRACT_DETECTORS entry would have collided with BP-SUB-001 on de-dup, and a prose-only signal in the agent prompt would have been untestable. There is no auto-detection, by measurement rather than omission: a CLAUDE.md has no frontmatter and no resolvable target model, and this operator's own `route` skill deliberately runs a different model per session -- the same file is read by whichever model comes next. So the model is named, and the citation is reported as conditional everywhere a human sees it (agent report copy, and the Step 7a listing that is the last surface before an approval file). Precision comes from the TARGET, not the verb list. Measured across the 409-file corpus: 392 BP-SUB-001 candidates, 31 (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target. Two independent raw-text greps found 0 as well, so the zero is the corpus rather than an over-narrow regex. Those 31 verb-only blocks -- "sjekk relevante config-filer", "Type-sjekk: pyright", "To verify plugin functionality" -- are exactly the false positives a verb-only version would have produced, which is BP-JUDG-001's 7/7 failure arriving one lens over. The numbers live in the register entry's note and are pinned by a test, because a session that cannot see the measurement reads the zero as a broken detector and loosens it. `recognized` is reported separately from `matchedCount`: a typo'd model name and a genuinely clean config both yield zero, and without the distinction the CLI would report a silent no-op as good news. Dogfooded on the real machine -- `opus-5` gives recognized:true/matchedCount:0, `oppus5` gives recognized:false. source.published is absent because the guide carries no visible publish date; its absence is asserted so a later session does not invent one to match the other entries' shape. Both quoted sentences were verified verbatim 2026-08-12. The payload stays additive -- forModel and per-candidate modelScope appear only under the flag, so a plain --subtract run is byte-identical to before (asserted on the serialized bytes, since a key set to undefined passes a shallow check). Suite 1724 -> 1752 (+28). The one remaining failure is the pre-existing drift-cli --output-file crash, untouched by this work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BRuXt6tZyowi8QYNKLSHQm
302 lines
19 KiB
JSON
302 lines
19 KiB
JSON
{
|
|
"version": 1,
|
|
"note": "Machine-readable best-practices register. SOURCE OF TRUTH for the optimization lens (v5.7 CA-OPT). Human-readable mirror lives in knowledge/*.md. Every entry is provenance-stamped (source.url + source.verified; optional corroborating sources[] with published dates feed the evidence-age freshness rule; source.supersededBy marks a source replaced by a newer one) and carries a confidence; only CONFIRMED claims are consumed user-facing (Verifiseringsplikt). Curated manually + by /config-audit knowledge-refresh (human-approved). Seeded from docs/v5.5-steering-model-plan.md V-rows + the Anthropic 'Steering Claude Code' blog.",
|
|
"entries": [
|
|
{
|
|
"id": "BP-MECH-001",
|
|
"claim": "Lifecycle automation phrased as an instruction in CLAUDE.md (\"every time\", \"before each\", \"always run X after Y\") should be a hook — a behavior the model chooses to follow is not deterministic.",
|
|
"mechanism": "hook",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Move the behavior to a PreToolUse/PostToolUse/Stop hook so it runs deterministically, outside the model's discretion.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "mechanism-fit",
|
|
"lensCheck": "claude-md-lifecycle-phrasing",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-MECH-002",
|
|
"claim": "A file- or path-specific constraint placed in root CLAUDE.md or an unscoped rule should be a path-scoped rule (paths: frontmatter), so it loads only when a matching file is touched.",
|
|
"mechanism": "rule",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Move it to .claude/rules/ with a paths: frontmatter; unscoped instructions cost tokens every turn whether relevant or not.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "mechanism-fit",
|
|
"lensCheck": "unscoped-path-specific-instruction",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-MECH-003",
|
|
"claim": "A multi-step procedure (deploy/release checklist) in CLAUDE.md should be a skill — CLAUDE.md is for facts Claude should hold all the time; procedures belong in skills.",
|
|
"mechanism": "skill",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Extract the procedure into .claude/skills/; its body then loads only on invoke instead of every turn.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "mechanism-fit",
|
|
"lensCheck": "procedure-in-claude-md",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-MECH-004",
|
|
"claim": "An absolute prohibition phrased as a \"never do X\" instruction is the wrong tool; for something that absolutely must not happen, use permissions or a PreToolUse hook.",
|
|
"mechanism": "permission",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Enforce hard prohibitions via permission deny rules or a PreToolUse hook (exit code 2 denies the call), not prose instructions.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "mechanism-fit",
|
|
"lensCheck": "never-instruction",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-MECH-005",
|
|
"claim": "A custom output style without keep-coding-instructions: true removes Claude Code's built-in software-engineering instructions when active.",
|
|
"mechanism": "output-style",
|
|
"appliesTo": "output-style",
|
|
"recommendation": "Set keep-coding-instructions: true, or prefer a built-in style (Explanatory / Learning / Proactive) before writing a custom one.",
|
|
"confidence": "confirmed",
|
|
"severity": "medium",
|
|
"category": "mechanism-fit",
|
|
"lensCheck": "CA-OST-001",
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/output-styles",
|
|
"title": "Output styles",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-001",
|
|
"claim": "Project-root CLAUDE.md and unscoped rules are re-injected from disk after compaction (they survive a /compact).",
|
|
"appliesTo": "claude-md",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/context-window",
|
|
"title": "Context window — what survives compaction",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-002",
|
|
"claim": "Path-scoped rules are lost after compaction until a matching file is read again, and they trigger on Read of a matching file (not on every tool use).",
|
|
"appliesTo": "rule",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/memory",
|
|
"title": "Memory — path-specific rules",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-003",
|
|
"claim": "A nested (non-root) CLAUDE.md is lost after compaction until a file in its directory is read again.",
|
|
"appliesTo": "claude-md",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/context-window",
|
|
"title": "Context window — what survives compaction",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-004",
|
|
"claim": "A skill's name + description load every turn; its body loads only on invoke.",
|
|
"appliesTo": "skill",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/skills",
|
|
"title": "Skills",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-005",
|
|
"claim": "Hook scripts run outside the model context, but any additionalContext they inject is saved to the transcript and is therefore subject to compaction.",
|
|
"appliesTo": "hook",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/hooks",
|
|
"title": "Hooks",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-LOAD-006",
|
|
"claim": "A subagent runs in an isolated, fresh context window; only its final summary returns to the main session (parent instructions are not auto-injected).",
|
|
"appliesTo": "agent",
|
|
"confidence": "confirmed",
|
|
"category": "loading-model",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/sub-agents",
|
|
"title": "Subagents",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-SIZE-001",
|
|
"claim": "Keep CLAUDE.md under 200 lines; give it an owner and review changes to it like code. Every line costs tokens whether relevant or not.",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Trim CLAUDE.md to facts; move procedures to skills and path-specific rules to .claude/rules/.",
|
|
"confidence": "confirmed",
|
|
"severity": "medium",
|
|
"category": "size-budget",
|
|
"lensCheck": "CA-CML-001",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-SIZE-002",
|
|
"claim": "The skill-listing description cap is 1,536 characters (maxSkillDescriptionChars, configurable, v2.1.105+); the name + description load every turn.",
|
|
"appliesTo": "skill",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "size-budget",
|
|
"lensCheck": "CA-SKL-002",
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/skills",
|
|
"title": "Skills",
|
|
"verified": "2026-06-20"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-SUB-001",
|
|
"claim": "Every line of CLAUDE.md loads into every session whether or not it is relevant, which consumes tokens and dilutes adherence. A line that states a local fact the model cannot derive (build commands, directory layout, conventions, team norms) earns that cost; a line that only restates general engineering behaviour pays it without being the kind of content CLAUDE.md is for, and is a candidate for removal.",
|
|
"mechanism": "deletion",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Review the block for removal, then re-add it only if the model actually stumbles on it repeatedly. Local facts (remotes, versions, paths, conventions) and policy invariants are the floor and are never removal candidates.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "subtraction",
|
|
"lensCheck": "compensatory-instruction",
|
|
"source": {
|
|
"url": "https://claude.com/blog/steering-claude-code-skills-hooks-rules-subagents-and-more",
|
|
"title": "Steering Claude Code: skills, hooks, rules, subagents and more",
|
|
"verified": "2026-07-31"
|
|
},
|
|
"sources": [
|
|
{
|
|
"url": "https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models",
|
|
"title": "The new rules of context engineering for Claude 5 generation models",
|
|
"published": "2026-07-24",
|
|
"verified": "2026-08-03",
|
|
"note": "Near-verbatim coverage: 'briefly describe what your repo is for, but spend most of the tokens on gotchas inside of the codebase'; 'Avoid stating the obvious things Claude should know by looking at your file system or your repo.'"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "BP-MODEL-001",
|
|
"claim": "A subagent's `model` frontmatter field defaults to `inherit`, so a subagent that names no model runs on the main conversation's model. Routing mechanical or read-only subagents to a cheaper alias (`haiku`, `sonnet`) while the orchestrating session keeps the stronger model is the documented way to control cost. The pin is not absolute: Claude Code resolves the model as CLAUDE_CODE_SUBAGENT_MODEL, then a per-invocation `model` parameter, then the frontmatter, then the main conversation's model.",
|
|
"mechanism": "model",
|
|
"appliesTo": "agent",
|
|
"recommendation": "Set `model:` explicitly on subagents whose work is mechanical or read-only (search, extraction, summarisation) and leave the orchestrator on the stronger model. Omitting the field is not a neutral default — it inherits, so every subagent costs what the session costs.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "model-fit",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/sub-agents",
|
|
"title": "Create custom subagents — supported frontmatter fields / choose a model",
|
|
"verified": "2026-08-10"
|
|
},
|
|
"sources": [
|
|
{
|
|
"url": "https://claude.com/blog/claude-model-and-effort-level-in-claude-code",
|
|
"title": "Choosing a Claude model and effort level in Claude Code",
|
|
"published": "2026-07-07",
|
|
"verified": "2026-08-10",
|
|
"note": "Verbatim: 'Pick a smaller model when the work is routine. For example, edits you can describe precisely, mechanical changes, or questions about code that's already in context.'"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "BP-MODEL-002",
|
|
"claim": "Reasoning effort is an axis separate from model choice: five levels (`low`, `medium`, `high`, `xhigh`, `max`) on current models, four on Opus 4.6 and Sonnet 4.6, which omit `xhigh`; the default is `high` on every model that supports effort except Opus 4.7, which defaults to `xhigh`. Higher is not universally better — `max` \"can improve performance on demanding tasks but may show diminishing returns and is prone to overthinking\". Effort is settable in six places: `/effort`, the slider in `/model`, the `--effort` flag, CLAUDE_CODE_EFFORT_LEVEL, `effortLevel` in settings, and `effort:` in skill or subagent frontmatter; the environment variable takes precedence over all of them.",
|
|
"mechanism": "effort",
|
|
"appliesTo": "agent",
|
|
"recommendation": "Treat effort as a per-task dial rather than a global maximum: pin a lower `effort:` in the frontmatter of mechanical skills and subagents, and reserve `xhigh`/`max` for work whose product is judgment. The scale is calibrated per model, so the same level name is not the same amount of thinking across models — and CLAUDE_CODE_EFFORT_LEVEL silently overrides every other source, so verify which level is actually in force.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "model-fit",
|
|
"lensCheck": null,
|
|
"source": {
|
|
"url": "https://code.claude.com/docs/en/model-config",
|
|
"title": "Model configuration — adjust effort level / set the effort level",
|
|
"verified": "2026-08-10"
|
|
},
|
|
"sources": [
|
|
{
|
|
"url": "https://claude.com/blog/claude-model-and-effort-level-in-claude-code",
|
|
"title": "Choosing a Claude model and effort level in Claude Code",
|
|
"published": "2026-07-07",
|
|
"verified": "2026-08-10",
|
|
"note": "Verbatim: 'Claude will be more predisposed to double-checking additional hypotheses or verifying correctness at higher effort levels, but it generally won't artificially inflate usage for simple tasks at higher effort levels.'; 'In fact, our team pays close attention to \"overthinking\" during model training as it degrades effectiveness.'"
|
|
}
|
|
]
|
|
},
|
|
{
|
|
"id": "BP-JUDG-001",
|
|
"claim": "An instruction that fixes a form decision — how code or prose should look (comment density, docstring length, naming shape, sentence or paragraph count) — as an absolute rule buys a guardrail current models no longer need, and is wrong for the cases the rule did not anticipate. Anthropic removed its own example from the Claude Code system prompt: \"Never write multi-paragraph docstrings or multi-line comment blocks — one short line max\" was replaced by \"Write code that reads like the surrounding code: match its comment density, naming, and idiom.\"",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Where an absolute governs a form decision rather than a local fact, state the outcome you want and let the model judge the instance. This does not apply to safety rules, tool or version facts, or a house style you hold deliberately — those are the reason the claim is not machine-checkable.",
|
|
"confidence": "confirmed",
|
|
"category": "judgment-fit",
|
|
"lensCheck": null,
|
|
"note": "KNOWLEDGE ONLY — no detector, by measurement (docs/b2-judgment-lens-fasit.local.md §9). Across 409 real CLAUDE.md files (38488 lines, 8689 prose blocks) the class fired 7 times and all 7 were false positives; an independent grep in both word orders found 5 lines and 1 line respectively, none an instruction. Where the shape does occur — 45 matching lines across 4755 skill/agent/command files — it is the author's editorial policy (emoji, sentence length, slide titles), which nothing in the text separates from a vendor's over-tight guardrail. Precision-first: no CA-OPT code was allocated and OPT next-free stays 2.",
|
|
"source": {
|
|
"url": "https://claude.com/blog/the-new-rules-of-context-engineering-for-claude-5-generation-models",
|
|
"title": "The new rules of context engineering for Claude 5 generation models",
|
|
"published": "2026-07-24",
|
|
"verified": "2026-08-12"
|
|
}
|
|
},
|
|
{
|
|
"id": "BP-PROMPT-001",
|
|
"claim": "Claude Opus 5 catches and fixes its own mistakes without being told to, and over-verifies when explicitly instructed to double-check or to delegate verification to a subagent — this adds token cost without improving output quality. The source states both halves: \"Avoid instructing re-checks it already performs ('double-check your answer,' 're-verify before responding')\" and \"If your prompt contains explicit verification instructions ('include a final verification step for any non-trivial task,' 'use a subagent to verify'), remove them\".",
|
|
"mechanism": "deletion",
|
|
"appliesTo": "claude-md",
|
|
"recommendation": "Remove explicit self-verification or delegate-to-subagent-for-verification instructions when targeting Claude Opus 5; re-add only if the model actually stumbles. The claim is model-scoped, not universal — a config that runs a different model in a different session still needs them.",
|
|
"confidence": "confirmed",
|
|
"severity": "low",
|
|
"category": "prompting-fit",
|
|
"modelScope": ["opus-5"],
|
|
"lensCheck": null,
|
|
"note": "lensCheck deliberately null, same discipline as BP-JUDG-001: this is a knowledge-cited ANNOTATION on an existing BP-SUB-001 candidate (scanners/lib/prompting-model-scope.mjs), gated behind an explicit --for-model flag, not a second competing detector. It never widens the candidate set. Auto-detection is not viable — a CLAUDE.md carries no frontmatter and no statically-resolvable target model. source.published is absent because the page carries no visible publish or last-updated date (re-checked 2026-08-12); both quoted sentences were verified verbatim on that date. MEASURED 2026-08-12 across 409 real CLAUDE.md files: 392 BP-SUB-001 candidates, 31 of them (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target — so the annotation fires 0 times on this corpus. Two independent raw-text greps (reflexive phrasings; subagent-near-verify in both word orders) also found 0, so the zero is the corpus, not an over-narrow regex. The TARGET requirement is what earns its place: without it the same 31 verb-only blocks — 'sjekk relevante config-filer', 'Type-sjekk: pyright', 'To verify plugin functionality' — would all have been tagged, which is the BP-JUDG-001 failure mode (7/7 false positives) arriving one lens over. Precision on the corpus is 0 wrong out of 31 chances to be wrong; recall is untested there because the class is absent — the detector's true positives are the source doc's own example phrasings, pinned in tests/lib/prompting-model-scope.test.mjs.",
|
|
"source": {
|
|
"url": "https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5",
|
|
"title": "Prompting Claude Opus 5 — Self-correction / Task scope and over-verification",
|
|
"verified": "2026-08-12"
|
|
}
|
|
}
|
|
]
|
|
}
|