feat(scanners): a redundancy claim that belongs to one model is scoped to it
Anthropic documents that Claude Opus 5 verifies its own work, and that telling it to double-check or to delegate verification to a subagent causes over-verification -- token cost with no quality gain. The general subtraction detector (BP-SUB-001) already surfaces those blocks for every user, with no model-awareness at all. `optimize --subtract --for-model <name>` adds the missing half. It ANNOTATES a subset of the candidates --subtract already produced; it is not a second detector and can never widen the candidate set. A second SUBTRACT_DETECTORS entry would have collided with BP-SUB-001 on de-dup, and a prose-only signal in the agent prompt would have been untestable. There is no auto-detection, by measurement rather than omission: a CLAUDE.md has no frontmatter and no resolvable target model, and this operator's own `route` skill deliberately runs a different model per session -- the same file is read by whichever model comes next. So the model is named, and the citation is reported as conditional everywhere a human sees it (agent report copy, and the Step 7a listing that is the last surface before an approval file). Precision comes from the TARGET, not the verb list. Measured across the 409-file corpus: 392 BP-SUB-001 candidates, 31 (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target. Two independent raw-text greps found 0 as well, so the zero is the corpus rather than an over-narrow regex. Those 31 verb-only blocks -- "sjekk relevante config-filer", "Type-sjekk: pyright", "To verify plugin functionality" -- are exactly the false positives a verb-only version would have produced, which is BP-JUDG-001's 7/7 failure arriving one lens over. The numbers live in the register entry's note and are pinned by a test, because a session that cannot see the measurement reads the zero as a broken detector and loosens it. `recognized` is reported separately from `matchedCount`: a typo'd model name and a genuinely clean config both yield zero, and without the distinction the CLI would report a silent no-op as good news. Dogfooded on the real machine -- `opus-5` gives recognized:true/matchedCount:0, `oppus5` gives recognized:false. source.published is absent because the guide carries no visible publish date; its absence is asserted so a later session does not invent one to match the other entries' shape. Both quoted sentences were verified verbatim 2026-08-12. The payload stays additive -- forModel and per-candidate modelScope appear only under the flag, so a plain --subtract run is byte-identical to before (asserted on the serialized bytes, since a key set to undefined passes a shallow check). Suite 1724 -> 1752 (+28). The one remaining failure is the pre-existing drift-cli --output-file crash, untouched by this work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BRuXt6tZyowi8QYNKLSHQm
This commit is contained in:
parent
05b4e9d797
commit
7df8e0d65b
11 changed files with 650 additions and 10 deletions
|
|
@ -279,6 +279,24 @@
|
|||
"published": "2026-07-24",
|
||||
"verified": "2026-08-12"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "BP-PROMPT-001",
|
||||
"claim": "Claude Opus 5 catches and fixes its own mistakes without being told to, and over-verifies when explicitly instructed to double-check or to delegate verification to a subagent — this adds token cost without improving output quality. The source states both halves: \"Avoid instructing re-checks it already performs ('double-check your answer,' 're-verify before responding')\" and \"If your prompt contains explicit verification instructions ('include a final verification step for any non-trivial task,' 'use a subagent to verify'), remove them\".",
|
||||
"mechanism": "deletion",
|
||||
"appliesTo": "claude-md",
|
||||
"recommendation": "Remove explicit self-verification or delegate-to-subagent-for-verification instructions when targeting Claude Opus 5; re-add only if the model actually stumbles. The claim is model-scoped, not universal — a config that runs a different model in a different session still needs them.",
|
||||
"confidence": "confirmed",
|
||||
"severity": "low",
|
||||
"category": "prompting-fit",
|
||||
"modelScope": ["opus-5"],
|
||||
"lensCheck": null,
|
||||
"note": "lensCheck deliberately null, same discipline as BP-JUDG-001: this is a knowledge-cited ANNOTATION on an existing BP-SUB-001 candidate (scanners/lib/prompting-model-scope.mjs), gated behind an explicit --for-model flag, not a second competing detector. It never widens the candidate set. Auto-detection is not viable — a CLAUDE.md carries no frontmatter and no statically-resolvable target model. source.published is absent because the page carries no visible publish or last-updated date (re-checked 2026-08-12); both quoted sentences were verified verbatim on that date. MEASURED 2026-08-12 across 409 real CLAUDE.md files: 392 BP-SUB-001 candidates, 31 of them (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target — so the annotation fires 0 times on this corpus. Two independent raw-text greps (reflexive phrasings; subagent-near-verify in both word orders) also found 0, so the zero is the corpus, not an over-narrow regex. The TARGET requirement is what earns its place: without it the same 31 verb-only blocks — 'sjekk relevante config-filer', 'Type-sjekk: pyright', 'To verify plugin functionality' — would all have been tagged, which is the BP-JUDG-001 failure mode (7/7 false positives) arriving one lens over. Precision on the corpus is 0 wrong out of 31 chances to be wrong; recall is untested there because the class is absent — the detector's true positives are the source doc's own example phrasings, pinned in tests/lib/prompting-model-scope.test.mjs.",
|
||||
"source": {
|
||||
"url": "https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5",
|
||||
"title": "Prompting Claude Opus 5 — Self-correction / Task scope and over-verification",
|
||||
"verified": "2026-08-12"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue