feat(scanners): a redundancy claim that belongs to one model is scoped to it
Anthropic documents that Claude Opus 5 verifies its own work, and that telling it to double-check or to delegate verification to a subagent causes over-verification -- token cost with no quality gain. The general subtraction detector (BP-SUB-001) already surfaces those blocks for every user, with no model-awareness at all. `optimize --subtract --for-model <name>` adds the missing half. It ANNOTATES a subset of the candidates --subtract already produced; it is not a second detector and can never widen the candidate set. A second SUBTRACT_DETECTORS entry would have collided with BP-SUB-001 on de-dup, and a prose-only signal in the agent prompt would have been untestable. There is no auto-detection, by measurement rather than omission: a CLAUDE.md has no frontmatter and no resolvable target model, and this operator's own `route` skill deliberately runs a different model per session -- the same file is read by whichever model comes next. So the model is named, and the citation is reported as conditional everywhere a human sees it (agent report copy, and the Step 7a listing that is the last surface before an approval file). Precision comes from the TARGET, not the verb list. Measured across the 409-file corpus: 392 BP-SUB-001 candidates, 31 (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target. Two independent raw-text greps found 0 as well, so the zero is the corpus rather than an over-narrow regex. Those 31 verb-only blocks -- "sjekk relevante config-filer", "Type-sjekk: pyright", "To verify plugin functionality" -- are exactly the false positives a verb-only version would have produced, which is BP-JUDG-001's 7/7 failure arriving one lens over. The numbers live in the register entry's note and are pinned by a test, because a session that cannot see the measurement reads the zero as a broken detector and loosens it. `recognized` is reported separately from `matchedCount`: a typo'd model name and a genuinely clean config both yield zero, and without the distinction the CLI would report a silent no-op as good news. Dogfooded on the real machine -- `opus-5` gives recognized:true/matchedCount:0, `oppus5` gives recognized:false. source.published is absent because the guide carries no visible publish date; its absence is asserted so a later session does not invent one to match the other entries' shape. Both quoted sentences were verified verbatim 2026-08-12. The payload stays additive -- forModel and per-candidate modelScope appear only under the flag, so a plain --subtract run is byte-identical to before (asserted on the serialized bytes, since a key set to undefined passes a shallow check). Suite 1724 -> 1752 (+28). The one remaining failure is the pre-existing drift-cli --output-file crash, untouched by this work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01BRuXt6tZyowi8QYNKLSHQm
This commit is contained in:
parent
05b4e9d797
commit
7df8e0d65b
11 changed files with 650 additions and 10 deletions
34
CLAUDE.md
34
CLAUDE.md
|
|
@ -17,7 +17,7 @@ Per-command flags, patterns, and feature lists live in `README.md` and `/config-
|
|||
| `/config-audit tokens` | Prompt-cache-aware token hotspots, each tagged with its load pattern; cache-aware |
|
||||
| `/config-audit manifest` | Ranked table of every token source + always-loaded subtotal |
|
||||
| `/config-audit feature-gap` | Context-aware feature recommendations grouped by impact |
|
||||
| `/config-audit optimize` | Mechanism-fit lens (procedure→skill, lifecycle→hook, path→rule, never→permission). Agent-driven, **not byte-stable**. `--subtract` adds the subtraction axis (what no longer earns its always-loaded rent, `BP-SUB-001`) — opt-in, proposes only; `--subtract --apply` executes the removals the operator picks |
|
||||
| `/config-audit optimize` | Mechanism-fit lens (procedure→skill, lifecycle→hook, path→rule, never→permission). Agent-driven, **not byte-stable**. `--subtract` adds the subtraction axis (what no longer earns its always-loaded rent, `BP-SUB-001`) — opt-in, proposes only; `--subtract --apply` executes the removals the operator picks; `--subtract --for-model <name>` annotates the candidates a named model documents as redundant (`BP-PROMPT-001`) |
|
||||
| `/config-audit fix` | Auto-fix deterministic issues with backup + verification |
|
||||
| `/config-audit rollback` | Restore configuration from backup |
|
||||
| `/config-audit plan` | Create action plan from findings |
|
||||
|
|
@ -165,6 +165,38 @@ a trailing `:54-56` locator is a recorded v1 miss, not a bug to fix by loosening
|
|||
|
||||
**Subtraction floor (invariant).** `optimize --subtract` is the only lens that proposes removing config, so `scanners/lib/floor-exclusion.mjs` runs as a deterministic pre-step *before* the judge — a load-bearing block is never a candidate, and that guarantee must not be moved into the agent prompt. Two rules follow from it: (1) **staleness is not a deletion signal** — an outdated version pin inside a floor block is a `drift`/`CA-CML` dead-reference concern; (2) **tier 2 ≠ tier 3** — a compensatory block that keeps earning its place returns, and reporting it as dead weight is wrong even when the label matches. Norwegian keywords need the Unicode boundaries in `subtraction-prefilter.mjs`; JS `\b` is ASCII-only, so `/\bunngå\b/` silently never matches.
|
||||
|
||||
**Model-scoped annotation (invariant).** `--for-model <name>` (`BP-PROMPT-001`,
|
||||
`scanners/lib/prompting-model-scope.mjs`) is an **annotation on existing
|
||||
`BP-SUB-001` candidates, never a detector** — it tags a subset of what
|
||||
`--subtract` already found and can never widen the candidate set. Four
|
||||
properties are load-bearing. (1) **The model must be named.** There is no
|
||||
auto-detection and there cannot be: a CLAUDE.md has no frontmatter and no
|
||||
resolvable target model, and the operator's own `route` skill deliberately runs
|
||||
a different model per session — so the same file is read by whichever model
|
||||
comes next. (2) **Precision is the TARGET, not the verb list.** The verb set is
|
||||
as broad as `subtraction-prefilter`'s `IMPERATIVE_RE`; what narrows it is
|
||||
requiring a co-occurring reflexive ("your own work", "før du svarer") or
|
||||
delegated ("subagent") target. Measured across the 409-file corpus: 392
|
||||
`BP-SUB-001` candidates, **31 carry a verify verb, 0 also carry a target**, and
|
||||
two independent raw-text greps also found 0 — so the zero is the corpus, not an
|
||||
over-narrow regex. Those 31 (`sjekk relevante config-filer`, `Type-sjekk:
|
||||
pyright`) are exactly the false positives a verb-only version would have
|
||||
produced, which is `BP-JUDG-001`'s 7/7 failure arriving one lens over. The
|
||||
numbers live in the register entry's `note` and are pinned by
|
||||
`best-practices-register.test.mjs`, because a later session that cannot see the
|
||||
measurement reads the zero as a broken detector and loosens it. (3) **`recognized`
|
||||
is not `matchedCount`.** A typo'd model name and a genuinely clean config both
|
||||
yield zero matches; without the separate flag the CLI would report a silent
|
||||
no-op as good news. Dogfooded on the real machine: `opus-5` → `recognized:true,
|
||||
matchedCount:0`; `oppus5` → `recognized:false`. (4) **The citation is
|
||||
conditional wherever a human sees it** — agent report copy and the Step 7a
|
||||
approval listing both say so. `--for-model` and `--apply` are separate CLIs that
|
||||
never see each other's flags, so a CLI-level refusal is impossible; the
|
||||
safeguard has to live in the copy. The payload stays additive: `forModel` and
|
||||
per-candidate `modelScope` appear **only** when the flag is passed, so a plain
|
||||
`--subtract` run is byte-identical to the pre-flag payload (asserted on the
|
||||
serialized bytes, since a key set to `undefined` passes a shallow key check).
|
||||
|
||||
**Subtraction write path (invariant).** `--apply` routes through `scanners/lib/subtraction-write.mjs`, never through `fix-engine` or the `plan`/`implement` pipeline, and both exclusions are **measured**: the subtraction axis is absent from the orchestrated envelope, so `verifyFixes`' re-scan would mark every removal `verified` whether or not it happened (a success-shaped no-op), and the findings pipeline needs a finding code — which names a deterministic check, not a prose judgement. Three properties are load-bearing and each has a guard seen red against its own defect: removals are validated against the ORIGINAL content and applied in **descending** line order (an ascending pass shifts later spans out from under themselves); the **range** check is not redundant with the text check (`line: 0` makes `slice(-1, 0)` empty, so an empty `text` matches and `splice(-1, 1)` deletes the file's LAST line); and `createBackup` skips a nonexistent path while still returning an id, so coverage of every file about to be written is **asserted from the manifest** before a byte changes. The floor is *repeated* here, not moved: `floor-exclusion` still vetoes before anything is proposed, and the engine refuses a load-bearing block again so a hand-built approval cannot route around it. The archive rule (`mv` to `_archive/`) is file-level and does not apply to a block excision — the timestamped backup is the recovery artifact, and inventing a second copy with no restorer behind it would be worse than none.
|
||||
|
||||
## Testing
|
||||
|
|
|
|||
25
README.md
25
README.md
|
|
@ -266,6 +266,7 @@ Your team configuration changes over time. Track it:
|
|||
| `/config-audit optimize` | Optimization lens (mechanism-fit): config that works but fits a better mechanism — procedure→skill, lifecycle→hook, unscoped path→rule, "never"→permission. Hybrid motor (deterministic pre-filter + opus precision gate), every finding cites a best-practices-register rule |
|
||||
| `/config-audit optimize --subtract` | **Subtraction lens** — the inverse question no other command asks: what no longer earns its always-loaded rent? Ranks CLAUDE.md blocks that correct general model *behaviour* rather than stating a local fact, split into **dead** (never missed) and **earned** (returns if the model stumbles), with the token payoff (`BP-SUB-001`). **Load-bearing local facts are excluded deterministically before the judge sees anything** — remotes, versions, paths, filenames, policy invariants and unresolvable entity names are never candidates, and an ordered list is treated as a contract. Opt-in and proposes only; add `--apply` to execute the removals you pick. Pair with `--global` to reach the user-level CLAUDE.md, where the always-loaded cost actually sits |
|
||||
| `/config-audit optimize --subtract --apply` | **Execute approved removals.** You pick which blocks go by number; nothing is inferred. Every removal is checked against the file as it reads *now* — an approval that no longer matches is refused rather than applied to whatever moved into those lines — and the floor is re-asserted at write time, so a load-bearing block cannot be removed even by a hand-built approval. A dry run always precedes the write, the backup's manifest is verified to cover the file being written before a byte changes, and `/config-audit rollback` restores it. A removal targeting your machine-wide `~/.claude/CLAUDE.md` is **refused until you approve that scope explicitly** — it costs, and saves, in every project on every turn |
|
||||
| `/config-audit optimize --subtract --for-model <name>` | **Model-scoped subtraction.** Some instructions are dead weight only for a *particular* model: Anthropic documents that Claude Opus 5 verifies its own work and **over-verifies** when told to double-check or to delegate verification to a subagent, adding token cost with no quality gain (`BP-PROMPT-001`). This flag annotates the `--subtract` candidates that carry a reflexive or delegated verification target — "double-check your own work", "use a subagent to verify" — while leaving *external* verification ("check the CI status") untagged. It **never widens the candidate set**, and there is deliberately no auto-detection: a CLAUDE.md has no frontmatter and no resolvable target model, and the same file is read by whichever model the next session runs — so the citation is reported as **conditional**, and an unrecognized model name is reported as unrecognized rather than as a silent zero |
|
||||
| `/config-audit fix` | Auto-fix deterministic issues with backup + verification |
|
||||
| `/config-audit rollback` | Restore configuration from a previous backup |
|
||||
| `/config-audit plan` | Generate prioritized action plan from audit findings |
|
||||
|
|
@ -647,6 +648,30 @@ classification), or when it names a capitalized entity the mechanism cannot reso
|
|||
dictionary. That last rule is a deliberate conservative default: it declines to decide and
|
||||
keeps the block, paying in recall rather than risk.
|
||||
|
||||
### Model-scoped candidates (`--for-model`)
|
||||
|
||||
Some instructions are dead weight only for a *particular* model. Anthropic documents that
|
||||
Claude Opus 5 verifies its own work, and that explicit instructions to double-check or to
|
||||
delegate verification to a subagent cause **over-verification** — token cost with no gain in
|
||||
quality. `--subtract --for-model opus-5` annotates the subtraction candidates that carry a
|
||||
reflexive target ("double-check your own work", "før du svarer") or a delegated one ("verify
|
||||
with a subagent"), while leaving *external* verification ("check the CI status") untagged.
|
||||
|
||||
It is an annotation, never a detector: it tags a subset of what `--subtract` already found and
|
||||
can never widen the candidate set. There is deliberately **no auto-detection** — a CLAUDE.md
|
||||
has no frontmatter and no resolvable target model, and the same file is read by whichever model
|
||||
the next session happens to run. That is also why the citation is reported as *conditional*
|
||||
rather than as a settled fact about the file, both in the report and in the approval listing
|
||||
shown before anything is written.
|
||||
|
||||
Precision comes from requiring the target, not from a narrow verb list. Measured across 409
|
||||
real CLAUDE.md files: 392 subtraction candidates, **31 carry a verify verb, and 0 also carry a
|
||||
reflexive or delegated target** — a zero confirmed by two independent raw-text greps, so it is
|
||||
the corpus rather than an over-narrow rule. Those 31 verb-only blocks are precisely the false
|
||||
positives a looser version would have produced. A model name the register does not cover is
|
||||
reported as **unrecognized** rather than as a bare zero, so a typo never reads as "your config
|
||||
is already clean".
|
||||
|
||||
The write half (`--apply`) keeps the same asymmetry. It is not a `fix` action and not a
|
||||
`plan`/`implement` step, and both exclusions are measurements rather than preferences: the
|
||||
subtraction axis never enters the orchestrated envelope, so `fix`'s re-scan verification would
|
||||
|
|
|
|||
|
|
@ -38,8 +38,8 @@ whether the line is *really* that kind of instruction:
|
|||
## The subtraction lens (`--subtract` only)
|
||||
|
||||
Present only when the payload has a `subtract` block. It asks the inverse of
|
||||
every other lens: *what is no longer earning its always-loaded rent?* Three
|
||||
things make it different, and all three are non-negotiable.
|
||||
every other lens: *what is no longer earning its always-loaded rent?* Four
|
||||
things make it different, and all four are non-negotiable.
|
||||
|
||||
**1. The floor is not yours to decide.** A deterministic pre-step has already
|
||||
excluded every block carrying a local fact — a code span, path, domain, version
|
||||
|
|
@ -64,6 +64,15 @@ earned its place — its subject matter recurs in the repo's own history — is
|
|||
2 even when its classification is "compensatory". Reporting it as dead weight is
|
||||
wrong even though the label matches.
|
||||
|
||||
**4. A model-scoped citation is conditional, never authoritative.** When a
|
||||
candidate carries `modelScope` (only under `--for-model`), apply the SAME tier-2
|
||||
/ tier-3 judgement as any other `compensatory-instruction` candidate — the model
|
||||
tag sharpens the citation, it does not bypass precision rule 1. State it as
|
||||
conditional in the report copy ("redundant if targeting {model}; this operator
|
||||
may run other models in other sessions"), never as a settled fact for all future
|
||||
sessions. The tag is an annotation on a candidate the general detector already
|
||||
found; it is not evidence that the block is dead.
|
||||
|
||||
Rank kept candidates by always-loaded token cost, and state the total payoff.
|
||||
Frame it as *rent*, never as a mistake: this config was correct when written.
|
||||
|
||||
|
|
@ -82,6 +91,12 @@ You receive an `optimize-lens` payload (JSON) with:
|
|||
register, detectors }`. Each candidate spans `line`–`endLine` (a whole leaf
|
||||
block, not one line) and carries `signalText` plus the BP-SUB-001 register
|
||||
block. Everything load-bearing was already removed before you saw this.
|
||||
Under `--for-model <name>` the block also carries `forModel`
|
||||
(`{ requested, recognized, matchedCount }`), and a SUBSET of its candidates
|
||||
carry `modelScope` (`{ registerId, claim, requestedModel }`). If
|
||||
`recognized` is `false`, the operator named a model the register does not
|
||||
cover — say so, and do not treat the absence of tags as evidence of a clean
|
||||
config.
|
||||
|
||||
Always **Read the actual CLAUDE.md file(s)** named in the candidates before
|
||||
judging — `signalText` is one line out of context; the surrounding lines decide
|
||||
|
|
@ -150,6 +165,8 @@ to return) — ranked by token cost, with a payoff total. For each:}
|
|||
**{file}:{line}-{endLine}** — {what the block says, in one line} · ~{N} tok/turn
|
||||
Tier: {dead | earned — and why}
|
||||
Source: {register.source.url}
|
||||
Model-scoped: {only if the candidate carries `modelScope`} {claim}, for
|
||||
{requestedModel} (conditional — verify this is still the model you target)
|
||||
```
|
||||
|
||||
Omit any section with zero kept findings (except keep the "left alone" note when
|
||||
|
|
|
|||
|
|
@ -25,6 +25,8 @@ is hybrid: a cheap deterministic pre-filter finds candidates, then the opus
|
|||
- **Unscoped path-specific instructions → path-scoped rules** (BP-MECH-002)
|
||||
- **Absolute "never" prohibitions → permissions / hooks** (BP-MECH-004)
|
||||
- **`--subtract`:** instructions that no longer earn their always-loaded rent (BP-SUB-001)
|
||||
- **`--subtract --for-model <name>`:** of those, the ones a named model documents
|
||||
as redundant — e.g. self-verification instructions on Claude Opus 5 (BP-PROMPT-001)
|
||||
|
||||
Each finding cites its register rule + source URL. A clean CLAUDE.md returns "no
|
||||
opportunities" — that is a good result, not a failure.
|
||||
|
|
@ -35,8 +37,9 @@ opportunities" — that is a good result, not a failure.
|
|||
|
||||
Split `$ARGUMENTS` into a path (first non-flag argument; default: current working
|
||||
directory) and flags. Recognized flags: `--global` (include the user `~/.claude`
|
||||
cascade in discovery), `--subtract` (add the subtraction axis, below) and
|
||||
`--apply` (execute approved removals — Step 7).
|
||||
cascade in discovery), `--subtract` (add the subtraction axis, below),
|
||||
`--for-model <name>` (annotate model-scoped candidates, below) and `--apply`
|
||||
(execute approved removals — Step 7).
|
||||
|
||||
`--apply` only means anything alongside `--subtract`. If it is present without
|
||||
it, say so and continue with the ordinary lens run:
|
||||
|
|
@ -46,12 +49,29 @@ it, say so and continue with the ordinary lens run:
|
|||
Running the ordinary lens; re-run with `--subtract --apply` to remove anything.
|
||||
```
|
||||
|
||||
`--for-model <name>` has the same dependency. If it is present without
|
||||
`--subtract`, say so and continue with the ordinary lens run:
|
||||
|
||||
```
|
||||
`--for-model` annotates subtraction candidates, so it needs `--subtract` too.
|
||||
Running the ordinary lens; re-run with `--subtract --for-model <name>`.
|
||||
```
|
||||
|
||||
**`--subtract` — the inverse question.** Every other lens asks what to *add* or
|
||||
*move*; this one asks what no longer earns its always-loaded rent. It is opt-in
|
||||
because it asks something different, and because deleting is not undoable by
|
||||
reading. Pair it with `--global` to reach the user-level CLAUDE.md, where the
|
||||
always-loaded cost actually sits (it loads in every repo, every session).
|
||||
|
||||
**`--for-model <name>` — whose redundancy?** Some instructions are only dead
|
||||
weight for a *particular* model. Anthropic documents that Claude Opus 5 verifies
|
||||
its own work and over-verifies when told to double-check or to delegate
|
||||
verification to a subagent. That claim is model-scoped, so the model must be
|
||||
named: a CLAUDE.md carries no frontmatter and no target model, and the same file
|
||||
is read by whichever model the next session happens to run. The flag never adds
|
||||
candidates — it annotates ones `--subtract` already found. Example:
|
||||
`--subtract --for-model opus-5`.
|
||||
|
||||
If `--subtract` is present, say so up front:
|
||||
|
||||
```
|
||||
|
|
@ -78,7 +98,17 @@ GLOBAL_FLAG=""
|
|||
if echo "$ARGUMENTS" | grep -q -- "--global"; then GLOBAL_FLAG="--global"; fi
|
||||
SUBTRACT_FLAG=""
|
||||
if echo "$ARGUMENTS" | grep -q -- "--subtract"; then SUBTRACT_FLAG="--subtract"; fi
|
||||
node ${CLAUDE_PLUGIN_ROOT}/scanners/optimize-lens-cli.mjs "<target-path>" --output-file ~/.claude/config-audit/sessions/{session-id}/optimize-lens.json $GLOBAL_FLAG $SUBTRACT_FLAG 2>/dev/null; echo $?
|
||||
# --for-model takes a VALUE, so flag and value are two separate variables. Never
|
||||
# pack them into one ("--for-model x"): the shell here is zsh, which does not
|
||||
# word-split an unquoted expansion, so one variable would reach the CLI as a
|
||||
# single malformed argv entry and be silently ignored (M-BUG-45).
|
||||
FOR_MODEL_FLAG=""
|
||||
FOR_MODEL_VALUE=""
|
||||
if echo "$ARGUMENTS" | grep -q -- "--for-model "; then
|
||||
FOR_MODEL_FLAG="--for-model"
|
||||
FOR_MODEL_VALUE=$(echo "$ARGUMENTS" | sed -n 's/.*--for-model \([^ ]*\).*/\1/p')
|
||||
fi
|
||||
node ${CLAUDE_PLUGIN_ROOT}/scanners/optimize-lens-cli.mjs "<target-path>" --output-file ~/.claude/config-audit/sessions/{session-id}/optimize-lens.json $GLOBAL_FLAG $SUBTRACT_FLAG $FOR_MODEL_FLAG $FOR_MODEL_VALUE 2>/dev/null; echo $?
|
||||
```
|
||||
|
||||
Exit code 0 is normal. Only exit code 3 is a real error → "The lens couldn't run.
|
||||
|
|
@ -94,6 +124,17 @@ Under `--subtract` it also has a `subtract` block (`candidates`, `register`) and
|
|||
`counts.subtractCandidates`. Each subtraction candidate spans `line`–`endLine`
|
||||
(a whole block). Include the whole `subtract` block when spawning the agent.
|
||||
|
||||
Under `--for-model` the `subtract` block additionally has `forModel`
|
||||
(`{ requested, recognized, matchedCount }`), and a subset of its candidates carry
|
||||
`modelScope` (`{ registerId, claim, requestedModel }`). If `recognized` is
|
||||
`false`, the register does not cover that model name — tell the user before
|
||||
showing results, so a zero is never read as "your config is already clean":
|
||||
|
||||
```
|
||||
I don't have a model-specific rule for "{requested}", so nothing was annotated
|
||||
for it. The ordinary subtraction results below are unaffected.
|
||||
```
|
||||
|
||||
**Early exit:** if `counts.deterministic === 0` and `counts.candidates === 0`
|
||||
(and, under `--subtract`, `counts.subtractCandidates === 0`), skip the agent and
|
||||
tell the user plainly:
|
||||
|
|
@ -142,7 +183,17 @@ takes configuration away, so nothing here happens without a named choice.
|
|||
|
||||
**7a — show what is on the table, with honest sizing.** List the kept
|
||||
subtraction findings numbered, each with its file, line span and first line of
|
||||
text. Do not imply a bigger win than there is:
|
||||
text. A finding carrying `modelScope` must show its condition here too — this
|
||||
is the last point a human sees it before it reaches an approval file, so the
|
||||
citation must not read as unconditional:
|
||||
|
||||
```
|
||||
{n}. {file}:{line}-{endLine} — {first line of text}
|
||||
Model-scoped: redundant if you are targeting {requestedModel}. Other
|
||||
sessions on this config may run a different model.
|
||||
```
|
||||
|
||||
Do not imply a bigger win than there is:
|
||||
|
||||
```
|
||||
Removing all of these saves roughly {n} tokens per turn — on a typical
|
||||
|
|
@ -222,6 +273,12 @@ End with context-sensitive next steps, explaining WHY each is useful:
|
|||
- `--subtract` **proposes; only `--apply` writes**, and only blocks the operator
|
||||
named. Every removal is preceded by a backup whose manifest is verified to
|
||||
cover the file being written, and `/config-audit rollback` restores it.
|
||||
- **`--for-model` annotates; it never detects.** It tags a subset of the
|
||||
candidates `--subtract` already produced, and there is no auto-detection by
|
||||
design: a CLAUDE.md has no frontmatter and no resolvable target model, and the
|
||||
same file is read by whichever model the next session runs. So the citation is
|
||||
always **conditional** — it must be shown that way in the report and in the
|
||||
Step 7a approval listing, never as a settled fact about the file.
|
||||
- **Removal is not a `fix` action and not a `plan`/`implement` step**, by
|
||||
measurement rather than preference: the subtraction axis never enters the
|
||||
orchestrated envelope, so `fix`'s re-scan verification would mark every
|
||||
|
|
|
|||
|
|
@ -279,6 +279,24 @@
|
|||
"published": "2026-07-24",
|
||||
"verified": "2026-08-12"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "BP-PROMPT-001",
|
||||
"claim": "Claude Opus 5 catches and fixes its own mistakes without being told to, and over-verifies when explicitly instructed to double-check or to delegate verification to a subagent — this adds token cost without improving output quality. The source states both halves: \"Avoid instructing re-checks it already performs ('double-check your answer,' 're-verify before responding')\" and \"If your prompt contains explicit verification instructions ('include a final verification step for any non-trivial task,' 'use a subagent to verify'), remove them\".",
|
||||
"mechanism": "deletion",
|
||||
"appliesTo": "claude-md",
|
||||
"recommendation": "Remove explicit self-verification or delegate-to-subagent-for-verification instructions when targeting Claude Opus 5; re-add only if the model actually stumbles. The claim is model-scoped, not universal — a config that runs a different model in a different session still needs them.",
|
||||
"confidence": "confirmed",
|
||||
"severity": "low",
|
||||
"category": "prompting-fit",
|
||||
"modelScope": ["opus-5"],
|
||||
"lensCheck": null,
|
||||
"note": "lensCheck deliberately null, same discipline as BP-JUDG-001: this is a knowledge-cited ANNOTATION on an existing BP-SUB-001 candidate (scanners/lib/prompting-model-scope.mjs), gated behind an explicit --for-model flag, not a second competing detector. It never widens the candidate set. Auto-detection is not viable — a CLAUDE.md carries no frontmatter and no statically-resolvable target model. source.published is absent because the page carries no visible publish or last-updated date (re-checked 2026-08-12); both quoted sentences were verified verbatim on that date. MEASURED 2026-08-12 across 409 real CLAUDE.md files: 392 BP-SUB-001 candidates, 31 of them (7.9%) carry a verify verb, and 0 also carry a reflexive or delegated target — so the annotation fires 0 times on this corpus. Two independent raw-text greps (reflexive phrasings; subagent-near-verify in both word orders) also found 0, so the zero is the corpus, not an over-narrow regex. The TARGET requirement is what earns its place: without it the same 31 verb-only blocks — 'sjekk relevante config-filer', 'Type-sjekk: pyright', 'To verify plugin functionality' — would all have been tagged, which is the BP-JUDG-001 failure mode (7/7 false positives) arriving one lens over. Precision on the corpus is 0 wrong out of 31 chances to be wrong; recall is untested there because the class is absent — the detector's true positives are the source doc's own example phrasings, pinned in tests/lib/prompting-model-scope.test.mjs.",
|
||||
"source": {
|
||||
"url": "https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5",
|
||||
"title": "Prompting Claude Opus 5 — Self-correction / Task scope and over-verification",
|
||||
"verified": "2026-08-12"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
|
|||
109
scanners/lib/prompting-model-scope.mjs
Normal file
109
scanners/lib/prompting-model-scope.mjs
Normal file
|
|
@ -0,0 +1,109 @@
|
|||
/**
|
||||
* Model-scoped ANNOTATION on top of the subtraction lens.
|
||||
*
|
||||
* This module generates no candidates of its own. `optimize --subtract` already
|
||||
* surfaces compensatory instructions through the single `compensatory-instruction`
|
||||
* detector (`BP-SUB-001`); this answers a narrower question over text that
|
||||
* already passed it: is this specifically the class of instruction a named model
|
||||
* documents as redundant — self-verification, or verification delegated to a
|
||||
* subagent?
|
||||
*
|
||||
* A second competing detector is deliberately NOT what this is. The register
|
||||
* entry it cites carries `lensCheck: null`, the same discipline `BP-JUDG-001`
|
||||
* shipped under: a plausible-looking detector for "instruction a model no longer
|
||||
* needs" measured 7/7 false positives across 409 real CLAUDE.md files, so this
|
||||
* claim class rides an existing measured detector rather than widening the
|
||||
* candidate set.
|
||||
*
|
||||
* Precision comes from requiring a reflexive/delegate TARGET alongside the
|
||||
* verify verb, never from narrowing the verb list. A bare "check"/"verify" also
|
||||
* matches EXTERNAL verification ("check the CI status") — which stays a true
|
||||
* negative here even though it is, correctly, still a `BP-SUB-001` candidate
|
||||
* upstream.
|
||||
*
|
||||
* Pure: text → annotation or null. Zero external dependencies.
|
||||
*/
|
||||
import { LB, RB } from './subtraction-prefilter.mjs';
|
||||
|
||||
/**
|
||||
* The reflexive self-verification target. This is what narrows a bare
|
||||
* verify/check imperative down to the specific claim the model's documented
|
||||
* self-correction contradicts.
|
||||
*/
|
||||
const SELF_TARGET_RE = new RegExp(
|
||||
LB +
|
||||
'(?:your (?:own )?(?:work|output|answer|changes)|yourself|' +
|
||||
'before (?:responding|submitting|finalizing)|dine egne?|deg selv|før du svarer)' +
|
||||
RB,
|
||||
'i',
|
||||
);
|
||||
|
||||
/**
|
||||
* Verify verbs — deliberately as broad as `subtraction-prefilter.mjs`'s
|
||||
* `IMPERATIVE_RE`. Breadth here is safe because a co-occurring target is
|
||||
* required; narrowing the list would only lose true positives.
|
||||
*/
|
||||
const VERIFY_VERB_RE = new RegExp(
|
||||
LB +
|
||||
'(?:double-check|re-verify|re-check|confirm|verify|review|check|' +
|
||||
'dobbeltsjekk|verifiser|sjekk)' +
|
||||
RB,
|
||||
'i',
|
||||
);
|
||||
|
||||
/**
|
||||
* Delegated verification. Order-free on purpose: it must match both
|
||||
* "verify X with a subagent" and "use a subagent to verify X".
|
||||
*/
|
||||
const DELEGATE_VERIFY_RE = new RegExp(
|
||||
LB + '(?:subagent|sub-agent|another (?:agent|instance)|task tool)' + RB,
|
||||
'i',
|
||||
);
|
||||
|
||||
/**
|
||||
* Does this text carry a self- or delegate-targeted verification instruction?
|
||||
*
|
||||
* @param {string} text
|
||||
* @returns {boolean}
|
||||
*/
|
||||
export function isModelContradictedVerification(text) {
|
||||
return (
|
||||
VERIFY_VERB_RE.test(text) &&
|
||||
(SELF_TARGET_RE.test(text) || DELEGATE_VERIFY_RE.test(text))
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Model names are compared on their alphanumeric skeleton, so "opus-5",
|
||||
* "Opus 5" and "OPUS5" are one model. A typo'd name simply fails to match —
|
||||
* the CLI reports that it did not recognize the name rather than reporting a
|
||||
* silent zero.
|
||||
*
|
||||
* @param {unknown} s
|
||||
* @returns {string}
|
||||
*/
|
||||
const normalizeModel = (s) =>
|
||||
String(s || '')
|
||||
.toLowerCase()
|
||||
.replace(/[^a-z0-9]/g, '');
|
||||
|
||||
/**
|
||||
* Annotate a subtraction candidate with a model-scoped register citation.
|
||||
*
|
||||
* @param {string} text candidate block text (already a `BP-SUB-001` candidate)
|
||||
* @param {string|null|undefined} targetModel the model named by `--for-model`
|
||||
* @param {Array<{id: string, claim: string, modelScope?: string[]}>} entries
|
||||
* confirmed register entries with `category: 'prompting-fit'`
|
||||
* @returns {{registerId: string, claim: string, requestedModel: string}|null}
|
||||
*/
|
||||
export function matchModelScope(text, targetModel, entries) {
|
||||
if (!targetModel || !isModelContradictedVerification(text)) return null;
|
||||
const wanted = normalizeModel(targetModel);
|
||||
const entry = (entries || []).find((e) =>
|
||||
(e.modelScope || []).some((m) => normalizeModel(m) === wanted),
|
||||
);
|
||||
if (!entry) return null;
|
||||
return { registerId: entry.id, claim: entry.claim, requestedModel: targetModel };
|
||||
}
|
||||
|
||||
export { normalizeModel };
|
||||
|
|
@ -69,8 +69,8 @@ export const SUBTRACT_DETECTORS = Object.freeze([
|
|||
* Every Norwegian keyword ending in æ/ø/å was silently dead until the dogfood
|
||||
* run surfaced it. Do not reintroduce `\b` around this vocabulary.
|
||||
*/
|
||||
const LB = '(?<![\\wæøåÆØÅ])';
|
||||
const RB = '(?![\\wæøåÆØÅ])';
|
||||
export const LB = '(?<![\\wæøåÆØÅ])';
|
||||
export const RB = '(?![\\wæøåÆØÅ])';
|
||||
|
||||
const ABSOLUTE_RE = new RegExp(
|
||||
LB +
|
||||
|
|
|
|||
|
|
@ -20,6 +20,12 @@
|
|||
*
|
||||
* Usage:
|
||||
* node optimize-lens-cli.mjs [path] [--output-file <path>] [--global]
|
||||
* [--subtract [--for-model <name>]]
|
||||
*
|
||||
* `--for-model <name>` annotates the subtraction candidates a named model
|
||||
* documents as redundant (BP-PROMPT-001). It never widens the candidate set,
|
||||
* and there is deliberately no auto-detection: a CLAUDE.md has no frontmatter
|
||||
* and no statically-resolvable target model, so the model must be named.
|
||||
*
|
||||
* Exit codes: 0=ok, 3=unrecoverable error. Zero external dependencies.
|
||||
*/
|
||||
|
|
@ -32,11 +38,18 @@ import { parseFrontmatter } from './lib/yaml-parser.mjs';
|
|||
import { loadRegister, getEntry } from './lib/best-practices-register.mjs';
|
||||
import { prefilterClaudeMd, LENS_DETECTORS } from './lib/lens-prefilter.mjs';
|
||||
import { subtractionCandidates, SUBTRACT_DETECTORS } from './lib/subtraction-prefilter.mjs';
|
||||
import { matchModelScope, normalizeModel } from './lib/prompting-model-scope.mjs';
|
||||
import { scan as optScan } from './optimization-lens-scanner.mjs';
|
||||
import { requireValidArgs } from './lib/cli-args.mjs';
|
||||
|
||||
/** Flag surface, measured 2026-08-09. Anything else is exit 3. */
|
||||
const ARG_SPEC = { boolean: ['--global', '--subtract'], value: ['--output-file'] };
|
||||
const ARG_SPEC = {
|
||||
boolean: ['--global', '--subtract'],
|
||||
// `--for-model` is a VALUE flag, so a bare `--for-model` is reported as
|
||||
// "needs a value" rather than "unknown flag" — the two are different failures
|
||||
// and reporting the wrong one hides which mistake the caller made.
|
||||
value: ['--output-file', '--for-model'],
|
||||
};
|
||||
|
||||
// Files under `.claude/plugins/` are shipped by an installed plugin — vendored
|
||||
// CLAUDE.md plus its bundled tests/fixtures and examples. They are not the user's
|
||||
|
|
@ -60,11 +73,13 @@ async function main() {
|
|||
let outputFile = null;
|
||||
let includeGlobal = false;
|
||||
let subtract = false;
|
||||
let targetModel = null;
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
if (args[i] === '--global') includeGlobal = true;
|
||||
else if (args[i] === '--subtract') subtract = true;
|
||||
else if (args[i] === '--output-file' && args[i + 1]) outputFile = args[++i];
|
||||
else if (args[i] === '--for-model' && args[i + 1]) targetModel = args[++i];
|
||||
else if (!args[i].startsWith('-')) targetPath = args[i];
|
||||
}
|
||||
|
||||
|
|
@ -108,6 +123,23 @@ async function main() {
|
|||
// fire on a plain `/config-audit optimize` run (brief §7 q3).
|
||||
const subtractCands = [];
|
||||
const subtractEntry = subtract && register ? confirmedEntry(register, 'BP-SUB-001') : null;
|
||||
// Model-scoped annotation entries (BP-PROMPT-001 and any later sibling). These
|
||||
// never produce candidates of their own — they only tag candidates the
|
||||
// BP-SUB-001 detector above already surfaced.
|
||||
const promptEntries =
|
||||
subtract && register
|
||||
? (register.entries || []).filter(
|
||||
(e) => e.category === 'prompting-fit' && e.confidence === 'confirmed',
|
||||
)
|
||||
: [];
|
||||
// `recognized` is reported separately from the match count so a typo'd model
|
||||
// name is distinguishable from a config that genuinely carries nothing.
|
||||
const modelRecognized =
|
||||
!!targetModel &&
|
||||
promptEntries.some((e) =>
|
||||
(e.modelScope || []).some((m) => normalizeModel(m) === normalizeModel(targetModel)),
|
||||
);
|
||||
let modelMatchedCount = 0;
|
||||
|
||||
for (const file of claudeMdFiles) {
|
||||
let content;
|
||||
|
|
@ -122,6 +154,8 @@ async function main() {
|
|||
|
||||
if (subtractEntry) {
|
||||
for (const cand of subtractionCandidates(body)) {
|
||||
const modelScope = matchModelScope(cand.text, targetModel, promptEntries);
|
||||
if (modelScope) modelMatchedCount++;
|
||||
subtractCands.push({
|
||||
file: file.absPath,
|
||||
line: bodyStartLine - 1 + cand.startLine,
|
||||
|
|
@ -137,6 +171,9 @@ async function main() {
|
|||
severity: subtractEntry.severity || 'low',
|
||||
source: subtractEntry.source,
|
||||
},
|
||||
// Spread only when matched: a run without --for-model must not grow
|
||||
// even a key set to undefined.
|
||||
...(modelScope ? { modelScope } : {}),
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
@ -217,6 +254,15 @@ async function main() {
|
|||
: [],
|
||||
detectors: SUBTRACT_DETECTORS.map((d) => ({ ...d })),
|
||||
};
|
||||
// Present ONLY when a model was named — a plain --subtract run stays
|
||||
// byte-identical to the pre-flag payload.
|
||||
if (targetModel) {
|
||||
payload.subtract.forModel = {
|
||||
requested: targetModel,
|
||||
recognized: modelRecognized,
|
||||
matchedCount: modelMatchedCount,
|
||||
};
|
||||
}
|
||||
payload.counts.subtractCandidates = subtractCands.length;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -139,6 +139,73 @@ describe('bundled register integrity (Verifiseringsplikt)', () => {
|
|||
);
|
||||
});
|
||||
|
||||
// The model-scoped prompting entry is an ANNOTATION on an existing BP-SUB-001
|
||||
// candidate, not a detector of its own — same discipline as BP-JUDG-001 above,
|
||||
// and asserted for the same reason: without this, a later session "completes"
|
||||
// the entry by wiring a second SUBTRACT_DETECTORS entry, which would both
|
||||
// widen the candidate set and collide with BP-SUB-001 on de-dup.
|
||||
//
|
||||
// `modelScope` is pinned because it is a DATA CONTRACT, not a label:
|
||||
// `matchModelScope` looks the requested `--for-model` name up in exactly this
|
||||
// array. Renaming or dropping it makes every lookup silently return null —
|
||||
// the CLI would report `recognized: false` for a model the register still
|
||||
// claims to cover, which is a quiet wrong answer, not a loud failure.
|
||||
it('carries the model-scoped prompting entry (BP-PROMPT-001) as an annotation, not a detector', () => {
|
||||
const e = getEntry(reg, 'BP-PROMPT-001');
|
||||
assert.ok(e, 'BP-PROMPT-001 missing from the bundled register');
|
||||
assert.equal(e.confidence, 'confirmed', 'BP-PROMPT-001 must be confirmed');
|
||||
assert.equal(e.category, 'prompting-fit', 'BP-PROMPT-001 wrong category');
|
||||
assert.equal(
|
||||
e.mechanism,
|
||||
'deletion',
|
||||
'BP-PROMPT-001 rides the subtraction axis — an addition-shaped claim must not reuse this entry'
|
||||
);
|
||||
assert.equal(
|
||||
e.lensCheck ?? null,
|
||||
null,
|
||||
'BP-PROMPT-001 must NOT name a lensCheck — it annotates BP-SUB-001 candidates, it does not detect'
|
||||
);
|
||||
assert.deepStrictEqual(
|
||||
e.modelScope,
|
||||
['opus-5'],
|
||||
'BP-PROMPT-001 modelScope is the lookup key matchModelScope resolves --for-model against'
|
||||
);
|
||||
assert.equal(
|
||||
e.source.url,
|
||||
'https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5',
|
||||
'BP-PROMPT-001 primary source must be the official Opus 5 prompting guide'
|
||||
);
|
||||
// The guide carries no visible publish/last-updated date (re-checked
|
||||
// 2026-08-12). Asserting its ABSENCE keeps a later session from inventing
|
||||
// one to satisfy a pattern the other entries happen to have.
|
||||
assert.equal(
|
||||
e.source.published,
|
||||
undefined,
|
||||
'BP-PROMPT-001 source has no visible published date — do not fabricate one'
|
||||
);
|
||||
assert.match(
|
||||
String(e.note),
|
||||
/--for-model/,
|
||||
'BP-PROMPT-001 must record that it is gated behind an explicit flag'
|
||||
);
|
||||
// The corpus numbers live in the entry, not only in a gitignored fasit,
|
||||
// for the same reason BP-JUDG-001's 409 does: a later session that cannot
|
||||
// see the measurement re-derives it, and the cheapest re-derivation is to
|
||||
// read the zero as a broken detector and loosen it. 31 is the load-bearing
|
||||
// half — how many verb-only blocks the TARGET requirement rejected, i.e.
|
||||
// the false positives a naive version of this check would have produced.
|
||||
assert.match(
|
||||
String(e.note),
|
||||
/409/,
|
||||
'BP-PROMPT-001 must carry the corpus size it was measured against'
|
||||
);
|
||||
assert.match(
|
||||
String(e.note),
|
||||
/(^|\s)31(\s|,)/,
|
||||
'BP-PROMPT-001 must carry the count the target requirement rejected — the reason it is narrow'
|
||||
);
|
||||
});
|
||||
|
||||
// The register's own consumers key on lensCheck; an entry that names one it
|
||||
// does not have would reach a payload with no detector behind it.
|
||||
it('every lensCheck in the register is backed by a detector', () => {
|
||||
|
|
|
|||
115
tests/lib/prompting-model-scope.test.mjs
Normal file
115
tests/lib/prompting-model-scope.test.mjs
Normal file
|
|
@ -0,0 +1,115 @@
|
|||
/**
|
||||
* prompting-model-scope tests — the model-scoped ANNOTATION layer on top of the
|
||||
* subtraction lens's single `compensatory-instruction` detector (BP-SUB-001).
|
||||
*
|
||||
* This module does NOT generate new subtraction candidates. It answers a
|
||||
* narrower question over text that already passed the general detector: does
|
||||
* this block specifically claim a model no longer needs — self-verification, or
|
||||
* delegated-to-a-subagent verification? Precision comes from requiring a
|
||||
* reflexive/delegate TARGET alongside the verify verb, not from narrowing the
|
||||
* verb list — a bare "check"/"verify" also matches EXTERNAL verification
|
||||
* ("check the CI status"), which must stay a true negative here even though it
|
||||
* is (correctly) still a BP-SUB-001 candidate upstream.
|
||||
*/
|
||||
import { describe, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import {
|
||||
isModelContradictedVerification,
|
||||
matchModelScope,
|
||||
normalizeModel,
|
||||
} from '../../scanners/lib/prompting-model-scope.mjs';
|
||||
|
||||
describe('isModelContradictedVerification — true positives', () => {
|
||||
const positives = [
|
||||
'Always double-check your own work before responding.',
|
||||
'Re-verify before responding to the user.',
|
||||
'Verify complex changes with a subagent.',
|
||||
'Use a subagent to verify the diff before finishing.',
|
||||
'Dobbeltsjekk ditt eget arbeid før du svarer.',
|
||||
'Verifiser med en subagent før du er ferdig.',
|
||||
];
|
||||
for (const text of positives) {
|
||||
it(`matches: "${text}"`, () => {
|
||||
assert.equal(isModelContradictedVerification(text), true);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('isModelContradictedVerification — true negatives (external verification)', () => {
|
||||
const negatives = [
|
||||
'Check the CI status before merging.',
|
||||
'Verify the deployment succeeded.',
|
||||
'Sjekk build-status i CI før du merger.',
|
||||
'Review the pull request for style issues.',
|
||||
'Confirm the ticket is assigned to you.',
|
||||
'Read the file before editing it.',
|
||||
];
|
||||
for (const text of negatives) {
|
||||
it(`does not match: "${text}"`, () => {
|
||||
assert.equal(isModelContradictedVerification(text), false);
|
||||
});
|
||||
}
|
||||
});
|
||||
|
||||
describe('normalizeModel', () => {
|
||||
it('lowercases and strips non-alphanumerics', () => {
|
||||
for (const s of ['opus-5', 'Opus 5', 'OPUS5', 'opus_5']) {
|
||||
assert.equal(normalizeModel(s), 'opus5');
|
||||
}
|
||||
});
|
||||
it('handles null/undefined/empty', () => {
|
||||
assert.equal(normalizeModel(null), '');
|
||||
assert.equal(normalizeModel(undefined), '');
|
||||
assert.equal(normalizeModel(''), '');
|
||||
});
|
||||
});
|
||||
|
||||
describe('matchModelScope', () => {
|
||||
const entries = [
|
||||
{ id: 'BP-PROMPT-001', claim: 'test claim', modelScope: ['opus-5'] },
|
||||
];
|
||||
const positiveText = 'Always double-check your own work.';
|
||||
|
||||
it('returns null when targetModel is absent', () => {
|
||||
assert.equal(matchModelScope(positiveText, null, entries), null);
|
||||
assert.equal(matchModelScope(positiveText, undefined, entries), null);
|
||||
assert.equal(matchModelScope(positiveText, '', entries), null);
|
||||
});
|
||||
|
||||
it('returns null when the text is not a model-contradicted verification claim', () => {
|
||||
assert.equal(matchModelScope('Check the CI status.', 'opus-5', entries), null);
|
||||
});
|
||||
|
||||
it('returns null when no entry matches the requested model', () => {
|
||||
assert.equal(matchModelScope(positiveText, 'sonnet-5', entries), null);
|
||||
});
|
||||
|
||||
it('returns null when entries is empty', () => {
|
||||
assert.equal(matchModelScope(positiveText, 'opus-5', []), null);
|
||||
});
|
||||
|
||||
it('matches on an exact normalized model name', () => {
|
||||
const m = matchModelScope(positiveText, 'opus-5', entries);
|
||||
assert.ok(m);
|
||||
assert.equal(m.registerId, 'BP-PROMPT-001');
|
||||
assert.equal(m.claim, 'test claim');
|
||||
assert.equal(m.requestedModel, 'opus-5');
|
||||
});
|
||||
|
||||
it('matches across normalization variants of the same model name', () => {
|
||||
for (const variant of ['Opus 5', 'OPUS-5', 'opus5']) {
|
||||
const m = matchModelScope(positiveText, variant, entries);
|
||||
assert.ok(m, `expected a match for "${variant}"`);
|
||||
assert.equal(m.registerId, 'BP-PROMPT-001');
|
||||
}
|
||||
});
|
||||
|
||||
it('picks the entry whose modelScope contains the requested model, ignoring others', () => {
|
||||
const multi = [
|
||||
{ id: 'BP-PROMPT-OTHER', claim: 'unrelated', modelScope: ['haiku-5'] },
|
||||
{ id: 'BP-PROMPT-001', claim: 'test claim', modelScope: ['opus-5'] },
|
||||
];
|
||||
const m = matchModelScope(positiveText, 'opus-5', multi);
|
||||
assert.equal(m.registerId, 'BP-PROMPT-001');
|
||||
});
|
||||
});
|
||||
|
|
@ -100,3 +100,157 @@ describe('optimize-lens-cli — candidate identity (M-BUG-11)', () => {
|
|||
}
|
||||
});
|
||||
});
|
||||
|
||||
/**
|
||||
* `--for-model` — the model-scoped ANNOTATION layer (BP-PROMPT-001).
|
||||
*
|
||||
* The flag never widens the candidate set: it tags a SUBSET of the candidates
|
||||
* the general `compensatory-instruction` detector (BP-SUB-001) already produced.
|
||||
* The fixture below is chosen to prove exactly that — all three lines are
|
||||
* BP-SUB-001 candidates, but only two carry a reflexive/delegate verification
|
||||
* target. The third ("check the CI status") is EXTERNAL verification: still a
|
||||
* subtraction candidate, never a model-scoped one. A blanket tagger would pass
|
||||
* a fixture where every candidate qualifies; this one it cannot pass.
|
||||
*
|
||||
* `recognized` exists because a typo'd model name would otherwise be a silent
|
||||
* no-op indistinguishable from "your config is already clean" — the CLI has to
|
||||
* report whether it knew the name, not just a bare zero.
|
||||
*/
|
||||
const VERIFY_MD = [
|
||||
'# Project',
|
||||
'',
|
||||
'## Workflow',
|
||||
'',
|
||||
'Always double-check your own work before responding to the user.',
|
||||
'',
|
||||
'Verify complex changes with a subagent before you finish.',
|
||||
'',
|
||||
'Check the CI status before merging any pull request.',
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
/** Run the lens CLI with arbitrary extra argv; never throws on a non-zero exit. */
|
||||
async function runLensArgs(files, extraArgs) {
|
||||
const root = await mkdtemp(join(tmpdir(), 'ca-lens-model-'));
|
||||
try {
|
||||
for (const [rel, content] of Object.entries(files)) {
|
||||
const abs = join(root, rel);
|
||||
await mkdir(join(abs, '..'), { recursive: true });
|
||||
await writeFile(abs, content, 'utf-8');
|
||||
}
|
||||
const out = join(root, 'payload.json');
|
||||
let code = 0;
|
||||
let stderr = '';
|
||||
try {
|
||||
const r = await execFileP('node', [CLI, root, '--output-file', out, ...extraArgs]);
|
||||
stderr = r.stderr || '';
|
||||
} catch (err) {
|
||||
code = typeof err.code === 'number' ? err.code : 1;
|
||||
stderr = err.stderr || '';
|
||||
}
|
||||
let payload = null;
|
||||
let raw = null;
|
||||
try {
|
||||
raw = await readFile(out, 'utf-8');
|
||||
payload = JSON.parse(raw);
|
||||
} catch {
|
||||
payload = null;
|
||||
}
|
||||
// Every run gets its own temp root, so absolute paths differ between two
|
||||
// otherwise-identical runs. Normalizing the root is what makes a byte
|
||||
// comparison across runs mean "same payload" rather than "same directory".
|
||||
const normalized = raw === null ? null : raw.split(root).join('<ROOT>');
|
||||
return { code, stderr, payload, normalized };
|
||||
} finally {
|
||||
await rm(root, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
describe('optimize-lens-cli — --for-model argument surface', () => {
|
||||
it('rejects a bare --for-model as "needs a value", not "unknown flag"', async () => {
|
||||
const { code, stderr } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract', '--for-model']);
|
||||
assert.equal(code, 3, 'a malformed flag is exit 3 (it did not do the job)');
|
||||
assert.match(stderr, /value/i, `expected a "needs a value" message, got: ${stderr}`);
|
||||
assert.doesNotMatch(
|
||||
stderr,
|
||||
/unknown/i,
|
||||
'--for-model must be a KNOWN value flag; reporting it as unknown hides the real error'
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe('optimize-lens-cli — --for-model annotation', () => {
|
||||
it('tags only the candidates whose verification target is reflexive or delegated', async () => {
|
||||
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
|
||||
'--subtract',
|
||||
'--for-model',
|
||||
'opus-5',
|
||||
]);
|
||||
assert.equal(code, 0);
|
||||
const cands = payload.subtract.candidates;
|
||||
assert.equal(cands.length, 3, 'fixture must produce three BP-SUB-001 candidates');
|
||||
|
||||
const tagged = cands.filter((c) => c.modelScope);
|
||||
assert.equal(tagged.length, 2, 'exactly the two self/delegate-targeted blocks are model-scoped');
|
||||
for (const t of tagged) {
|
||||
assert.equal(t.modelScope.registerId, 'BP-PROMPT-001');
|
||||
assert.equal(t.modelScope.requestedModel, 'opus-5');
|
||||
assert.ok(t.modelScope.claim, 'the citation must carry the register claim');
|
||||
}
|
||||
|
||||
const external = cands.find((c) => /CI status/.test(c.signalText));
|
||||
assert.ok(external, 'the external-verification candidate must still be present');
|
||||
assert.equal(
|
||||
external.modelScope,
|
||||
undefined,
|
||||
'external verification is a BP-SUB-001 candidate but NOT model-scoped'
|
||||
);
|
||||
});
|
||||
|
||||
it('reports the model as recognized, with a match count', async () => {
|
||||
const { payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
|
||||
'--subtract',
|
||||
'--for-model',
|
||||
'Opus 5',
|
||||
]);
|
||||
assert.deepStrictEqual(payload.subtract.forModel, {
|
||||
requested: 'Opus 5',
|
||||
recognized: true,
|
||||
matchedCount: 2,
|
||||
});
|
||||
});
|
||||
|
||||
it('reports an unrecognized model honestly instead of a silent zero', async () => {
|
||||
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
|
||||
'--subtract',
|
||||
'--for-model',
|
||||
'oppus5',
|
||||
]);
|
||||
assert.equal(code, 0, 'an unknown model name is not an error — it is a reported non-match');
|
||||
assert.deepStrictEqual(payload.subtract.forModel, {
|
||||
requested: 'oppus5',
|
||||
recognized: false,
|
||||
matchedCount: 0,
|
||||
});
|
||||
for (const c of payload.subtract.candidates) {
|
||||
assert.equal(c.modelScope, undefined, 'no candidate may be tagged for an unrecognized model');
|
||||
}
|
||||
});
|
||||
|
||||
it('is a no-op without --subtract, matching the --apply-without-subtract shape', async () => {
|
||||
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--for-model', 'opus-5']);
|
||||
assert.equal(code, 0);
|
||||
assert.equal(payload.subtract, undefined, '--for-model alone must not switch the subtraction axis on');
|
||||
});
|
||||
|
||||
it('leaves a plain --subtract run byte-identical to a run without the flag', async () => {
|
||||
// The additive-payload invariant: --for-model is the ONLY path that grows a
|
||||
// field. Asserted on the serialized bytes, because a key added with an
|
||||
// undefined value would pass a shallow key check and still change the shape.
|
||||
const before = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract']);
|
||||
assert.equal(before.code, 0);
|
||||
assert.equal(before.payload.subtract.forModel, undefined, 'no forModel key without --for-model');
|
||||
const again = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract']);
|
||||
assert.equal(again.normalized, before.normalized, 'a --subtract run must be byte-stable');
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue