open/config-audit
1
0
Fork 0
Code Issues Releases 1 Activity

feat(scanners): a redundancy claim that belongs to one model is scoped to it

Browse source 
Anthropic documents that Claude Opus 5 verifies its own work, and that telling
it to double-check or to delegate verification to a subagent causes
over-verification -- token cost with no quality gain. The general subtraction
detector (BP-SUB-001) already surfaces those blocks for every user, with no
model-awareness at all.

`optimize --subtract --for-model <name>` adds the missing half. It ANNOTATES a
subset of the candidates --subtract already produced; it is not a second
detector and can never widen the candidate set. A second SUBTRACT_DETECTORS
entry would have collided with BP-SUB-001 on de-dup, and a prose-only signal in
the agent prompt would have been untestable.

There is no auto-detection, by measurement rather than omission: a CLAUDE.md has
no frontmatter and no resolvable target model, and this operator's own `route`
skill deliberately runs a different model per session -- the same file is read
by whichever model comes next. So the model is named, and the citation is
reported as conditional everywhere a human sees it (agent report copy, and the
Step 7a listing that is the last surface before an approval file).

Precision comes from the TARGET, not the verb list. Measured across the
409-file corpus: 392 BP-SUB-001 candidates, 31 (7.9%) carry a verify verb, and
0 also carry a reflexive or delegated target. Two independent raw-text greps
found 0 as well, so the zero is the corpus rather than an over-narrow regex.
Those 31 verb-only blocks -- "sjekk relevante config-filer", "Type-sjekk:
pyright", "To verify plugin functionality" -- are exactly the false positives a
verb-only version would have produced, which is BP-JUDG-001's 7/7 failure
arriving one lens over. The numbers live in the register entry's note and are
pinned by a test, because a session that cannot see the measurement reads the
zero as a broken detector and loosens it.

`recognized` is reported separately from `matchedCount`: a typo'd model name and
a genuinely clean config both yield zero, and without the distinction the CLI
would report a silent no-op as good news. Dogfooded on the real machine --
`opus-5` gives recognized:true/matchedCount:0, `oppus5` gives recognized:false.

source.published is absent because the guide carries no visible publish date;
its absence is asserted so a later session does not invent one to match the
other entries' shape. Both quoted sentences were verified verbatim 2026-08-12.

The payload stays additive -- forModel and per-candidate modelScope appear only
under the flag, so a plain --subtract run is byte-identical to before (asserted
on the serialized bytes, since a key set to undefined passes a shallow check).

Suite 1724 -> 1752 (+28). The one remaining failure is the pre-existing
drift-cli --output-file crash, untouched by this work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01BRuXt6tZyowi8QYNKLSHQm
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 23:16:23 +02:00
commit 7df8e0d65b
11 changed files with 650 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

67
tests/lib/best-practices-register.test.mjs
View file

@ -139,6 +139,73 @@ describe('bundled register integrity (Verifiseringsplikt)', () => {
);
});
// The model-scoped prompting entry is an ANNOTATION on an existing BP-SUB-001
// candidate, not a detector of its own — same discipline as BP-JUDG-001 above,
// and asserted for the same reason: without this, a later session "completes"
// the entry by wiring a second SUBTRACT_DETECTORS entry, which would both
// widen the candidate set and collide with BP-SUB-001 on de-dup.
//
// `modelScope` is pinned because it is a DATA CONTRACT, not a label:
// `matchModelScope` looks the requested `--for-model` name up in exactly this
// array. Renaming or dropping it makes every lookup silently return null —
// the CLI would report `recognized: false` for a model the register still
// claims to cover, which is a quiet wrong answer, not a loud failure.
it('carries the model-scoped prompting entry (BP-PROMPT-001) as an annotation, not a detector', () => {
const e = getEntry(reg, 'BP-PROMPT-001');
assert.ok(e, 'BP-PROMPT-001 missing from the bundled register');
assert.equal(e.confidence, 'confirmed', 'BP-PROMPT-001 must be confirmed');
assert.equal(e.category, 'prompting-fit', 'BP-PROMPT-001 wrong category');
assert.equal(
e.mechanism,
'deletion',
'BP-PROMPT-001 rides the subtraction axis — an addition-shaped claim must not reuse this entry'
);
assert.equal(
e.lensCheck ?? null,
null,
'BP-PROMPT-001 must NOT name a lensCheck — it annotates BP-SUB-001 candidates, it does not detect'
);
assert.deepStrictEqual(
e.modelScope,
['opus-5'],
'BP-PROMPT-001 modelScope is the lookup key matchModelScope resolves --for-model against'
);
assert.equal(
e.source.url,
'https://platform.claude.com/docs/en/build-with-claude/prompt-engineering/prompting-claude-opus-5',
'BP-PROMPT-001 primary source must be the official Opus 5 prompting guide'
);
// The guide carries no visible publish/last-updated date (re-checked
// 2026-08-12). Asserting its ABSENCE keeps a later session from inventing
// one to satisfy a pattern the other entries happen to have.
assert.equal(
e.source.published,
undefined,
'BP-PROMPT-001 source has no visible published date — do not fabricate one'
);
assert.match(
String(e.note),
/--for-model/,
'BP-PROMPT-001 must record that it is gated behind an explicit flag'
);
// The corpus numbers live in the entry, not only in a gitignored fasit,
// for the same reason BP-JUDG-001's 409 does: a later session that cannot
// see the measurement re-derives it, and the cheapest re-derivation is to
// read the zero as a broken detector and loosen it. 31 is the load-bearing
// half — how many verb-only blocks the TARGET requirement rejected, i.e.
// the false positives a naive version of this check would have produced.
assert.match(
String(e.note),
/409/,
'BP-PROMPT-001 must carry the corpus size it was measured against'
);
assert.match(
String(e.note),
/(^|\s)31(\s|,)/,
'BP-PROMPT-001 must carry the count the target requirement rejected — the reason it is narrow'
);
});
// The register's own consumers key on lensCheck; an entry that names one it
// does not have would reach a payload with no detector behind it.
it('every lensCheck in the register is backed by a detector', () => {

115
tests/lib/prompting-model-scope.test.mjs Normal file
View file

@ -0,0 +1,115 @@
/**
* prompting-model-scope tests — the model-scoped ANNOTATION layer on top of the
* subtraction lens's single `compensatory-instruction` detector (BP-SUB-001).
*
* This module does NOT generate new subtraction candidates. It answers a
* narrower question over text that already passed the general detector: does
* this block specifically claim a model no longer needs — self-verification, or
* delegated-to-a-subagent verification? Precision comes from requiring a
* reflexive/delegate TARGET alongside the verify verb, not from narrowing the
* verb list — a bare "check"/"verify" also matches EXTERNAL verification
* ("check the CI status"), which must stay a true negative here even though it
* is (correctly) still a BP-SUB-001 candidate upstream.
*/
import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import {
isModelContradictedVerification,
matchModelScope,
normalizeModel,
} from '../../scanners/lib/prompting-model-scope.mjs';
describe('isModelContradictedVerification — true positives', () => {
const positives = [
'Always double-check your own work before responding.',
'Re-verify before responding to the user.',
'Verify complex changes with a subagent.',
'Use a subagent to verify the diff before finishing.',
'Dobbeltsjekk ditt eget arbeid før du svarer.',
'Verifiser med en subagent før du er ferdig.',
];
for (const text of positives) {
it(`matches: "${text}"`, () => {
assert.equal(isModelContradictedVerification(text), true);
});
}
});
describe('isModelContradictedVerification — true negatives (external verification)', () => {
const negatives = [
'Check the CI status before merging.',
'Verify the deployment succeeded.',
'Sjekk build-status i CI før du merger.',
'Review the pull request for style issues.',
'Confirm the ticket is assigned to you.',
'Read the file before editing it.',
];
for (const text of negatives) {
it(`does not match: "${text}"`, () => {
assert.equal(isModelContradictedVerification(text), false);
});
}
});
describe('normalizeModel', () => {
it('lowercases and strips non-alphanumerics', () => {
for (const s of ['opus-5', 'Opus 5', 'OPUS5', 'opus_5']) {
assert.equal(normalizeModel(s), 'opus5');
}
});
it('handles null/undefined/empty', () => {
assert.equal(normalizeModel(null), '');
assert.equal(normalizeModel(undefined), '');
assert.equal(normalizeModel(''), '');
});
});
describe('matchModelScope', () => {
const entries = [
{ id: 'BP-PROMPT-001', claim: 'test claim', modelScope: ['opus-5'] },
];
const positiveText = 'Always double-check your own work.';
it('returns null when targetModel is absent', () => {
assert.equal(matchModelScope(positiveText, null, entries), null);
assert.equal(matchModelScope(positiveText, undefined, entries), null);
assert.equal(matchModelScope(positiveText, '', entries), null);
});
it('returns null when the text is not a model-contradicted verification claim', () => {
assert.equal(matchModelScope('Check the CI status.', 'opus-5', entries), null);
});
it('returns null when no entry matches the requested model', () => {
assert.equal(matchModelScope(positiveText, 'sonnet-5', entries), null);
});
it('returns null when entries is empty', () => {
assert.equal(matchModelScope(positiveText, 'opus-5', []), null);
});
it('matches on an exact normalized model name', () => {
const m = matchModelScope(positiveText, 'opus-5', entries);
assert.ok(m);
assert.equal(m.registerId, 'BP-PROMPT-001');
assert.equal(m.claim, 'test claim');
assert.equal(m.requestedModel, 'opus-5');
});
it('matches across normalization variants of the same model name', () => {
for (const variant of ['Opus 5', 'OPUS-5', 'opus5']) {
const m = matchModelScope(positiveText, variant, entries);
assert.ok(m, `expected a match for "${variant}"`);
assert.equal(m.registerId, 'BP-PROMPT-001');
}
});
it('picks the entry whose modelScope contains the requested model, ignoring others', () => {
const multi = [
{ id: 'BP-PROMPT-OTHER', claim: 'unrelated', modelScope: ['haiku-5'] },
{ id: 'BP-PROMPT-001', claim: 'test claim', modelScope: ['opus-5'] },
];
const m = matchModelScope(positiveText, 'opus-5', multi);
assert.equal(m.registerId, 'BP-PROMPT-001');
});
});

154
tests/scanners/optimize-lens-cli.test.mjs
View file

@ -100,3 +100,157 @@ describe('optimize-lens-cli — candidate identity (M-BUG-11)', () => {
}
});
});
/**
* `--for-model` — the model-scoped ANNOTATION layer (BP-PROMPT-001).
*
* The flag never widens the candidate set: it tags a SUBSET of the candidates
* the general `compensatory-instruction` detector (BP-SUB-001) already produced.
* The fixture below is chosen to prove exactly that — all three lines are
* BP-SUB-001 candidates, but only two carry a reflexive/delegate verification
* target. The third ("check the CI status") is EXTERNAL verification: still a
* subtraction candidate, never a model-scoped one. A blanket tagger would pass
* a fixture where every candidate qualifies; this one it cannot pass.
*
* `recognized` exists because a typo'd model name would otherwise be a silent
* no-op indistinguishable from "your config is already clean" — the CLI has to
* report whether it knew the name, not just a bare zero.
*/
const VERIFY_MD = [
'# Project',
'',
'## Workflow',
'',
'Always double-check your own work before responding to the user.',
'',
'Verify complex changes with a subagent before you finish.',
'',
'Check the CI status before merging any pull request.',
'',
].join('\n');
/** Run the lens CLI with arbitrary extra argv; never throws on a non-zero exit. */
async function runLensArgs(files, extraArgs) {
const root = await mkdtemp(join(tmpdir(), 'ca-lens-model-'));
try {
for (const [rel, content] of Object.entries(files)) {
const abs = join(root, rel);
await mkdir(join(abs, '..'), { recursive: true });
await writeFile(abs, content, 'utf-8');
}
const out = join(root, 'payload.json');
let code = 0;
let stderr = '';
try {
const r = await execFileP('node', [CLI, root, '--output-file', out, ...extraArgs]);
stderr = r.stderr || '';
} catch (err) {
code = typeof err.code === 'number' ? err.code : 1;
stderr = err.stderr || '';
}
let payload = null;
let raw = null;
try {
raw = await readFile(out, 'utf-8');
payload = JSON.parse(raw);
} catch {
payload = null;
}
// Every run gets its own temp root, so absolute paths differ between two
// otherwise-identical runs. Normalizing the root is what makes a byte
// comparison across runs mean "same payload" rather than "same directory".
const normalized = raw === null ? null : raw.split(root).join('<ROOT>');
return { code, stderr, payload, normalized };
} finally {
await rm(root, { recursive: true, force: true });
}
}
describe('optimize-lens-cli — --for-model argument surface', () => {
it('rejects a bare --for-model as "needs a value", not "unknown flag"', async () => {
const { code, stderr } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract', '--for-model']);
assert.equal(code, 3, 'a malformed flag is exit 3 (it did not do the job)');
assert.match(stderr, /value/i, `expected a "needs a value" message, got: ${stderr}`);
assert.doesNotMatch(
stderr,
/unknown/i,
'--for-model must be a KNOWN value flag; reporting it as unknown hides the real error'
);
});
});
describe('optimize-lens-cli — --for-model annotation', () => {
it('tags only the candidates whose verification target is reflexive or delegated', async () => {
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
'--subtract',
'--for-model',
'opus-5',
]);
assert.equal(code, 0);
const cands = payload.subtract.candidates;
assert.equal(cands.length, 3, 'fixture must produce three BP-SUB-001 candidates');
const tagged = cands.filter((c) => c.modelScope);
assert.equal(tagged.length, 2, 'exactly the two self/delegate-targeted blocks are model-scoped');
for (const t of tagged) {
assert.equal(t.modelScope.registerId, 'BP-PROMPT-001');
assert.equal(t.modelScope.requestedModel, 'opus-5');
assert.ok(t.modelScope.claim, 'the citation must carry the register claim');
}
const external = cands.find((c) => /CI status/.test(c.signalText));
assert.ok(external, 'the external-verification candidate must still be present');
assert.equal(
external.modelScope,
undefined,
'external verification is a BP-SUB-001 candidate but NOT model-scoped'
);
});
it('reports the model as recognized, with a match count', async () => {
const { payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
'--subtract',
'--for-model',
'Opus 5',
]);
assert.deepStrictEqual(payload.subtract.forModel, {
requested: 'Opus 5',
recognized: true,
matchedCount: 2,
});
});
it('reports an unrecognized model honestly instead of a silent zero', async () => {
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, [
'--subtract',
'--for-model',
'oppus5',
]);
assert.equal(code, 0, 'an unknown model name is not an error — it is a reported non-match');
assert.deepStrictEqual(payload.subtract.forModel, {
requested: 'oppus5',
recognized: false,
matchedCount: 0,
});
for (const c of payload.subtract.candidates) {
assert.equal(c.modelScope, undefined, 'no candidate may be tagged for an unrecognized model');
}
});
it('is a no-op without --subtract, matching the --apply-without-subtract shape', async () => {
const { code, payload } = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--for-model', 'opus-5']);
assert.equal(code, 0);
assert.equal(payload.subtract, undefined, '--for-model alone must not switch the subtraction axis on');
});
it('leaves a plain --subtract run byte-identical to a run without the flag', async () => {
// The additive-payload invariant: --for-model is the ONLY path that grows a
// field. Asserted on the serialized bytes, because a key added with an
// undefined value would pass a shallow key check and still change the shape.
const before = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract']);
assert.equal(before.code, 0);
assert.equal(before.payload.subtract.forModel, undefined, 'no forModel key without --for-model');
const again = await runLensArgs({ 'CLAUDE.md': VERIFY_MD }, ['--subtract']);
assert.equal(again.normalized, before.normalized, 'a --subtract run must be byte-stable');
});
});