BREAKING CHANGE: the {NNN} in CA-{SCANNER}-{NNN} identifies the check that
produced the finding. It used to be the finding's position in that scanner's
output for that run, which made it unstable across CONFIGURATIONS, not just
across releases as STATE framed it. Measured on two fixtures: "No custom
subagents" was CA-GAP-007 on minimal-project and CA-GAP-004 on healthy-project.
A user who fixed an unrelated earlier gap silently renumbered every later one,
so a .config-audit-ignore pin retargeted to a neighbouring finding with no
version change at all.
Second measured arm: README already documented the opposite scheme. It and the
scanner headers describe ~20 numbers as check codes (CA-SKL-003 = oversized
body, CA-PLH-015 = folder shadowing, CA-TOK-006 = schema deferral), and the
counter could only produce those in the all-fire case -- source-order positions
are 4, 3 and 8. The documentation described the scheme; the implementation was
what was wrong. Every published number is preserved by construction and pinned
exhaustively in tests/lib/finding-codes.test.mjs.
scanners/lib/finding-codes.mjs is the single authority. Every finding() call
passes a `code`; an undeclared or missing one THROWS. No counter fallback --
that would reproduce D1's findGapId -> 'unknown' silent degradation and let a
half-converted scanner ship IDs that look valid. findingCounter/resetCounter
are deleted outright, not left as no-ops. Retirement is now a mechanism:
RETIRED_CODES tombstones a withdrawn key so its number is never reissued,
seeded with GAP t3_8 -- the D1 removal that opened this chunk.
IDs are consequently NOT unique per finding: one check failing in three files
emits three findings sharing an ID. That inverts which consumer is correct, so
every f.id/findingId site was classified before the change. diff-engine and
most of fix-engine already keyed on scanner+title+file (drift was never lying);
fix-engine's verification did not, and keyed on the ID alone -- fixing one of
two sibling instances marked both fixed, and the untouched one, still present
in the re-scan, was reported as a REGRESSION. Red test first, then keyed on
(findingId, file), which both planFixes and applyFixes already carry.
plugin-health's crossIds Set was measured and is a clean negative: cross
findings are allFindings.slice(crossPluginStart) and codes 18/19 are emitted
only in that tail, so the partition holds by construction.
unknownSuppressions() reports a pin that names no declared check, in the
--output-file payload (ux-rules rule 2 -- a stderr-only warning is invisible to
the commands) and only when one exists, so a clean config is byte-identical.
That is what makes the break safe: a stale pin goes loud instead of dying quiet.
Frozen tests/snapshots/v5.0.0/ untouched on disk. IDs are masked out of that
comparison (mask-finding-ids.mjs) rather than re-derived -- re-deriving
positional IDs would assert the retired scheme against itself, and #58's
isGapEntry off-by-one is the measured example of that misfiring. The dead
re-derivation is removed from strip-retired-gap.mjs. default-output snapshots
re-approved after confirming the diff is IDs and nothing else.
Guards, each seen red against its own defect: a missing code (scanner errors
out mid-sweep), an orphan declaration, a resurrected retired key, and a
documented ID naming no check. The sweep asserts the union across all 16
scanners, never per scanner -- a per-scanner assertion goes green on a partial
conversion.
Fasit written before implementation: docs/mbug28-id-semantics-fasit.local.md,
including one correction made before running (CML has 12 checks over 13 call
sites -- the anchored and calibrated char-budget arms are one check, which a
repeated-title sweep found and my call-site count had missed).
Suite 1535 -> 1573, 0 failing.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MyqCQKK2ornJ1jFWwqx17E
182 lines
8.2 KiB
JavaScript
182 lines
8.2 KiB
JavaScript
/**
|
|
* OST Scanner — Output-style validation (v5.6 C)
|
|
*
|
|
* Output styles are live (the standalone `/output-style` command was removed in
|
|
* v2.1.91; styles are now managed via `/config`). They are the most surprising
|
|
* steering surface because they rewrite the system prompt:
|
|
*
|
|
* CA-OST-001 A custom (user/project) output style that does NOT set
|
|
* `keep-coding-instructions: true` → when active, Claude Code
|
|
* REMOVES its built-in software-engineering instructions (how to
|
|
* scope changes, write comments, verify work) and keeps only the
|
|
* style's text. `keep-coding-instructions` defaults to false, so
|
|
* this is the headline footgun. Severity medium.
|
|
*
|
|
* CA-OST-002 A PLUGIN output style with `force-for-plugin: true` → Claude
|
|
* Code auto-applies it whenever the plugin is enabled, OVERRIDING
|
|
* the user's selected `outputStyle`. If several enabled plugins set
|
|
* it, the first loaded wins. Severity low (awareness). Note:
|
|
* `force-for-plugin` is plugin-styles-only per the docs, so this
|
|
* keys on `source === 'plugin'` — a user/project style cannot
|
|
* trigger the override (it would simply be ignored).
|
|
*
|
|
* CA-OST-003 A settings `outputStyle` value that matches no built-in and no
|
|
* discovered custom style → dead config: Claude Code falls back to
|
|
* the default style, so the configured behavior is silently not
|
|
* applied. Severity medium.
|
|
*
|
|
* Every claim traces to a CONFIRMED row of docs/v5.5-steering-model-plan.md
|
|
* (V9/V10/V11/V12), verified against code.claude.com/docs/en/output-styles and
|
|
* .../plugins-reference. The scanner is fixture-gated: with no output styles and
|
|
* no `outputStyle` setting it emits nothing (keeps the SC-5 snapshot byte-stable).
|
|
*
|
|
* Zero external dependencies.
|
|
*/
|
|
|
|
import { readFile } from 'node:fs/promises';
|
|
import { finding, scannerResult } from './lib/output.mjs';
|
|
import { SEVERITY } from './lib/severity.mjs';
|
|
import { readActiveConfig } from './lib/active-config-reader.mjs';
|
|
import { parseFrontmatter, parseJson } from './lib/yaml-parser.mjs';
|
|
|
|
const SCANNER = 'OST';
|
|
|
|
// Built-in output styles, verified against code.claude.com/docs/en/output-styles.
|
|
// Compared case-insensitively so OST-003 never false-flags a valid built-in.
|
|
const BUILTIN_STYLES = ['default', 'explanatory', 'learning', 'proactive'];
|
|
|
|
/**
|
|
* Read + parse the frontmatter of each enumerated output style once.
|
|
* @param {Array<object>} styles - readActiveConfig().outputStyles entries
|
|
*/
|
|
async function withFrontmatter(styles) {
|
|
const out = [];
|
|
for (const s of styles) {
|
|
let frontmatter = null;
|
|
try {
|
|
frontmatter = parseFrontmatter(await readFile(s.path, 'utf-8')).frontmatter;
|
|
} catch { /* unreadable → treat as no frontmatter */ }
|
|
out.push({ ...s, frontmatter });
|
|
}
|
|
return out;
|
|
}
|
|
|
|
/**
|
|
* Resolve the effective `outputStyle` setting from the cascade (user → project →
|
|
* local; later scope wins). Returns null when unset everywhere.
|
|
* @param {object} activeConfig
|
|
* @returns {Promise<{value:string, scope:string, path:string} | null>}
|
|
*/
|
|
async function resolveOutputStyleSetting(activeConfig) {
|
|
const cascade = (activeConfig.settings && activeConfig.settings.cascade) || [];
|
|
let resolved = null;
|
|
for (const entry of cascade) {
|
|
if (!entry.exists || !entry.path) continue;
|
|
let json = null;
|
|
try { json = parseJson(await readFile(entry.path, 'utf-8')); } catch { continue; }
|
|
if (json && typeof json.outputStyle === 'string' && json.outputStyle.trim()) {
|
|
resolved = { value: json.outputStyle.trim(), scope: entry.scope, path: entry.path };
|
|
}
|
|
}
|
|
return resolved;
|
|
}
|
|
|
|
/**
|
|
* Main scanner entry point.
|
|
* @param {string} targetPath - repo root to scan
|
|
* @param {object} _discovery - unused (OST reads the active config cascade itself)
|
|
*/
|
|
export async function scan(targetPath, _discovery) {
|
|
const start = Date.now();
|
|
const findings = [];
|
|
|
|
const activeConfig = await readActiveConfig(targetPath);
|
|
const styles = await withFrontmatter(activeConfig.outputStyles || []);
|
|
|
|
// CA-OST-001 — user/project custom style missing keep-coding-instructions:true.
|
|
for (const s of styles) {
|
|
if (s.source !== 'project' && s.source !== 'user') continue;
|
|
const kci = s.frontmatter ? s.frontmatter.keep_coding_instructions : undefined;
|
|
if (kci === true) continue;
|
|
findings.push(finding({
|
|
scanner: SCANNER,
|
|
code: 'strips-coding-instructions',
|
|
severity: SEVERITY.medium,
|
|
title: 'Custom output style removes built-in coding instructions',
|
|
description:
|
|
`The ${s.source} output style "${s.name}" does not set ` +
|
|
'`keep-coding-instructions: true`. While this style is active, Claude Code ' +
|
|
'drops its built-in software-engineering instructions — how to scope changes, ' +
|
|
'write comments, and verify work — and keeps only this style\'s text. The ' +
|
|
'frontmatter flag defaults to false, so the strip is easy to miss.',
|
|
file: s.path,
|
|
evidence:
|
|
`output_style="${s.name}"; source=${s.source}; ` +
|
|
`keep-coding-instructions=${kci === undefined ? 'unset (default false)' : String(kci)}`,
|
|
recommendation:
|
|
'To keep Claude Code\'s software-engineering behavior while applying this style, ' +
|
|
'add `keep-coding-instructions: true` to the frontmatter. If the strip is ' +
|
|
'intentional (a non-coding persona), no change is needed.',
|
|
category: 'output-styles',
|
|
}));
|
|
}
|
|
|
|
// CA-OST-002 — plugin output style with force-for-plugin:true (overrides user choice).
|
|
for (const s of styles) {
|
|
if (s.source !== 'plugin') continue;
|
|
const ffp = s.frontmatter ? s.frontmatter.force_for_plugin : undefined;
|
|
if (ffp !== true) continue;
|
|
findings.push(finding({
|
|
scanner: SCANNER,
|
|
code: 'plugin-forces-style',
|
|
severity: SEVERITY.low,
|
|
title: 'Plugin output style overrides your selected output style',
|
|
description:
|
|
`The plugin "${s.pluginName}" ships an output style "${s.name}" with ` +
|
|
'`force-for-plugin: true`, so Claude Code applies it automatically whenever the ' +
|
|
'plugin is enabled — overriding whatever `outputStyle` you selected. When more ' +
|
|
'than one enabled plugin does this, the first one loaded wins.',
|
|
file: s.path,
|
|
evidence: `output_style="${s.name}"; source=plugin:${s.pluginName}; force-for-plugin=true`,
|
|
recommendation:
|
|
'If you did not expect this style, disable the plugin or remove ' +
|
|
'`force-for-plugin: true` from its output style. This is awareness only — the ' +
|
|
'plugin is behaving as designed.',
|
|
category: 'output-styles',
|
|
}));
|
|
}
|
|
|
|
// CA-OST-003 — settings outputStyle resolving to a non-existent style (dead config).
|
|
const resolved = await resolveOutputStyleSetting(activeConfig);
|
|
if (resolved) {
|
|
const known = new Set([
|
|
...BUILTIN_STYLES,
|
|
...styles.map(s => String(s.name).toLowerCase()),
|
|
]);
|
|
if (!known.has(resolved.value.toLowerCase())) {
|
|
const customNames = styles.map(s => s.name);
|
|
findings.push(finding({
|
|
scanner: SCANNER,
|
|
code: 'style-not-found',
|
|
severity: SEVERITY.medium,
|
|
title: 'Configured output style does not exist',
|
|
description:
|
|
`Your ${resolved.scope} settings set \`outputStyle: "${resolved.value}"\`, but no ` +
|
|
'built-in or discovered custom style has that name. Claude Code falls back to the ' +
|
|
'default style, so the output behavior you configured is silently never applied.',
|
|
file: resolved.path,
|
|
evidence:
|
|
`outputStyle="${resolved.value}"; scope=${resolved.scope}; ` +
|
|
`builtins=[${BUILTIN_STYLES.join(', ')}]; ` +
|
|
`known_custom=[${customNames.join(', ')}]`,
|
|
recommendation:
|
|
'Fix the value to match an existing style (built-ins: Default, Explanatory, ' +
|
|
'Learning, Proactive), create the missing style under `.claude/output-styles/`, ' +
|
|
'or remove the `outputStyle` setting.',
|
|
category: 'output-styles',
|
|
}));
|
|
}
|
|
}
|
|
|
|
return scannerResult(SCANNER, 'ok', findings, styles.length, Date.now() - start);
|
|
}
|