config-audit/tests/helpers/strip-retired-gap.mjs
Kjell Tore Guttormsen 7a794b47eb fix(scanners)!: a finding ID names the check, not the emission (M-BUG-28)
BREAKING CHANGE: the {NNN} in CA-{SCANNER}-{NNN} identifies the check that
produced the finding. It used to be the finding's position in that scanner's
output for that run, which made it unstable across CONFIGURATIONS, not just
across releases as STATE framed it. Measured on two fixtures: "No custom
subagents" was CA-GAP-007 on minimal-project and CA-GAP-004 on healthy-project.
A user who fixed an unrelated earlier gap silently renumbered every later one,
so a .config-audit-ignore pin retargeted to a neighbouring finding with no
version change at all.

Second measured arm: README already documented the opposite scheme. It and the
scanner headers describe ~20 numbers as check codes (CA-SKL-003 = oversized
body, CA-PLH-015 = folder shadowing, CA-TOK-006 = schema deferral), and the
counter could only produce those in the all-fire case -- source-order positions
are 4, 3 and 8. The documentation described the scheme; the implementation was
what was wrong. Every published number is preserved by construction and pinned
exhaustively in tests/lib/finding-codes.test.mjs.

scanners/lib/finding-codes.mjs is the single authority. Every finding() call
passes a `code`; an undeclared or missing one THROWS. No counter fallback --
that would reproduce D1's findGapId -> 'unknown' silent degradation and let a
half-converted scanner ship IDs that look valid. findingCounter/resetCounter
are deleted outright, not left as no-ops. Retirement is now a mechanism:
RETIRED_CODES tombstones a withdrawn key so its number is never reissued,
seeded with GAP t3_8 -- the D1 removal that opened this chunk.

IDs are consequently NOT unique per finding: one check failing in three files
emits three findings sharing an ID. That inverts which consumer is correct, so
every f.id/findingId site was classified before the change. diff-engine and
most of fix-engine already keyed on scanner+title+file (drift was never lying);
fix-engine's verification did not, and keyed on the ID alone -- fixing one of
two sibling instances marked both fixed, and the untouched one, still present
in the re-scan, was reported as a REGRESSION. Red test first, then keyed on
(findingId, file), which both planFixes and applyFixes already carry.
plugin-health's crossIds Set was measured and is a clean negative: cross
findings are allFindings.slice(crossPluginStart) and codes 18/19 are emitted
only in that tail, so the partition holds by construction.

unknownSuppressions() reports a pin that names no declared check, in the
--output-file payload (ux-rules rule 2 -- a stderr-only warning is invisible to
the commands) and only when one exists, so a clean config is byte-identical.
That is what makes the break safe: a stale pin goes loud instead of dying quiet.

Frozen tests/snapshots/v5.0.0/ untouched on disk. IDs are masked out of that
comparison (mask-finding-ids.mjs) rather than re-derived -- re-deriving
positional IDs would assert the retired scheme against itself, and #58's
isGapEntry off-by-one is the measured example of that misfiring. The dead
re-derivation is removed from strip-retired-gap.mjs. default-output snapshots
re-approved after confirming the diff is IDs and nothing else.

Guards, each seen red against its own defect: a missing code (scanner errors
out mid-sweep), an orphan declaration, a resurrected retired key, and a
documented ID naming no check. The sweep asserts the union across all 16
scanners, never per scanner -- a per-scanner assertion goes green on a partial
conversion.

Fasit written before implementation: docs/mbug28-id-semantics-fasit.local.md,
including one correction made before running (CML has 12 checks over 13 call
sites -- the anchored and calibrated char-budget arms are one check, which a
repeated-title sweep found and my call-site count had missed).

Suite 1535 -> 1573, 0 failing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MyqCQKK2ornJ1jFWwqx17E
2026-08-09 23:26:36 +02:00

198 lines
7.5 KiB
JavaScript

/**
* The REMOVAL twin of strip-added-scanner.mjs.
*
* The frozen v5.0.0 baselines predate v5.14's retirement of the GAP dimension
* `No autoMode classifier` (a pure /doctor Check 8 duplicate under the binding
* positioning). Where strip-added-scanner.mjs removes post-v5.0.0 ADDITIONS
* from live output so it matches the frozen expectation, this removes a
* RETIRED dimension from whichever side still carries it — the frozen
* expectation. Both sides run through the same normalizer, so every operation
* here must be a no-op on live output; that constraint is what dictates the
* three shapes below.
*
* Why strip rather than re-seed: re-seeding these four baselines would bake in
* whatever OTHER drift has accumulated since v5.0.0 across every scanner they
* cover, silently. Stripping keeps the strongest invariant — every byte the
* original scanners emit is still compared — and confines the exception to the
* one dimension we deliberately deleted.
*
* Three payload shapes carry GAP findings, all derived from one scan and all
* therefore in emission order:
* - envelopes (scan-orchestrator, posture) — `scanners[GAP].findings`
* - drift — flat `newFindings`/`resolvedFindings`/`unchangedFindings`/`movedFindings`
* - fix-cli — `manual[]` entries keyed `findingId`
* A single traversal handles all three rather than three special cases.
*
* What is adjusted, and what is deliberately NOT:
* - the retired finding is dropped. IDs are NOT re-derived: this helper used
* to renumber the survivors positionally, which was the scanner's own rule
* at the time. Since M-BUG-28 an ID names the check, so renumbering here
* would assert a retired scheme against itself — the frozen comparison masks
* IDs instead (`mask-finding-ids.mjs`), and the check→number pairs are
* pinned exhaustively in tests/lib/finding-codes.test.mjs.
* - counter bookkeeping is decremented mechanically, exactly as
* strip-added-scanner.mjs decrements `scanners_ok`.
* - the DERIVED utilization figures (`utilization.score`/`overhang` and the
* feature_coverage area `score`) are dropped from comparison instead of
* recomputed. Recomputing them inside a test helper would assert the new
* arithmetic against itself. They are covered exhaustively by dedicated
* unit tests in tests/lib/scoring.test.mjs (exact expected values per
* tier), which is the right place for arithmetic.
* - risk_score, risk_band, verdict, overallGrade, maturity, segment and
* drift's scoreChange are left alone: the retired dimension was
* severity `info`, which carries zero risk weight, and GAP is excluded from
* the overall grade (scoring.mjs). Measured 2026-08-09: all byte-identical
* across the change.
*
* Mutates in place and returns the payload for chaining inside a normalizer.
*/
const RETIRED_GAP_TITLES = new Set(['No autoMode classifier']);
const GAP_ID_RE = /^CA-GAP-\d+$/;
/** Decrement a numeric field by n when it exists. */
function decrement(obj, key, n) {
if (obj && typeof obj[key] === 'number') obj[key] -= n;
}
/**
* True for a FINDING-shaped object belonging to the GAP scanner.
*
* The `title` requirement is load-bearing: the scanner envelope entry also
* carries `scanner: 'GAP'`, and counting it as a finding shifts every
* re-derived ID by one.
*/
function isGapEntry(o) {
if (!o || typeof o !== 'object' || typeof o.title !== 'string') return false;
if (o.scanner === 'GAP') return true;
return GAP_ID_RE.test(o.id || '') || GAP_ID_RE.test(o.findingId || '');
}
/**
* Walk the payload, removing retired GAP entries from every array and
* collecting the survivors in traversal (= emission) order.
*/
function walk(node, state) {
if (Array.isArray(node)) {
for (let i = node.length - 1; i >= 0; i--) {
const item = node[i];
if (isGapEntry(item) && RETIRED_GAP_TITLES.has(item.title)) {
state.removed++;
if (item.severity) {
state.bySeverity[item.severity] = (state.bySeverity[item.severity] || 0) + 1;
}
node.splice(i, 1);
}
}
for (const item of node) {
if (isGapEntry(item)) state.survivors.push(item);
walk(item, state);
}
return;
}
if (node && typeof node === 'object') {
for (const v of Object.values(node)) walk(v, state);
}
}
/** Decrement every counter location a retired GAP finding contributes to. */
function adjustCounters(payload, state) {
const { removed, bySeverity } = state;
const envelopes = [payload, payload.scannerEnvelope].filter(
e => e && typeof e === 'object',
);
for (const env of envelopes) {
if (Array.isArray(env.scanners)) {
for (const s of env.scanners) {
if (s && s.scanner === 'GAP') {
for (const [sev, n] of Object.entries(bySeverity)) decrement(s.counts, sev, n);
}
}
}
if (env.aggregate) {
decrement(env.aggregate, 'total_findings', removed);
for (const [sev, n] of Object.entries(bySeverity)) decrement(env.aggregate.counts, sev, n);
}
}
// Posture-shaped roll-ups above the envelope.
decrement(payload, 'opportunityCount', removed);
if (Array.isArray(payload.areas)) {
for (const a of payload.areas) {
if (a && a.id === 'feature_coverage') decrement(a, 'findingCount', removed);
}
}
// Drift totals.
if (payload.summary) {
decrement(payload.summary, 'totalBefore', removed);
decrement(payload.summary, 'totalAfter', removed);
}
}
/**
* Strip retired GAP dimensions from any CLI payload and drop the derived
* utilization figures from comparison.
* @template T
* @param {T} payload
* @returns {T}
*/
export function stripRetiredGap(payload) {
if (!payload || typeof payload !== 'object') return payload;
const state = { removed: 0, bySeverity: {}, survivors: [] };
walk(payload, state);
if (state.removed > 0) {
adjustCounters(payload, state);
}
// Derived scores: dropped on BOTH sides unconditionally, so live and frozen
// stay symmetric. See the header note.
if (payload.utilization && typeof payload.utilization === 'object') {
delete payload.utilization.score;
delete payload.utilization.overhang;
}
if (Array.isArray(payload.areas)) {
for (const a of payload.areas) {
if (a && a.id === 'feature_coverage') delete a.score;
}
}
// Drift reports the same derived score under a different key shape.
if (Array.isArray(payload.areaChanges)) {
for (const a of payload.areaChanges) {
if (a && a.name === 'Feature Coverage') {
delete a.before?.score;
delete a.after?.score;
}
}
}
return payload;
}
/**
* Mask the GAP finding tally in a captured stderr scorecard.
*
* The scorecard prints only a count, with nothing in the text identifying WHICH
* dimensions it covers — so unlike the line-removal in stripAddedScannerStderr,
* a decrement here could not be a no-op on the live side. Both sides go through
* the same normalizer, so the tally is masked rather than adjusted; the exact
* GAP count is asserted on the structured payload, where the findings
* themselves are compared.
* @param {string} text
* @returns {string}
*/
export function maskGapTallyStderr(text) {
if (typeof text !== 'string') return text;
return text
.replace(
/^(\s*`?\[GAP\][^\n]*?)\d+ finding\(s\)/gm,
(_m, head) => `${head}<GAPCOUNT> finding(s)`,
)
// The scorecard footer repeats the tally, in a raw and a humanized wording.
.replace(/^(\s*)\d+( opportunities available)/gm, '$1<GAPCOUNT>$2')
.replace(/^(\s*)\d+( ways you could get more out of)/gm, '$1<GAPCOUNT>$2');
}