config-audit/scanners/lib/skill-listing-budget.mjs
Kjell Tore Guttormsen 4027cdcf54 fix(scanners): retire the autoMode GAP dimension, a /doctor duplicate (D1)
CC 2.1.226's /doctor Check 8 covers auto mode with usage-weighted judgement.
The binding positioning forbids carrying a feature whose whole value is
duplicating a /doctor check, so the "adopt this feature" nudge goes. The
deterministic side stays: SET still validates autoMode structure and still
flags it as dead config in shared project settings. GAP dimensions 25 -> 24.

The title lived in FOUR tables, not the two the removal was scoped against:
the dimension list, scoring TITLE_TO_ID, the humanizer's static translations,
and the scoring denominators (TIER_COUNTS t3 8->7, TOTAL_DIMENSIONS 25->24,
MAX_WEIGHTED 42->41) -- the one that moves a user-visible number. findGapId
falls back to 'unknown' silently, so a partial removal would have degraded
without failing. A blanket sync invariant now asserts all four against
GAP_CHECKS instead of comparing occurrences pairwise; each arm was verified
red against its own defect (denominator drift, orphaned humanizer entry,
resurrected dimension).

Frozen tests/snapshots/v5.0.0/ stays untouched. strip-retired-gap.mjs is the
removal twin of strip-added-scanner.mjs: it strips the retired dimension from
whichever side still carries it and re-derives GAP IDs, since retiring a
dimension from mid-list shifts every later ID by one. Derived utilization
figures are dropped from comparison rather than recomputed -- recomputing them
in a test helper would assert the new arithmetic against itself, and
scoring.test.mjs already pins them exactly. Re-seeding was rejected: it would
silently bake in any other drift across every scanner those four files cover.

risk_score, risk_band, verdict, overallGrade, maturity and segment are
byte-identical across the change (severity info carries zero risk weight; GAP
is excluded from the overall grade). Utilization shifts 43 -> 44 on the fixture.

D2 (CA-SKL-002) is NOT removed. Verified against the primary source first: the
CC changelog carries exactly one budget-fraction statement (L3786, 2.1.32) and
nothing supersedes it, so our 2% is current and 002 is not a duplicate with a
stale figure. /doctor's ~1% could not be reconciled from the changelog and it
discloses its own numbers as disk estimates, so it is recorded, not adopted.
Left explicitly unverified in a code note: L3786 says "character budget" while
we express tokens -- a 4x difference nobody can settle from the wording.

Suite 1531 -> 1535, all green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RsfPGxgwbR3MY54wDC6hat
2026-08-09 22:43:04 +02:00

216 lines
10 KiB
JavaScript
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

/**
* Skill-listing budget — single source of truth.
*
* Claude Code shows the model a listing of every active skill's `description`
* so it can decide which skill to invoke. That listing is budgeted two ways:
* - per description: capped at 1,536 chars (CC 2.1.105, changelog L1502);
* anything past the cap is silently truncated.
* - in aggregate: the whole listing is allotted ~2% of the context window
* (CC 2.1.32, changelog L2860). We do NOT know the user's context window,
* so the aggregate budget anchors on a conservative 200k window (4,000 tok)
* and discloses the assumption.
*
* Two scanners consume this module so the budget is defined in exactly one place:
* - SKL (skill-listing-scanner) DIAGNOSES overflow (CA-SKL-001 per-description
* cap, CA-SKL-002 aggregate).
* - GAP (feature-gap-scanner) PRESCRIBES the remedy: when the listing is over
* budget and `disableBundledSkills` is un-pulled, it recommends that lever.
*
* Zero external dependencies.
*/
import { join } from 'node:path';
import { estimateTokens, enumeratePlugins, enumerateSkills } from './active-config-reader.mjs';
import { readTextFile } from './file-discovery.mjs';
import { parseFrontmatter, parseJson } from './yaml-parser.mjs';
import { CONTEXT_WINDOW_ANCHOR, LARGE_CONTEXT_WINDOW, withCommas } from './context-window.mjs';
// Verified per-description skill-listing cap (CC 2.1.105, changelog L1502).
// Descriptions longer than this are truncated in the listing the model sees.
export const DESCRIPTION_CAP = 1536;
// Aggregate listing budget (CC 2.1.32, changelog L2860): the skill listing the
// model reads is allotted ~2% of the context window. The context window is
// unknown, so we anchor on a conservative 200k window — the smallest common
// size, which fires earliest — and disclose the assumption in the evidence.
// The 200k/1M window constants live in context-window.mjs (single source of
// truth, shared with the CML CLAUDE.md char-budget check); re-exported here so
// existing importers of this module keep working.
// D2 re-verification (2026-08-09, CC 2.1.226). CC 2.1.226's /doctor reports a
// combined skill+command+agent listing and puts the budget near ~1% (~10,000
// tok on a 1M window) — half of ours. Checked against the primary source
// before touching the number: the CC changelog contains EXACTLY ONE
// budget-fraction statement (L3786, under 2.1.32) and no later entry
// supersedes it, so 2% stands and CA-SKL-002 is NOT a /doctor duplicate
// carrying a stale figure. /doctor's arithmetic could not be reconciled from
// the changelog, and /doctor discloses its own numbers as disk estimates
// (chars÷4), so its ~1% is recorded, not adopted.
//
// NOT VERIFIED, deliberately left alone: L3786 says "skill CHARACTER budget
// now scales with context window (2% of context)". We express the budget in
// TOKENS (0.02 × 200k = 4000 tok). Whether CC's budget is 2% counted in
// characters or in tokens is not resolvable from the changelog wording, and no
// primary source settles it — a 4× difference rides on the answer. Changing
// the constant on that ambiguity would be a guess; it stays until a primary
// source decides it.
export const BUDGET_FRACTION = 0.02;
export const AGGREGATE_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * CONTEXT_WINDOW_ANCHOR); // 4000
export const LARGE_CONTEXT_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * LARGE_CONTEXT_WINDOW); // 20000
export { CONTEXT_WINDOW_ANCHOR, LARGE_CONTEXT_WINDOW, withCommas };
// The honest framing required because the budget depends on a context window we
// cannot observe (jf. TOK CALIBRATION_NOTE). Appended to budget-overflow evidence.
export const BUDGET_CALIBRATION_NOTE =
'the budget scales with the context window - this anchors on a conservative 200k ' +
`window; at ${withCommas(LARGE_CONTEXT_WINDOW)} context the budget is ~${withCommas(LARGE_CONTEXT_BUDGET_TOKENS)} ` +
'tok and you are likely within it. this is an estimate, not measured telemetry';
// Skill-body size guidance (CA-SKL-003). A SKILL.md body over ~5,000 tokens
// (~500 lines / ~20k chars) should split reference content into supporting files
// (Claude Code skill-authoring guidance). Unlike the listing budget above, the
// body is an ON-DEMAND cost: it loads only when the skill is invoked, not every
// turn — so this is a LOW-severity efficiency signal, not an always-loaded bill.
export const BODY_TOKEN_THRESHOLD = 5000;
// Honest framing for the body-size finding: distinguishes on-demand from
// always-loaded cost and flags the figure as an estimate. Appended to evidence.
export const BODY_CALIBRATION_NOTE =
'this is the skill BODY (SKILL.md below the frontmatter), which loads ON DEMAND only when the ' +
'skill is invoked - NOT every turn like the always-loaded listing. estimate (chars/4), not measured telemetry';
/**
* @typedef {object} BudgetAssessment
* @property {number} scanned - number of descriptions assessed
* @property {number} aggregateChars - sum of each length capped at DESCRIPTION_CAP
* @property {number} aggregateTokens - estimateTokens(aggregateChars, 'markdown')
* @property {number} budgetTokens - AGGREGATE_BUDGET_TOKENS (the 200k-anchored budget)
* @property {boolean} overBudget - aggregateTokens strictly greater than budgetTokens
* @property {number} overBy - tokens over budget (0 when not over)
*/
/**
* Pure aggregate-budget assessment. Each description contributes only up to the
* cap (the tail past the cap is dropped from the listing, and CA-SKL-001 already
* flags it — so the aggregate does not double-count it).
*
* @param {number[]} descLengths - one entry per active skill (description char count)
* @param {number} [budgetTokens=AGGREGATE_BUDGET_TOKENS] - the listing budget to
* measure against. Defaults to the 200k-anchored 4,000 tok; B8 passes a
* window-calibrated budget. Defaulting keeps existing callers byte-stable.
* @returns {BudgetAssessment}
*/
export function assessSkillListingBudget(descLengths, budgetTokens = AGGREGATE_BUDGET_TOKENS) {
let aggregateChars = 0;
for (const len of descLengths) {
const safe = (typeof len === 'number' && Number.isFinite(len) && len > 0) ? len : 0;
aggregateChars += Math.min(safe, DESCRIPTION_CAP);
}
const aggregateTokens = estimateTokens(aggregateChars, 'markdown');
const overBudget = aggregateTokens > budgetTokens;
return {
scanned: descLengths.length,
aggregateChars,
aggregateTokens,
budgetTokens,
overBudget,
overBy: overBudget ? aggregateTokens - budgetTokens : 0,
};
}
/**
* @typedef {object} ActiveSkillEntry
* @property {string} name
* @property {'user'|'plugin'} source
* @property {string|null} pluginName
* @property {string} path
* @property {number} descLength
* @property {number} bodyChars - SKILL.md body length below the frontmatter (on-demand cost)
* @property {number} bodyLines - body line count
* @property {number} bodyTokens - estimateTokens(bodyChars, 'markdown')
*/
/**
* Enumerate every active skill (user + plugin) and measure the listing budget.
* HOME-scoped: resolves ~/.claude via process.env.HOME (enumeratePlugins /
* enumerateSkills). Callers that run under test MUST override HOME (see the
* hermetic-home helper / runScannerWithHome pattern).
*
* @param {number} [budgetTokens=AGGREGATE_BUDGET_TOKENS] - listing budget for the
* aggregate assessment (B8 window-calibration); defaults keep callers byte-stable.
* @returns {Promise<{ skills: ActiveSkillEntry[], aggregate: BudgetAssessment }>}
*/
export async function measureActiveSkillListing(budgetTokens = AGGREGATE_BUDGET_TOKENS) {
const plugins = await enumeratePlugins();
const allSkills = await enumerateSkills(plugins);
const skills = [];
for (const skill of allSkills) {
if (!skill || typeof skill.path !== 'string') continue;
const content = await readTextFile(skill.path);
if (!content) continue;
const parsed = parseFrontmatter(content);
const fm = parsed?.frontmatter || null;
const desc = (fm && typeof fm.description === 'string') ? fm.description : '';
const body = (parsed && typeof parsed.body === 'string') ? parsed.body : '';
const bodyChars = body.length;
skills.push({
name: skill.name,
source: skill.source,
pluginName: skill.pluginName,
path: skill.path,
descLength: desc.length,
bodyChars,
bodyLines: bodyChars === 0 ? 0 : body.split('\n').length,
bodyTokens: estimateTokens(bodyChars, 'markdown'),
});
}
const aggregate = assessSkillListingBudget(skills.map((s) => s.descLength), budgetTokens);
return { skills, aggregate };
}
/**
* Read an env flag, treating null, "", "0", "false", "no", "off" as un-set.
* @param {string|undefined} v
* @returns {boolean}
*/
export function envFlag(v) {
if (v == null) return false;
const s = String(v).trim().toLowerCase();
return s !== '' && s !== '0' && s !== 'false' && s !== 'no' && s !== 'off';
}
/**
* Resolve whether the `disableBundledSkills` lever is effectively ON, reading the
* env var and the settings cascade directly (user ~/.claude, then project, then
* project-local).
*
* Reads the files directly rather than relying on config-discovery
* classification: when discovery walks ~/.claude from the .claude root, the
* user settings.json has a relPath of "settings.json" (no ".claude" segment)
* and is NOT tagged as settings-json — so the dominant user-scope location for
* this global preference would otherwise be missed. HOME-scoped via
* process.env.HOME.
*
* @param {string} [projectPath] - project root, to also read project + local settings
* @returns {Promise<boolean>}
*/
export async function isBundledSkillsDisabled(projectPath) {
if (envFlag(process.env.CLAUDE_CODE_DISABLE_BUNDLED_SKILLS)) return true;
const home = process.env.HOME || process.env.USERPROFILE || '';
const candidates = [];
if (home) candidates.push(join(home, '.claude', 'settings.json'));
if (projectPath) {
candidates.push(join(projectPath, '.claude', 'settings.json'));
candidates.push(join(projectPath, '.claude', 'settings.local.json'));
}
for (const p of candidates) {
const content = await readTextFile(p);
if (!content) continue;
const parsed = parseJson(content);
if (parsed && parsed.disableBundledSkills === true) return true;
}
return false;
}