CC 2.1.226's /doctor Check 8 covers auto mode with usage-weighted judgement. The binding positioning forbids carrying a feature whose whole value is duplicating a /doctor check, so the "adopt this feature" nudge goes. The deterministic side stays: SET still validates autoMode structure and still flags it as dead config in shared project settings. GAP dimensions 25 -> 24. The title lived in FOUR tables, not the two the removal was scoped against: the dimension list, scoring TITLE_TO_ID, the humanizer's static translations, and the scoring denominators (TIER_COUNTS t3 8->7, TOTAL_DIMENSIONS 25->24, MAX_WEIGHTED 42->41) -- the one that moves a user-visible number. findGapId falls back to 'unknown' silently, so a partial removal would have degraded without failing. A blanket sync invariant now asserts all four against GAP_CHECKS instead of comparing occurrences pairwise; each arm was verified red against its own defect (denominator drift, orphaned humanizer entry, resurrected dimension). Frozen tests/snapshots/v5.0.0/ stays untouched. strip-retired-gap.mjs is the removal twin of strip-added-scanner.mjs: it strips the retired dimension from whichever side still carries it and re-derives GAP IDs, since retiring a dimension from mid-list shifts every later ID by one. Derived utilization figures are dropped from comparison rather than recomputed -- recomputing them in a test helper would assert the new arithmetic against itself, and scoring.test.mjs already pins them exactly. Re-seeding was rejected: it would silently bake in any other drift across every scanner those four files cover. risk_score, risk_band, verdict, overallGrade, maturity and segment are byte-identical across the change (severity info carries zero risk weight; GAP is excluded from the overall grade). Utilization shifts 43 -> 44 on the fixture. D2 (CA-SKL-002) is NOT removed. Verified against the primary source first: the CC changelog carries exactly one budget-fraction statement (L3786, 2.1.32) and nothing supersedes it, so our 2% is current and 002 is not a duplicate with a stale figure. /doctor's ~1% could not be reconciled from the changelog and it discloses its own numbers as disk estimates, so it is recorded, not adopted. Left explicitly unverified in a code note: L3786 says "character budget" while we express tokens -- a 4x difference nobody can settle from the wording. Suite 1531 -> 1535, all green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RsfPGxgwbR3MY54wDC6hat
216 lines
10 KiB
JavaScript
216 lines
10 KiB
JavaScript
/**
|
||
* Skill-listing budget — single source of truth.
|
||
*
|
||
* Claude Code shows the model a listing of every active skill's `description`
|
||
* so it can decide which skill to invoke. That listing is budgeted two ways:
|
||
* - per description: capped at 1,536 chars (CC 2.1.105, changelog L1502);
|
||
* anything past the cap is silently truncated.
|
||
* - in aggregate: the whole listing is allotted ~2% of the context window
|
||
* (CC 2.1.32, changelog L2860). We do NOT know the user's context window,
|
||
* so the aggregate budget anchors on a conservative 200k window (4,000 tok)
|
||
* and discloses the assumption.
|
||
*
|
||
* Two scanners consume this module so the budget is defined in exactly one place:
|
||
* - SKL (skill-listing-scanner) DIAGNOSES overflow (CA-SKL-001 per-description
|
||
* cap, CA-SKL-002 aggregate).
|
||
* - GAP (feature-gap-scanner) PRESCRIBES the remedy: when the listing is over
|
||
* budget and `disableBundledSkills` is un-pulled, it recommends that lever.
|
||
*
|
||
* Zero external dependencies.
|
||
*/
|
||
|
||
import { join } from 'node:path';
|
||
import { estimateTokens, enumeratePlugins, enumerateSkills } from './active-config-reader.mjs';
|
||
import { readTextFile } from './file-discovery.mjs';
|
||
import { parseFrontmatter, parseJson } from './yaml-parser.mjs';
|
||
import { CONTEXT_WINDOW_ANCHOR, LARGE_CONTEXT_WINDOW, withCommas } from './context-window.mjs';
|
||
|
||
// Verified per-description skill-listing cap (CC 2.1.105, changelog L1502).
|
||
// Descriptions longer than this are truncated in the listing the model sees.
|
||
export const DESCRIPTION_CAP = 1536;
|
||
|
||
// Aggregate listing budget (CC 2.1.32, changelog L2860): the skill listing the
|
||
// model reads is allotted ~2% of the context window. The context window is
|
||
// unknown, so we anchor on a conservative 200k window — the smallest common
|
||
// size, which fires earliest — and disclose the assumption in the evidence.
|
||
// The 200k/1M window constants live in context-window.mjs (single source of
|
||
// truth, shared with the CML CLAUDE.md char-budget check); re-exported here so
|
||
// existing importers of this module keep working.
|
||
// D2 re-verification (2026-08-09, CC 2.1.226). CC 2.1.226's /doctor reports a
|
||
// combined skill+command+agent listing and puts the budget near ~1% (~10,000
|
||
// tok on a 1M window) — half of ours. Checked against the primary source
|
||
// before touching the number: the CC changelog contains EXACTLY ONE
|
||
// budget-fraction statement (L3786, under 2.1.32) and no later entry
|
||
// supersedes it, so 2% stands and CA-SKL-002 is NOT a /doctor duplicate
|
||
// carrying a stale figure. /doctor's arithmetic could not be reconciled from
|
||
// the changelog, and /doctor discloses its own numbers as disk estimates
|
||
// (chars÷4), so its ~1% is recorded, not adopted.
|
||
//
|
||
// NOT VERIFIED, deliberately left alone: L3786 says "skill CHARACTER budget
|
||
// now scales with context window (2% of context)". We express the budget in
|
||
// TOKENS (0.02 × 200k = 4000 tok). Whether CC's budget is 2% counted in
|
||
// characters or in tokens is not resolvable from the changelog wording, and no
|
||
// primary source settles it — a 4× difference rides on the answer. Changing
|
||
// the constant on that ambiguity would be a guess; it stays until a primary
|
||
// source decides it.
|
||
export const BUDGET_FRACTION = 0.02;
|
||
export const AGGREGATE_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * CONTEXT_WINDOW_ANCHOR); // 4000
|
||
export const LARGE_CONTEXT_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * LARGE_CONTEXT_WINDOW); // 20000
|
||
export { CONTEXT_WINDOW_ANCHOR, LARGE_CONTEXT_WINDOW, withCommas };
|
||
|
||
// The honest framing required because the budget depends on a context window we
|
||
// cannot observe (jf. TOK CALIBRATION_NOTE). Appended to budget-overflow evidence.
|
||
export const BUDGET_CALIBRATION_NOTE =
|
||
'the budget scales with the context window - this anchors on a conservative 200k ' +
|
||
`window; at ${withCommas(LARGE_CONTEXT_WINDOW)} context the budget is ~${withCommas(LARGE_CONTEXT_BUDGET_TOKENS)} ` +
|
||
'tok and you are likely within it. this is an estimate, not measured telemetry';
|
||
|
||
// Skill-body size guidance (CA-SKL-003). A SKILL.md body over ~5,000 tokens
|
||
// (~500 lines / ~20k chars) should split reference content into supporting files
|
||
// (Claude Code skill-authoring guidance). Unlike the listing budget above, the
|
||
// body is an ON-DEMAND cost: it loads only when the skill is invoked, not every
|
||
// turn — so this is a LOW-severity efficiency signal, not an always-loaded bill.
|
||
export const BODY_TOKEN_THRESHOLD = 5000;
|
||
|
||
// Honest framing for the body-size finding: distinguishes on-demand from
|
||
// always-loaded cost and flags the figure as an estimate. Appended to evidence.
|
||
export const BODY_CALIBRATION_NOTE =
|
||
'this is the skill BODY (SKILL.md below the frontmatter), which loads ON DEMAND only when the ' +
|
||
'skill is invoked - NOT every turn like the always-loaded listing. estimate (chars/4), not measured telemetry';
|
||
|
||
/**
|
||
* @typedef {object} BudgetAssessment
|
||
* @property {number} scanned - number of descriptions assessed
|
||
* @property {number} aggregateChars - sum of each length capped at DESCRIPTION_CAP
|
||
* @property {number} aggregateTokens - estimateTokens(aggregateChars, 'markdown')
|
||
* @property {number} budgetTokens - AGGREGATE_BUDGET_TOKENS (the 200k-anchored budget)
|
||
* @property {boolean} overBudget - aggregateTokens strictly greater than budgetTokens
|
||
* @property {number} overBy - tokens over budget (0 when not over)
|
||
*/
|
||
|
||
/**
|
||
* Pure aggregate-budget assessment. Each description contributes only up to the
|
||
* cap (the tail past the cap is dropped from the listing, and CA-SKL-001 already
|
||
* flags it — so the aggregate does not double-count it).
|
||
*
|
||
* @param {number[]} descLengths - one entry per active skill (description char count)
|
||
* @param {number} [budgetTokens=AGGREGATE_BUDGET_TOKENS] - the listing budget to
|
||
* measure against. Defaults to the 200k-anchored 4,000 tok; B8 passes a
|
||
* window-calibrated budget. Defaulting keeps existing callers byte-stable.
|
||
* @returns {BudgetAssessment}
|
||
*/
|
||
export function assessSkillListingBudget(descLengths, budgetTokens = AGGREGATE_BUDGET_TOKENS) {
|
||
let aggregateChars = 0;
|
||
for (const len of descLengths) {
|
||
const safe = (typeof len === 'number' && Number.isFinite(len) && len > 0) ? len : 0;
|
||
aggregateChars += Math.min(safe, DESCRIPTION_CAP);
|
||
}
|
||
const aggregateTokens = estimateTokens(aggregateChars, 'markdown');
|
||
const overBudget = aggregateTokens > budgetTokens;
|
||
return {
|
||
scanned: descLengths.length,
|
||
aggregateChars,
|
||
aggregateTokens,
|
||
budgetTokens,
|
||
overBudget,
|
||
overBy: overBudget ? aggregateTokens - budgetTokens : 0,
|
||
};
|
||
}
|
||
|
||
/**
|
||
* @typedef {object} ActiveSkillEntry
|
||
* @property {string} name
|
||
* @property {'user'|'plugin'} source
|
||
* @property {string|null} pluginName
|
||
* @property {string} path
|
||
* @property {number} descLength
|
||
* @property {number} bodyChars - SKILL.md body length below the frontmatter (on-demand cost)
|
||
* @property {number} bodyLines - body line count
|
||
* @property {number} bodyTokens - estimateTokens(bodyChars, 'markdown')
|
||
*/
|
||
|
||
/**
|
||
* Enumerate every active skill (user + plugin) and measure the listing budget.
|
||
* HOME-scoped: resolves ~/.claude via process.env.HOME (enumeratePlugins /
|
||
* enumerateSkills). Callers that run under test MUST override HOME (see the
|
||
* hermetic-home helper / runScannerWithHome pattern).
|
||
*
|
||
* @param {number} [budgetTokens=AGGREGATE_BUDGET_TOKENS] - listing budget for the
|
||
* aggregate assessment (B8 window-calibration); defaults keep callers byte-stable.
|
||
* @returns {Promise<{ skills: ActiveSkillEntry[], aggregate: BudgetAssessment }>}
|
||
*/
|
||
export async function measureActiveSkillListing(budgetTokens = AGGREGATE_BUDGET_TOKENS) {
|
||
const plugins = await enumeratePlugins();
|
||
const allSkills = await enumerateSkills(plugins);
|
||
|
||
const skills = [];
|
||
for (const skill of allSkills) {
|
||
if (!skill || typeof skill.path !== 'string') continue;
|
||
const content = await readTextFile(skill.path);
|
||
if (!content) continue;
|
||
const parsed = parseFrontmatter(content);
|
||
const fm = parsed?.frontmatter || null;
|
||
const desc = (fm && typeof fm.description === 'string') ? fm.description : '';
|
||
const body = (parsed && typeof parsed.body === 'string') ? parsed.body : '';
|
||
const bodyChars = body.length;
|
||
skills.push({
|
||
name: skill.name,
|
||
source: skill.source,
|
||
pluginName: skill.pluginName,
|
||
path: skill.path,
|
||
descLength: desc.length,
|
||
bodyChars,
|
||
bodyLines: bodyChars === 0 ? 0 : body.split('\n').length,
|
||
bodyTokens: estimateTokens(bodyChars, 'markdown'),
|
||
});
|
||
}
|
||
|
||
const aggregate = assessSkillListingBudget(skills.map((s) => s.descLength), budgetTokens);
|
||
return { skills, aggregate };
|
||
}
|
||
|
||
/**
|
||
* Read an env flag, treating null, "", "0", "false", "no", "off" as un-set.
|
||
* @param {string|undefined} v
|
||
* @returns {boolean}
|
||
*/
|
||
export function envFlag(v) {
|
||
if (v == null) return false;
|
||
const s = String(v).trim().toLowerCase();
|
||
return s !== '' && s !== '0' && s !== 'false' && s !== 'no' && s !== 'off';
|
||
}
|
||
|
||
/**
|
||
* Resolve whether the `disableBundledSkills` lever is effectively ON, reading the
|
||
* env var and the settings cascade directly (user ~/.claude, then project, then
|
||
* project-local).
|
||
*
|
||
* Reads the files directly rather than relying on config-discovery
|
||
* classification: when discovery walks ~/.claude from the .claude root, the
|
||
* user settings.json has a relPath of "settings.json" (no ".claude" segment)
|
||
* and is NOT tagged as settings-json — so the dominant user-scope location for
|
||
* this global preference would otherwise be missed. HOME-scoped via
|
||
* process.env.HOME.
|
||
*
|
||
* @param {string} [projectPath] - project root, to also read project + local settings
|
||
* @returns {Promise<boolean>}
|
||
*/
|
||
export async function isBundledSkillsDisabled(projectPath) {
|
||
if (envFlag(process.env.CLAUDE_CODE_DISABLE_BUNDLED_SKILLS)) return true;
|
||
|
||
const home = process.env.HOME || process.env.USERPROFILE || '';
|
||
const candidates = [];
|
||
if (home) candidates.push(join(home, '.claude', 'settings.json'));
|
||
if (projectPath) {
|
||
candidates.push(join(projectPath, '.claude', 'settings.json'));
|
||
candidates.push(join(projectPath, '.claude', 'settings.local.json'));
|
||
}
|
||
|
||
for (const p of candidates) {
|
||
const content = await readTextFile(p);
|
||
if (!content) continue;
|
||
const parsed = parseJson(content);
|
||
if (parsed && parsed.disableBundledSkills === true) return true;
|
||
}
|
||
return false;
|
||
}
|