feat(skill-listing): add CA-SKL-002 aggregate listing-budget check

Syklus 2 of Fase 4 Items 2+3. Flags when the sum of active skill
descriptions exceeds the listing budget (~2% of context, CC 2.1.32).

Design (operator-confirmed "fact-first, 200k anchor"):
- low severity (estimate) vs medium for the verified 1,536-char cap
- each description counted up to the 1,536 cap (what actually loads in
  the listing) — avoids double-counting the tail CA-SKL-001 flags
- fires when sum > 2% x 200k = 4000 tok; evidence leads with the measured
  sum + a calibration note that the budget scales 5x on 1M-context models
- aggregate emitted after the per-skill loop so the common case reads
  001=cap, 002=aggregate (finding IDs are a sequential counter, not stable
  semantic IDs — tests match on title, never NNN)

Also:
- tailored humanizer static entry for the aggregate title
- fix latent HOME leak in posture-grade-stability.test.mjs: it spawned
  posture.mjs without hermeticEnv(), so a real ~/.claude leaked HOME-scoped
  SKL/COL findings into the baseline grade (Token Efficiency A->B). Now
  isolated like the 8 other CLI-spawning tests.
- docs sync: test count 868->875, scanner-internals, gap-matrix, plan status

Suite 875/875, no snapshot drift, self-audit clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ter3E2JSi1Khgmuf2kady8
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 18:06:17 +02:00
commit 66433fee48
8 changed files with 215 additions and 11 deletions

View file

@ -742,6 +742,11 @@ export const TRANSLATIONS = {
description: 'Claude Code only shows the first part of each skill\'s description when it decides which skill to use. This one runs past the limit, so the end — often the trigger phrases — is silently dropped.',
recommendation: 'Shorten the description so the trigger words come first and it fits under the limit. For skills you do not use, turning them off frees up the listing entirely.',
},
'Aggregate skill descriptions may exceed the listing budget': {
title: 'All your skills together may be too much for the listing',
description: 'Claude Code keeps every active skill\'s description in one shared listing it reads to choose which skill to use, and that listing has a limited size. Added up, your skills\' descriptions run past that size on a smaller setup, so Claude Code may drop some of them — and stop seeing those skills. This is an estimate; a larger setup has more room.',
recommendation: 'Free up room: turn off bundled skills you do not use, collapse the heaviest ones so only their names show, or shorten the longest descriptions. The details show the measured total and the room available.',
},
},
patterns: [],
_default: {

View file

@ -9,14 +9,19 @@
* happens). A description past the cap loses its tail from what the model
* actually sees the very trigger phrases meant to route invocation.
*
* Detection (v1, high confidence no estimation):
* Detection:
* CA-SKL-001 active skill description > 1,536 chars truncated (medium)
* CA-SKL-002 sum of active descriptions exceeds the listing budget (low)
*
* The aggregate listing budget (skill descriptions sum vs ~2% of the context
* window, v2.1.32 L2860) is deliberately NOT scanned in v1: it requires
* assuming the user's context-window size, which would turn a verified fact
* into a guess. Lead with the cap; the aggregate check is a possible later
* addition (would carry a CALIBRATION_NOTE).
* The aggregate listing budget (changelog L2860, CC 2.1.32) allots the skill
* listing the model reads ~2% of the context window. We do NOT know the user's
* context window, so CA-SKL-002 anchors on a conservative 200k window and says
* so loudly: it leads with the measured sum (a fact) and carries a calibration
* note explaining the budget scales 5x on a 1M-context model. Severity is low
* (an estimate) versus medium for the verified per-description cap. Each
* description is counted only up to the 1,536-char cap, because that is all
* Claude Code loads into the listing the tail past the cap is dropped and is
* already flagged by CA-SKL-001 (so the aggregate does not double-count it).
*
* Two-lens note vs TOK pattern F: TOK pattern F flags *project-local* skills
* with descriptions > 500 chars as a structural per-turn "bloat" heuristic.
@ -29,7 +34,7 @@
import { finding, scannerResult } from './lib/output.mjs';
import { SEVERITY } from './lib/severity.mjs';
import { enumeratePlugins, enumerateSkills } from './lib/active-config-reader.mjs';
import { enumeratePlugins, enumerateSkills, estimateTokens } from './lib/active-config-reader.mjs';
import { readTextFile } from './lib/file-discovery.mjs';
import { parseFrontmatter } from './lib/yaml-parser.mjs';
@ -39,6 +44,26 @@ const SCANNER = 'SKL';
// Descriptions longer than this are truncated in the listing the model sees.
const DESCRIPTION_CAP = 1536;
// Aggregate listing budget (CC 2.1.32, changelog L2860): the skill listing the
// model reads is allotted ~2% of the context window. The context window is
// unknown, so we anchor on a conservative 200k window — the smallest common
// size, which fires earliest — and disclose the assumption in the evidence.
const BUDGET_FRACTION = 0.02;
const CONTEXT_WINDOW_ANCHOR = 200_000;
const AGGREGATE_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * CONTEXT_WINDOW_ANCHOR); // 4000
const LARGE_CONTEXT_WINDOW = 1_000_000;
const LARGE_CONTEXT_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * LARGE_CONTEXT_WINDOW); // 20000
// Dependency-free thousands separator (repo invariant: zero external deps).
const withCommas = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
// Appended to CA-SKL-002 evidence — the honest framing required because the
// budget depends on a context window we cannot observe (jf. TOK CALIBRATION_NOTE).
const BUDGET_CALIBRATION_NOTE =
'the budget scales with the context window - this anchors on a conservative 200k ' +
`window; at ${withCommas(LARGE_CONTEXT_WINDOW)} context the budget is ~${withCommas(LARGE_CONTEXT_BUDGET_TOKENS)} ` +
'tok and you are likely within it. this is an estimate, not measured telemetry';
/**
* Main scanner entry point.
*
@ -53,6 +78,7 @@ export async function scan(_targetPath, _discovery) {
const allSkills = await enumerateSkills(plugins);
let scanned = 0;
let aggregateChars = 0;
for (const skill of allSkills) {
if (!skill || typeof skill.path !== 'string') continue;
const content = await readTextFile(skill.path);
@ -60,6 +86,9 @@ export async function scan(_targetPath, _discovery) {
scanned++;
const fm = parseFrontmatter(content)?.frontmatter || null;
const desc = (fm && typeof fm.description === 'string') ? fm.description : '';
// Aggregate budget counts only what loads in the listing: each description
// up to the cap (the tail past the cap is dropped, and CA-SKL-001 flags it).
aggregateChars += Math.min(desc.length, DESCRIPTION_CAP);
if (desc.length <= DESCRIPTION_CAP) continue;
const sourceLabel = skill.source === 'plugin'
@ -90,5 +119,34 @@ export async function scan(_targetPath, _discovery) {
}));
}
// CA-SKL-002 (aggregate). Emitted after the per-skill findings so the common
// "one oversized skill + aggregate" case reads 001=cap, 002=aggregate.
const aggregateTokens = estimateTokens(aggregateChars, 'markdown');
if (aggregateTokens > AGGREGATE_BUDGET_TOKENS) {
findings.push(finding({
scanner: SCANNER,
severity: SEVERITY.low,
title: 'Aggregate skill descriptions may exceed the listing budget',
description:
`The ${scanned} active skills carry about ${aggregateTokens} tokens of description text ` +
`(each description counted up to the ${DESCRIPTION_CAP}-char listing cap), above the ` +
`${AGGREGATE_BUDGET_TOKENS}-token budget Claude Code allots the skill listing on a 200k ` +
'context window (about 2% of context, CC 2.1.32). When the listing overflows that budget ' +
'Claude Code drops descriptions, so the model may stop seeing some skills entirely. This ' +
'is an estimate — the budget scales with your actual context window (see evidence).',
evidence:
`active_skills_scanned=${scanned}; description_chars=${aggregateChars} (each capped at ` +
`${DESCRIPTION_CAP}); description_tokens~${aggregateTokens}; budget@200k=` +
`${AGGREGATE_BUDGET_TOKENS} tok (skill listing ~2% of context, CC 2.1.32); over_by~` +
`${aggregateTokens - AGGREGATE_BUDGET_TOKENS} tok - ${BUDGET_CALIBRATION_NOTE}`,
recommendation:
'Reclaim skill-listing budget: set `disableBundledSkills: true` to drop bundled skills you ' +
'do not use from the listing, use `skillOverrides` (`name-only` collapses a description, ' +
'`off` removes a skill) on the heaviest entries, and trim long descriptions toward their ' +
'trigger phrases.',
category: 'token-efficiency',
}));
}
return scannerResult(SCANNER, 'ok', findings, scanned, Date.now() - start);
}