open/config-audit
1
0
Fork 0
Code Issues Releases 1 Activity

feat(skill-listing): add CA-SKL-002 aggregate listing-budget check

Browse source 
Syklus 2 of Fase 4 Items 2+3. Flags when the sum of active skill
descriptions exceeds the listing budget (~2% of context, CC 2.1.32).

Design (operator-confirmed "fact-first, 200k anchor"):
- low severity (estimate) vs medium for the verified 1,536-char cap
- each description counted up to the 1,536 cap (what actually loads in
  the listing) — avoids double-counting the tail CA-SKL-001 flags
- fires when sum > 2% x 200k = 4000 tok; evidence leads with the measured
  sum + a calibration note that the budget scales 5x on 1M-context models
- aggregate emitted after the per-skill loop so the common case reads
  001=cap, 002=aggregate (finding IDs are a sequential counter, not stable
  semantic IDs — tests match on title, never NNN)

Also:
- tailored humanizer static entry for the aggregate title
- fix latent HOME leak in posture-grade-stability.test.mjs: it spawned
  posture.mjs without hermeticEnv(), so a real ~/.claude leaked HOME-scoped
  SKL/COL findings into the baseline grade (Token Efficiency A->B). Now
  isolated like the 8 other CLI-spawning tests.
- docs sync: test count 868->875, scanner-internals, gap-matrix, plan status

Suite 875/875, no snapshot drift, self-audit clean.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Ter3E2JSi1Khgmuf2kady8
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 18:06:17 +02:00
commit 66433fee48
8 changed files with 215 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

72
scanners/skill-listing-scanner.mjs
View file

@ -9,14 +9,19 @@
* happens). A description past the cap loses its tail from what the model
* actually sees — the very trigger phrases meant to route invocation.
*
* Detection (v1, high confidence — no estimation):
* Detection:
* CA-SKL-001 active skill description > 1,536 chars → truncated (medium)
* CA-SKL-002 sum of active descriptions exceeds the listing budget (low)
*
* The aggregate listing budget (skill descriptions sum vs ~2% of the context
* window, v2.1.32 L2860) is deliberately NOT scanned in v1: it requires
* assuming the user's context-window size, which would turn a verified fact
* into a guess. Lead with the cap; the aggregate check is a possible later
* addition (would carry a CALIBRATION_NOTE).
* The aggregate listing budget (changelog L2860, CC 2.1.32) allots the skill
* listing the model reads ~2% of the context window. We do NOT know the user's
* context window, so CA-SKL-002 anchors on a conservative 200k window and says
* so loudly: it leads with the measured sum (a fact) and carries a calibration
* note explaining the budget scales 5x on a 1M-context model. Severity is low
* (an estimate) versus medium for the verified per-description cap. Each
* description is counted only up to the 1,536-char cap, because that is all
* Claude Code loads into the listing — the tail past the cap is dropped and is
* already flagged by CA-SKL-001 (so the aggregate does not double-count it).
*
* Two-lens note vs TOK pattern F: TOK pattern F flags *project-local* skills
* with descriptions > 500 chars as a structural per-turn "bloat" heuristic.
@ -29,7 +34,7 @@
import { finding, scannerResult } from './lib/output.mjs';
import { SEVERITY } from './lib/severity.mjs';
import { enumeratePlugins, enumerateSkills } from './lib/active-config-reader.mjs';
import { enumeratePlugins, enumerateSkills, estimateTokens } from './lib/active-config-reader.mjs';
import { readTextFile } from './lib/file-discovery.mjs';
import { parseFrontmatter } from './lib/yaml-parser.mjs';
@ -39,6 +44,26 @@ const SCANNER = 'SKL';
// Descriptions longer than this are truncated in the listing the model sees.
const DESCRIPTION_CAP = 1536;
// Aggregate listing budget (CC 2.1.32, changelog L2860): the skill listing the
// model reads is allotted ~2% of the context window. The context window is
// unknown, so we anchor on a conservative 200k window — the smallest common
// size, which fires earliest — and disclose the assumption in the evidence.
const BUDGET_FRACTION = 0.02;
const CONTEXT_WINDOW_ANCHOR = 200_000;
const AGGREGATE_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * CONTEXT_WINDOW_ANCHOR); // 4000
const LARGE_CONTEXT_WINDOW = 1_000_000;
const LARGE_CONTEXT_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * LARGE_CONTEXT_WINDOW); // 20000
// Dependency-free thousands separator (repo invariant: zero external deps).
const withCommas = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
// Appended to CA-SKL-002 evidence — the honest framing required because the
// budget depends on a context window we cannot observe (jf. TOK CALIBRATION_NOTE).
const BUDGET_CALIBRATION_NOTE =
'the budget scales with the context window - this anchors on a conservative 200k ' +
`window; at ${withCommas(LARGE_CONTEXT_WINDOW)} context the budget is ~${withCommas(LARGE_CONTEXT_BUDGET_TOKENS)} ` +
'tok and you are likely within it. this is an estimate, not measured telemetry';
/**
* Main scanner entry point.
*
@ -53,6 +78,7 @@ export async function scan(_targetPath, _discovery) {
const allSkills = await enumerateSkills(plugins);
let scanned = 0;
let aggregateChars = 0;
for (const skill of allSkills) {
if (!skill || typeof skill.path !== 'string') continue;
const content = await readTextFile(skill.path);
@ -60,6 +86,9 @@ export async function scan(_targetPath, _discovery) {
scanned++;
const fm = parseFrontmatter(content)?.frontmatter || null;
const desc = (fm && typeof fm.description === 'string') ? fm.description : '';
// Aggregate budget counts only what loads in the listing: each description
// up to the cap (the tail past the cap is dropped, and CA-SKL-001 flags it).
aggregateChars += Math.min(desc.length, DESCRIPTION_CAP);
if (desc.length <= DESCRIPTION_CAP) continue;
const sourceLabel = skill.source === 'plugin'
@ -90,5 +119,34 @@ export async function scan(_targetPath, _discovery) {
}));
}
// CA-SKL-002 (aggregate). Emitted after the per-skill findings so the common
// "one oversized skill + aggregate" case reads 001=cap, 002=aggregate.
const aggregateTokens = estimateTokens(aggregateChars, 'markdown');
if (aggregateTokens > AGGREGATE_BUDGET_TOKENS) {
findings.push(finding({
scanner: SCANNER,
severity: SEVERITY.low,
title: 'Aggregate skill descriptions may exceed the listing budget',
description:
`The ${scanned} active skills carry about ${aggregateTokens} tokens of description text ` +
`(each description counted up to the ${DESCRIPTION_CAP}-char listing cap), above the ` +
`${AGGREGATE_BUDGET_TOKENS}-token budget Claude Code allots the skill listing on a 200k ` +
'context window (about 2% of context, CC 2.1.32). When the listing overflows that budget ' +
'Claude Code drops descriptions, so the model may stop seeing some skills entirely. This ' +
'is an estimate — the budget scales with your actual context window (see evidence).',
evidence:
`active_skills_scanned=${scanned}; description_chars=${aggregateChars} (each capped at ` +
`${DESCRIPTION_CAP}); description_tokens~${aggregateTokens}; budget@200k=` +
`${AGGREGATE_BUDGET_TOKENS} tok (skill listing ~2% of context, CC 2.1.32); over_by~` +
`${aggregateTokens - AGGREGATE_BUDGET_TOKENS} tok - ${BUDGET_CALIBRATION_NOTE}`,
recommendation:
'Reclaim skill-listing budget: set `disableBundledSkills: true` to drop bundled skills you ' +
'do not use from the listing, use `skillOverrides` (`name-only` collapses a description, ' +
'`off` removes a skill) on the heaviest entries, and trim long descriptions toward their ' +
'trigger phrases.',
category: 'token-efficiency',
}));
}
return scannerResult(SCANNER, 'ok', findings, scanned, Date.now() - start);
}