feat(skill-listing): add CA-SKL-002 aggregate listing-budget check
Syklus 2 of Fase 4 Items 2+3. Flags when the sum of active skill descriptions exceeds the listing budget (~2% of context, CC 2.1.32). Design (operator-confirmed "fact-first, 200k anchor"): - low severity (estimate) vs medium for the verified 1,536-char cap - each description counted up to the 1,536 cap (what actually loads in the listing) — avoids double-counting the tail CA-SKL-001 flags - fires when sum > 2% x 200k = 4000 tok; evidence leads with the measured sum + a calibration note that the budget scales 5x on 1M-context models - aggregate emitted after the per-skill loop so the common case reads 001=cap, 002=aggregate (finding IDs are a sequential counter, not stable semantic IDs — tests match on title, never NNN) Also: - tailored humanizer static entry for the aggregate title - fix latent HOME leak in posture-grade-stability.test.mjs: it spawned posture.mjs without hermeticEnv(), so a real ~/.claude leaked HOME-scoped SKL/COL findings into the baseline grade (Token Efficiency A->B). Now isolated like the 8 other CLI-spawning tests. - docs sync: test count 868->875, scanner-internals, gap-matrix, plan status Suite 875/875, no snapshot drift, self-audit clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ter3E2JSi1Khgmuf2kady8
This commit is contained in:
parent
7bb254780a
commit
66433fee48
8 changed files with 215 additions and 11 deletions
|
|
@ -109,7 +109,7 @@ Default: auto-detects scope from git context. Override with `/config-audit full|
|
|||
node --test 'tests/**/*.test.mjs'
|
||||
```
|
||||
|
||||
868 tests across 55 test files (16 lib + 29 scanner + 1 hook + 1 agent + 3 commands + 1 knowledge + 4 top-level). Test fixtures in `tests/fixtures/`. Top-level humanizer tests: `json-backcompat.test.mjs`, `raw-backcompat.test.mjs`, `scenario-read-test.test.mjs`, `snapshot-default-output.test.mjs`.
|
||||
875 tests across 55 test files (16 lib + 29 scanner + 1 hook + 1 agent + 3 commands + 1 knowledge + 4 top-level). Test fixtures in `tests/fixtures/`. Top-level humanizer tests: `json-backcompat.test.mjs`, `raw-backcompat.test.mjs`, `scenario-read-test.test.mjs`, `snapshot-default-output.test.mjs`.
|
||||
|
||||
## Gotchas
|
||||
|
||||
|
|
|
|||
|
|
@ -1,6 +1,8 @@
|
|||
# Plan — Fase 4 Items 2 + 3 (skill-listing token-styring)
|
||||
|
||||
**Skrevet:** 2026-06-18 · **Scope:** kun `config-audit/` · **Status:** ✅ **LEVERT 2026-06-18.** Designvalg bekreftet: A=fold inn i SKL, B=v1 KUN 1 536-cap (aggregat utsatt), C=`skillOverrides` i KNOWN_KEYS (IKKE TYPE_CHECKS — verdien er objekt; avvik fra §C-skissen, verify-først). Syklus 1+3+4 ferdig; Syklus 2 (aggregat CA-SKL-002) gjenstår som valgfri stretch. Suite 868/868. Detaljer i `STATE.md`.
|
||||
**Skrevet:** 2026-06-18 · **Scope:** kun `config-audit/` · **Status:** ✅ **LEVERT 2026-06-18 (alle 4 sykluser).** Designvalg bekreftet: A=fold inn i SKL, B=**Syklus 2 NÅ LEVERT** — `CA-SKL-002` aggregat-budsjett implementert med «faktum-først, 200k-anker» (operatør-bekreftet 2026-06-18): low severity, sum av beskrivelser (hver tellet opp til 1 536-cap) > 2 %×200k = 4000 tok → funn m/ CALIBRATION_NOTE om 5×-skalering på 1M-kontekst. C=`skillOverrides` i KNOWN_KEYS (IKKE TYPE_CHECKS — verdien er objekt; avvik fra §C-skissen, verify-først). Suite **875/875**. Detaljer i `STATE.md`.
|
||||
|
||||
**Syklus 2-notat (avvik fra §B-skissen, verify-først):** (1) Aggregatet teller hver beskrivelse **capped på 1 536** (= det som faktisk lastes i listingen; unngår dobbelttelling av halen `CA-SKL-001` allerede flagger). (2) Finding-IDs er **sekvensiell teller** (`output.mjs:33`), ikke stabil semantisk ID — tester matcher på **tittel**, aldri NNN; aggregatet emitteres ETTER cap-loopen så vanlig-tilfellet leser 001=cap, 002=aggregat. (3) Eksponerte en latent **HOME-lekkasje** i `posture-grade-stability.test.mjs` (kjørte CLI uten hermetisk HOME → ekte ~/.claude lekket inn → Token Efficiency falt A→B); fikset med `hermeticEnv()` (samme mønster som 8 andre CLI-tester). (4) Egen tilpasset humanizer-`static`-entry for aggregat-tittelen (ikke generisk `_default`).
|
||||
**Forrige:** Fase 4 Item 1 (4.7→modell-nøytral) levert som `8376dab`. Baseline **856/856 grønn**.
|
||||
|
||||
## Mål
|
||||
|
|
|
|||
|
|
@ -97,7 +97,7 @@ Netto-nytt scanner-arbeid (DIS param-bevissthet, PLH shadow-folder-check, permis
|
|||
| settings.json | env MAX_THINKING_TOKENS=0 disables thinking | 2.1.166 | missing | update-knowledge | knowledge/claude-code-capabilities.md | L | S |
|
||||
| settings.json | parentSettingsBehavior / policyHelper (UNVERIFIED) | 2.1.133 | missing | new-feature-gap-check | scanners/settings-validator.mjs | L | S |
|
||||
| settings.json | skillOverrides | 2.1.129 | DONE | added to KNOWN_KEYS (NOT TYPE_CHECKS — value is per-skill object off/user-invocable-only/name-only) | scanners/settings-validator.mjs | L | S |
|
||||
| settings.json | ~~skillListingBudgetFraction / maxSkillDescriptionChars~~ (keys DO NOT EXIST — verified) | 2.1.105 | DONE | budget=2% ctx + cap=1536 are internal CC mechanisms, not settings keys; the verified 1,536-char cap is enforced by the new SKL scanner (CA-SKL-001) | scanners/skill-listing-scanner.mjs | L | M |
|
||||
| settings.json | ~~skillListingBudgetFraction / maxSkillDescriptionChars~~ (keys DO NOT EXIST — verified) | 2.1.105 + 2.1.32 | DONE | budget=2% ctx + cap=1536 are internal CC mechanisms, not settings keys; both enforced by the SKL scanner — `CA-SKL-001` (1,536-char cap) + `CA-SKL-002` (aggregate 2%-of-context budget, 200k-anchored estimate) | scanners/skill-listing-scanner.mjs | L | M |
|
||||
| CLAUDE.md / memory | 'too long' threshold now scales with context window | 2.1.169 | now-wrong | fix-false-positive | scanners/claude-md-linter.mjs | H | M |
|
||||
| CLAUDE.md / memory | --safe-mode / CLAUDE_CODE_SAFE_MODE disables customizations | 2.1.169 | missing | update-knowledge | knowledge/configuration-best-practices.md | L | S |
|
||||
| CLAUDE.md / memory | plugins in .claude/skills auto-load (cascade/token accounting) | 2.1.157 | partial | new-feature-gap-check | knowledge/configuration-best-practices.md | M | M |
|
||||
|
|
|
|||
|
|
@ -22,7 +22,7 @@ Scanner CLI: `node scanners/scan-orchestrator.mjs <path> [--global] [--full-mach
|
|||
| `cache-prefix-scanner.mjs` | CPS | Volatile content in lines 31–150 of CLAUDE.md cascade (beyond Pattern A's top-30 window) |
|
||||
| `disabled-in-schema-scanner.mjs` | DIS | Tools listed in BOTH `permissions.deny` AND `permissions.allow` — deny wins, allow entries are dead config |
|
||||
| `collision-scanner.mjs` | COL | Cross-plugin skill name collisions (low); user-vs-plugin overlaps (medium); `details.namespaces` payload |
|
||||
| `skill-listing-scanner.mjs` | SKL | Active skill descriptions over the verified 1,536-char listing cap (CC 2.1.105) → silently truncated in the model's skill listing (medium). HOME-scoped (all user + plugin skills). Remediation surfaces `disableBundledSkills` / `skillOverrides`. Distinct lens from TOK pattern F (project-local 500-char bloat heuristic) |
|
||||
| `skill-listing-scanner.mjs` | SKL | (1) `CA-SKL-001` (medium): active skill descriptions over the verified 1,536-char listing cap (CC 2.1.105) → silently truncated in the model's skill listing. (2) `CA-SKL-002` (low): sum of active descriptions (each counted up to the cap) over the listing budget (~2% of context, CC 2.1.32), anchored on a conservative 200k window with a calibration note that the budget scales 5× on 1M-context models — leads with the measured sum, an estimate not telemetry. HOME-scoped (all user + plugin skills). Remediation surfaces `disableBundledSkills` / `skillOverrides` / trim. Distinct lens from TOK pattern F (project-local 500-char bloat heuristic) |
|
||||
|
||||
## Scanner Lib (`scanners/lib/`)
|
||||
|
||||
|
|
|
|||
|
|
@ -742,6 +742,11 @@ export const TRANSLATIONS = {
|
|||
description: 'Claude Code only shows the first part of each skill\'s description when it decides which skill to use. This one runs past the limit, so the end — often the trigger phrases — is silently dropped.',
|
||||
recommendation: 'Shorten the description so the trigger words come first and it fits under the limit. For skills you do not use, turning them off frees up the listing entirely.',
|
||||
},
|
||||
'Aggregate skill descriptions may exceed the listing budget': {
|
||||
title: 'All your skills together may be too much for the listing',
|
||||
description: 'Claude Code keeps every active skill\'s description in one shared listing it reads to choose which skill to use, and that listing has a limited size. Added up, your skills\' descriptions run past that size on a smaller setup, so Claude Code may drop some of them — and stop seeing those skills. This is an estimate; a larger setup has more room.',
|
||||
recommendation: 'Free up room: turn off bundled skills you do not use, collapse the heaviest ones so only their names show, or shorten the longest descriptions. The details show the measured total and the room available.',
|
||||
},
|
||||
},
|
||||
patterns: [],
|
||||
_default: {
|
||||
|
|
|
|||
|
|
@ -9,14 +9,19 @@
|
|||
* happens). A description past the cap loses its tail from what the model
|
||||
* actually sees — the very trigger phrases meant to route invocation.
|
||||
*
|
||||
* Detection (v1, high confidence — no estimation):
|
||||
* Detection:
|
||||
* CA-SKL-001 active skill description > 1,536 chars → truncated (medium)
|
||||
* CA-SKL-002 sum of active descriptions exceeds the listing budget (low)
|
||||
*
|
||||
* The aggregate listing budget (skill descriptions sum vs ~2% of the context
|
||||
* window, v2.1.32 L2860) is deliberately NOT scanned in v1: it requires
|
||||
* assuming the user's context-window size, which would turn a verified fact
|
||||
* into a guess. Lead with the cap; the aggregate check is a possible later
|
||||
* addition (would carry a CALIBRATION_NOTE).
|
||||
* The aggregate listing budget (changelog L2860, CC 2.1.32) allots the skill
|
||||
* listing the model reads ~2% of the context window. We do NOT know the user's
|
||||
* context window, so CA-SKL-002 anchors on a conservative 200k window and says
|
||||
* so loudly: it leads with the measured sum (a fact) and carries a calibration
|
||||
* note explaining the budget scales 5x on a 1M-context model. Severity is low
|
||||
* (an estimate) versus medium for the verified per-description cap. Each
|
||||
* description is counted only up to the 1,536-char cap, because that is all
|
||||
* Claude Code loads into the listing — the tail past the cap is dropped and is
|
||||
* already flagged by CA-SKL-001 (so the aggregate does not double-count it).
|
||||
*
|
||||
* Two-lens note vs TOK pattern F: TOK pattern F flags *project-local* skills
|
||||
* with descriptions > 500 chars as a structural per-turn "bloat" heuristic.
|
||||
|
|
@ -29,7 +34,7 @@
|
|||
|
||||
import { finding, scannerResult } from './lib/output.mjs';
|
||||
import { SEVERITY } from './lib/severity.mjs';
|
||||
import { enumeratePlugins, enumerateSkills } from './lib/active-config-reader.mjs';
|
||||
import { enumeratePlugins, enumerateSkills, estimateTokens } from './lib/active-config-reader.mjs';
|
||||
import { readTextFile } from './lib/file-discovery.mjs';
|
||||
import { parseFrontmatter } from './lib/yaml-parser.mjs';
|
||||
|
||||
|
|
@ -39,6 +44,26 @@ const SCANNER = 'SKL';
|
|||
// Descriptions longer than this are truncated in the listing the model sees.
|
||||
const DESCRIPTION_CAP = 1536;
|
||||
|
||||
// Aggregate listing budget (CC 2.1.32, changelog L2860): the skill listing the
|
||||
// model reads is allotted ~2% of the context window. The context window is
|
||||
// unknown, so we anchor on a conservative 200k window — the smallest common
|
||||
// size, which fires earliest — and disclose the assumption in the evidence.
|
||||
const BUDGET_FRACTION = 0.02;
|
||||
const CONTEXT_WINDOW_ANCHOR = 200_000;
|
||||
const AGGREGATE_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * CONTEXT_WINDOW_ANCHOR); // 4000
|
||||
const LARGE_CONTEXT_WINDOW = 1_000_000;
|
||||
const LARGE_CONTEXT_BUDGET_TOKENS = Math.round(BUDGET_FRACTION * LARGE_CONTEXT_WINDOW); // 20000
|
||||
|
||||
// Dependency-free thousands separator (repo invariant: zero external deps).
|
||||
const withCommas = (n) => String(n).replace(/\B(?=(\d{3})+(?!\d))/g, ',');
|
||||
|
||||
// Appended to CA-SKL-002 evidence — the honest framing required because the
|
||||
// budget depends on a context window we cannot observe (jf. TOK CALIBRATION_NOTE).
|
||||
const BUDGET_CALIBRATION_NOTE =
|
||||
'the budget scales with the context window - this anchors on a conservative 200k ' +
|
||||
`window; at ${withCommas(LARGE_CONTEXT_WINDOW)} context the budget is ~${withCommas(LARGE_CONTEXT_BUDGET_TOKENS)} ` +
|
||||
'tok and you are likely within it. this is an estimate, not measured telemetry';
|
||||
|
||||
/**
|
||||
* Main scanner entry point.
|
||||
*
|
||||
|
|
@ -53,6 +78,7 @@ export async function scan(_targetPath, _discovery) {
|
|||
const allSkills = await enumerateSkills(plugins);
|
||||
|
||||
let scanned = 0;
|
||||
let aggregateChars = 0;
|
||||
for (const skill of allSkills) {
|
||||
if (!skill || typeof skill.path !== 'string') continue;
|
||||
const content = await readTextFile(skill.path);
|
||||
|
|
@ -60,6 +86,9 @@ export async function scan(_targetPath, _discovery) {
|
|||
scanned++;
|
||||
const fm = parseFrontmatter(content)?.frontmatter || null;
|
||||
const desc = (fm && typeof fm.description === 'string') ? fm.description : '';
|
||||
// Aggregate budget counts only what loads in the listing: each description
|
||||
// up to the cap (the tail past the cap is dropped, and CA-SKL-001 flags it).
|
||||
aggregateChars += Math.min(desc.length, DESCRIPTION_CAP);
|
||||
if (desc.length <= DESCRIPTION_CAP) continue;
|
||||
|
||||
const sourceLabel = skill.source === 'plugin'
|
||||
|
|
@ -90,5 +119,34 @@ export async function scan(_targetPath, _discovery) {
|
|||
}));
|
||||
}
|
||||
|
||||
// CA-SKL-002 (aggregate). Emitted after the per-skill findings so the common
|
||||
// "one oversized skill + aggregate" case reads 001=cap, 002=aggregate.
|
||||
const aggregateTokens = estimateTokens(aggregateChars, 'markdown');
|
||||
if (aggregateTokens > AGGREGATE_BUDGET_TOKENS) {
|
||||
findings.push(finding({
|
||||
scanner: SCANNER,
|
||||
severity: SEVERITY.low,
|
||||
title: 'Aggregate skill descriptions may exceed the listing budget',
|
||||
description:
|
||||
`The ${scanned} active skills carry about ${aggregateTokens} tokens of description text ` +
|
||||
`(each description counted up to the ${DESCRIPTION_CAP}-char listing cap), above the ` +
|
||||
`${AGGREGATE_BUDGET_TOKENS}-token budget Claude Code allots the skill listing on a 200k ` +
|
||||
'context window (about 2% of context, CC 2.1.32). When the listing overflows that budget ' +
|
||||
'Claude Code drops descriptions, so the model may stop seeing some skills entirely. This ' +
|
||||
'is an estimate — the budget scales with your actual context window (see evidence).',
|
||||
evidence:
|
||||
`active_skills_scanned=${scanned}; description_chars=${aggregateChars} (each capped at ` +
|
||||
`${DESCRIPTION_CAP}); description_tokens~${aggregateTokens}; budget@200k=` +
|
||||
`${AGGREGATE_BUDGET_TOKENS} tok (skill listing ~2% of context, CC 2.1.32); over_by~` +
|
||||
`${aggregateTokens - AGGREGATE_BUDGET_TOKENS} tok - ${BUDGET_CALIBRATION_NOTE}`,
|
||||
recommendation:
|
||||
'Reclaim skill-listing budget: set `disableBundledSkills: true` to drop bundled skills you ' +
|
||||
'do not use from the listing, use `skillOverrides` (`name-only` collapses a description, ' +
|
||||
'`off` removes a skill) on the heaviest entries, and trim long descriptions toward their ' +
|
||||
'trigger phrases.',
|
||||
category: 'token-efficiency',
|
||||
}));
|
||||
}
|
||||
|
||||
return scannerResult(SCANNER, 'ok', findings, scanned, Date.now() - start);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -4,6 +4,7 @@ import { resolve } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { execFile } from 'node:child_process';
|
||||
import { promisify } from 'node:util';
|
||||
import { hermeticEnv } from '../helpers/hermetic-home.mjs';
|
||||
|
||||
const exec = promisify(execFile);
|
||||
const __dirname = fileURLToPath(new URL('.', import.meta.url));
|
||||
|
|
@ -11,9 +12,13 @@ const FIXTURES = resolve(__dirname, '../fixtures');
|
|||
const POSTURE_BIN = resolve(__dirname, '../../scanners/posture.mjs');
|
||||
|
||||
async function runPostureJson(fixturePath) {
|
||||
// Isolate HOME: SKL and COL are HOME-scoped (enumerateSkills/enumeratePlugins),
|
||||
// so a real ~/.claude would leak the developer's installed plugins/skills into
|
||||
// this baseline grade and make it machine-dependent. See tests/helpers/hermetic-home.mjs.
|
||||
const { stdout } = await exec('node', [POSTURE_BIN, fixturePath, '--json'], {
|
||||
timeout: 30000,
|
||||
cwd: resolve(__dirname, '../..'),
|
||||
env: hermeticEnv(),
|
||||
});
|
||||
return JSON.parse(stdout);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -41,6 +41,34 @@ async function homeWithUserSkill(name, descLen) {
|
|||
return home;
|
||||
}
|
||||
|
||||
/** Write one user skill (name + description of `descLen` chars) into an existing HOME. */
|
||||
async function addUserSkill(home, name, descLen) {
|
||||
const dir = join(home, '.claude', 'skills', name);
|
||||
await mkdir(dir, { recursive: true });
|
||||
const desc = 'a'.repeat(descLen);
|
||||
await writeFile(
|
||||
join(dir, 'SKILL.md'),
|
||||
`---\nname: ${name}\ndescription: ${desc}\n---\nBody of the skill.\n`,
|
||||
);
|
||||
}
|
||||
|
||||
/** Build a fake HOME with `count` user skills, each description `descLen` chars. */
|
||||
async function homeWithNUserSkills(count, descLen, prefix = 'agg') {
|
||||
const home = uniqueDir(`${prefix}-${count}x${descLen}`);
|
||||
for (let i = 0; i < count; i++) {
|
||||
await addUserSkill(home, `${prefix}${i}`, descLen);
|
||||
}
|
||||
return home;
|
||||
}
|
||||
|
||||
// Aggregate listing budget: 2% (CC 2.1.32) of the conservative 200k anchor.
|
||||
const AGGREGATE_BUDGET_TOKENS = 4000; // 0.02 * 200_000
|
||||
// Token heuristic mirrors estimateTokens('markdown'): ceil(chars / 4).
|
||||
// 16 skills * 1000 chars = 16000 chars -> 4000 tok == budget (NOT over).
|
||||
// 17 skills * 1000 chars = 17000 chars -> 4250 tok > budget (over).
|
||||
const findAggregate = (findings) => findings.find(f => /aggregate/i.test(f.title));
|
||||
const findCaps = (findings) => findings.filter(f => /listing cap/i.test(f.title));
|
||||
|
||||
describe('SKL scanner — basic structure', () => {
|
||||
it('reports scanner prefix SKL', async () => {
|
||||
const home = uniqueDir('empty');
|
||||
|
|
@ -169,6 +197,112 @@ describe('SKL scanner — remediation levers (Item 2b folded in)', () => {
|
|||
});
|
||||
});
|
||||
|
||||
describe('SKL scanner — aggregate listing budget (CA-SKL-002)', () => {
|
||||
it('fires a low-severity aggregate finding when the description sum exceeds the 200k-anchored budget', async () => {
|
||||
const home = await homeWithNUserSkills(17, 1000); // 17000 chars -> ~4250 tok > 4000
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
const agg = findAggregate(result.findings);
|
||||
assert.ok(agg, `expected an aggregate finding; got: ${result.findings.map(f => f.title).join(' | ')}`);
|
||||
assert.equal(agg.severity, 'low', `aggregate should be low severity, got ${agg.severity}`);
|
||||
assert.match(agg.id, /^CA-SKL-\d{3}$/);
|
||||
// 17 skills of 1000 chars each are all under the 1536 cap, so no cap findings.
|
||||
assert.equal(findCaps(result.findings).length, 0,
|
||||
'no per-description cap findings expected when every description is under 1536');
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('does NOT fire when the aggregate is under budget', async () => {
|
||||
const home = await homeWithNUserSkills(3, 1000); // 3000 chars -> 750 tok < 4000
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
assert.equal(findAggregate(result.findings), undefined,
|
||||
`expected no aggregate finding; got: ${result.findings.map(f => f.title).join(' | ')}`);
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('treats the budget as a boundary: exactly at budget yields no finding, one skill over yields one', async () => {
|
||||
const atBudget = await homeWithNUserSkills(16, 1000, 'at'); // 16000 chars -> 4000 tok == budget
|
||||
try {
|
||||
const result = await runScannerWithHome(atBudget);
|
||||
assert.equal(findAggregate(result.findings), undefined,
|
||||
'aggregate exactly at the budget must not fire (strictly-greater rule)');
|
||||
} finally {
|
||||
await rm(atBudget, { recursive: true, force: true });
|
||||
}
|
||||
const overBudget = await homeWithNUserSkills(17, 1000, 'over'); // 17000 chars -> 4250 tok
|
||||
try {
|
||||
const result = await runScannerWithHome(overBudget);
|
||||
assert.ok(findAggregate(result.findings), 'aggregate one step over the budget must fire');
|
||||
} finally {
|
||||
await rm(overBudget, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('counts each description capped at 1536 — a single huge skill does NOT inflate the aggregate', async () => {
|
||||
// One 5000-char skill: capped contribution = 1536 chars -> 384 tok, far under budget.
|
||||
// It must trigger the per-description cap finding but NOT the aggregate.
|
||||
const home = await homeWithUserSkill('huge', 5000);
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
assert.equal(findCaps(result.findings).length, 1, 'the oversized skill should trip the cap finding');
|
||||
assert.equal(findAggregate(result.findings), undefined,
|
||||
'aggregate must count the capped 1536 contribution, not the full 5000 chars');
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('cap and aggregate findings are independent — both can fire in one run', async () => {
|
||||
// 1 oversized skill (cap) + 16 normal skills. Aggregate = min(5000,1536)+16000 = 17536 chars -> ~4384 tok.
|
||||
const home = await homeWithNUserSkills(16, 1000, 'norm');
|
||||
await addUserSkill(home, 'oversized', 5000);
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
assert.equal(findCaps(result.findings).length, 1, 'expected exactly one cap finding');
|
||||
assert.ok(findAggregate(result.findings), 'expected an aggregate finding alongside the cap finding');
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('evidence carries the measured sum, the 200k budget, and an honest context-window calibration note', async () => {
|
||||
const home = await homeWithNUserSkills(17, 1000);
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
const agg = findAggregate(result.findings);
|
||||
assert.ok(agg, 'expected an aggregate finding');
|
||||
const ev = String(agg.evidence);
|
||||
assert.match(ev, new RegExp(String(AGGREGATE_BUDGET_TOKENS)), 'evidence should state the 4000-token budget');
|
||||
assert.match(ev, /200k/, 'evidence should anchor the budget on a 200k window');
|
||||
assert.match(ev, /CC 2\.1\.32/, 'evidence should cite the 2%-of-context changelog fact');
|
||||
assert.match(ev, /1,000,000|20,000/, 'evidence should note the 1M-context scaling');
|
||||
assert.match(ev, /estimate/i, 'evidence should flag this as an estimate, not measured telemetry');
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
it('aggregate recommendation lists the same levers (disableBundledSkills, skillOverrides, trim)', async () => {
|
||||
const home = await homeWithNUserSkills(17, 1000);
|
||||
try {
|
||||
const result = await runScannerWithHome(home);
|
||||
const agg = findAggregate(result.findings);
|
||||
assert.ok(agg, 'expected an aggregate finding to carry remediation');
|
||||
const rec = String(agg.recommendation);
|
||||
assert.match(rec, /disableBundledSkills/);
|
||||
assert.match(rec, /skillOverrides/);
|
||||
assert.match(rec, /trim/i);
|
||||
} finally {
|
||||
await rm(home, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
});
|
||||
|
||||
describe('SKL scanner — suppression compatibility', () => {
|
||||
it('CA-SKL-001 is NOT matched by a CA-TOK-* glob suppression', async () => {
|
||||
const { applySuppressions } = await import('../../scanners/lib/suppression.mjs');
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue