feat(tokens): MCP tool-schema deferral check + CLI-over-MCP lever (v5.10 B4)

By default Claude Code defers MCP tool schemas (names-only, ~120 tok; full
schemas on demand via tool search). CA-TOK-006 detects config-file signals that
force the FULL schemas into the always-loaded prefix every turn:
  - settings.json env.ENABLE_TOOL_SEARCH="false"          (high)
  - "ToolSearch" in permissions.deny                       (high)
  - configured model is a Haiku model                      (medium)
  - per-server .mcp.json alwaysLoad:true (CC v2.1.121+)    (high)
auto[:N] is threshold mode (info, not a trigger).

New engine lib/mcp-deferral.mjs: pure assessMcpDeferral({settings,mcpServers})
(unit-tested, no IO) + thin IO wrapper assessMcpDeferralForRepo shared by TOK and
GAP. Severity scales with aggregate forced-upfront tokens (medium-confidence
reasons cap at medium). feature-gap cliOverMcpLeverFinding fires only as a
companion to CA-TOK-006 (prefer gh/aws/gcloud over MCP for common ops).

Honest scoping (Verifiseringsplikt): triggers on config files ONLY — never
process.env shell vars. Vertex / custom ANTHROPIC_BASE_URL / runtime /model
switch are launch state (would flap snapshots machine-dependently), so they are
DISCLOSED in every finding, not triggered. Tool-level anthropic/alwaysLoad and
claude.ai connectors likewise disclosed. Mechanism verified 2026-06-23 against
code.claude.com/docs (context-window.md, mcp.md#configure-tool-search +
#exempt-a-server-from-deferral, costs.md); the prefix-cache invalidation claim
was NOT-CONFIRMED in docs and is not asserted.

alwaysLoad added to CA-MCP VALID_SERVER_FIELDS (no longer flagged as unknown).
active-config-reader surfaces per-server alwaysLoad. Byte-stable: CA-TOK-006
fires only on new conditions; frozen v5.0.0 + SC-5 snapshots untouched.
Tests 1215 -> 1239 (engine 16, integration 5, lever 2, mcp-field guard 1).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-06-23 19:59:14 +02:00
commit 8f7e196046
13 changed files with 692 additions and 2 deletions

View file

@ -13,6 +13,7 @@ import { finding, scannerResult } from './lib/output.mjs';
import { SEVERITY } from './lib/severity.mjs';
import { findImports, parseJson, parseFrontmatter } from './lib/yaml-parser.mjs';
import { measureActiveSkillListing, isBundledSkillsDisabled, BUDGET_CALIBRATION_NOTE } from './lib/skill-listing-budget.mjs';
import { assessMcpDeferralForRepo } from './lib/mcp-deferral.mjs';
const SCANNER = 'GAP';
@ -134,6 +135,46 @@ export function bundledSkillsLeverFinding({ leverPulled, aggregate }) {
});
}
/**
* CLI-over-MCP lever remediation companion to CA-TOK-006 (v5.10 B4).
*
* Fires ONLY when MCP tool schemas are forced into the always-loaded prefix
* (tool search disabled, or a per-server alwaysLoad), i.e. when MCP is actually
* costing always-loaded tokens. When schemas are deferred (the default), MCP is
* effectively free until used, so there is nothing to recommend and we stay
* silent opportunity, not noise (mirrors the bundledSkills lever's "fire only
* under measured pressure" contract). CLI tools (gh / aws / gcloud) add zero
* context tokens until invoked, so they are the lever the deferral mechanism
* cannot reach for the forced-upfront servers.
*
* Pure and exported for unit testing.
*
* @param {{ assessment: (import('./lib/mcp-deferral.mjs').assessMcpDeferral)|null }} args
* @returns {object|null} a GAP finding, or null when nothing is forced upfront
*/
export function cliOverMcpLeverFinding({ assessment } = {}) {
if (!assessment || !assessment.forcedUpfront) return null;
const names = (assessment.affectedServers || []).map((m) => m.name).join(', ');
return finding({
scanner: SCANNER,
severity: SEVERITY.low,
title: 'Prefer CLI over MCP for common operations',
description:
`Your active project MCP tool schemas (~${assessment.aggregateTokens} tokens) are forced into the ` +
'always-loaded prefix every turn rather than deferred (see CA-TOK-006). CLI tools (gh, aws, gcloud, …) ' +
'add ZERO context tokens until you actually call them, so moving common operations off MCP and onto a ' +
'CLI reclaims always-loaded budget the deferral mechanism cannot.',
evidence:
`forced_schema_tokens~${assessment.aggregateTokens}; servers=${names}; ` +
`reason=${assessment.reason || 'alwaysLoad'} (companion to CA-TOK-006)`,
recommendation:
'For operations a CLI already covers (GitHub → gh, AWS → aws, GCP → gcloud), prefer the CLI over an ' +
'MCP server — CLI output enters context only when invoked. Keep MCP for capabilities with no CLI ' +
'equivalent, disable unused servers via /mcp, and re-enable tool-search deferral so the rest stay names-only.',
category: 'token-efficiency',
});
}
/** @type {GapCheck[]} */
const GAP_CHECKS = [
// --- Tier 1: Foundation ---
@ -441,6 +482,13 @@ export async function scan(targetPath, sharedDiscovery) {
const leverFinding = bundledSkillsLeverFinding({ leverPulled, aggregate });
if (leverFinding) findings.push(leverFinding);
// CLI-over-MCP lever — companion to CA-TOK-006: fires only when project-local
// MCP tool schemas are forced into the always-loaded prefix (tool search
// disabled or a per-server alwaysLoad). Reuses the same static assessment.
const mcpAssessment = await assessMcpDeferralForRepo(ctx.targetPath);
const cliLever = cliOverMcpLeverFinding({ assessment: mcpAssessment });
if (cliLever) findings.push(cliLever);
const filesScanned = discovery.files.length;
return scannerResult(SCANNER, 'ok', findings, filesScanned, Date.now() - start);
}

View file

@ -788,6 +788,7 @@ export async function readActiveMcpServers(repoPath, claudeJsonSlice = null, plu
toolCount,
toolCountUnknown: detected.toolCountUnknown,
estimatedTokens: estimateTokens(0, 'mcp', { toolCount: toolCount ?? 0 }),
alwaysLoad: def?.alwaysLoad === true,
});
}
@ -817,6 +818,7 @@ async function collectMcpFromFile(path, source, disabled, out, repoPath) {
toolCount,
toolCountUnknown: detected.toolCountUnknown,
estimatedTokens: estimateTokens(0, 'mcp', { toolCount: toolCount ?? 0 }),
alwaysLoad: def?.alwaysLoad === true,
});
}
}

View file

@ -0,0 +1,206 @@
/**
* MCP tool-schema deferral assessment (v5.10 B4).
*
* By default Claude Code DEFERS MCP tool schemas: only tool *names* enter the
* always-loaded prefix (~120 tokens total) and full schemas load on demand via
* tool search. Certain conditions force ALL full schemas into the always-loaded
* prefix instead paid on every turn whether or not a tool is used.
*
* This module is a STATIC, CONFIG-FILE assessment. It triggers only on signals
* that live in config files a scanner can read deterministically:
* - settings.json `env.ENABLE_TOOL_SEARCH` = "false" (HIGH confidence)
* - settings.json `permissions.deny` contains "ToolSearch" (HIGH confidence)
* - settings.json `model` is a Haiku model (MEDIUM confidence)
* - a per-server `.mcp.json` `alwaysLoad: true` (HIGH confidence)
*
* It deliberately does NOT read process.env shell variables. Tool search is also
* disabled on Vertex AI, with a custom ANTHROPIC_BASE_URL (non-first-party host),
* or after a runtime `/model` switch to Haiku but those are launch/runtime
* state, not config files, so a static scan cannot see them without becoming
* machine-dependent. They are disclosed (DEFERRAL_DISCLOSURE), never triggered.
*
* Mechanism verified 2026-06-23 against code.claude.com/docs:
* context-window.md (MCP tools deferred, ~120 tok), mcp.md#configure-tool-search
* + #exempt-a-server-from-deferral, costs.md (reduce MCP server overhead).
*
* The pure `assessMcpDeferral` takes already-parsed inputs so it is fully
* unit-testable without file IO. `assessMcpDeferralForRepo` is the thin IO
* wrapper shared by token-hotspots and feature-gap.
*/
import { resolve } from 'node:path';
import { readTextFile } from './file-discovery.mjs';
import { parseJson } from './yaml-parser.mjs';
import { readActiveMcpServers } from './active-config-reader.mjs';
// Aggregate forced-upfront schema cost (tokens) → severity ladder. MCP token
// estimates are base 500 + ~200/tool (active-config-reader estimateTokens), so
// these anchor on a couple of small servers (medium) vs a large one / several
// (high). Heuristic, not measured — disclosed in every finding.
export const FORCED_SCHEMA_TOKENS_MEDIUM = 1500;
export const FORCED_SCHEMA_TOKENS_HIGH = 5000;
// Appended to every CA-TOK-006 finding: the launch/runtime conditions a static
// config scan cannot see, so the user knows to check them manually.
export const DEFERRAL_DISCLOSURE =
'static config-file check: tool search is ALSO disabled (all MCP schemas forced ' +
'upfront) on Vertex AI, with a custom ANTHROPIC_BASE_URL (non-first-party host), ' +
'or after a runtime /model switch to a Haiku model — none visible to a static ' +
'scan, so verify at launch. Tool-level "anthropic/alwaysLoad" set server-side is ' +
'likewise invisible. Per-server alwaysLoad requires Claude Code v2.1.121+.';
/**
* Does a permissions.deny list disable the ToolSearch tool? Matches a bare
* "ToolSearch" tool name (with or without an argument suffix), same shape Claude
* Code uses for built-in tool denies.
*/
function denyListDisablesToolSearch(deny) {
if (!Array.isArray(deny)) return false;
return deny.some((entry) => {
if (typeof entry !== 'string') return false;
const tool = entry.replace(/\(.*\)$/, '').trim();
return tool === 'ToolSearch';
});
}
function isHaikuModel(model) {
return typeof model === 'string' && /haiku/i.test(model);
}
/**
* Map severity for the forced-upfront aggregate. High-confidence reasons scale
* with the token cost; medium-confidence reasons (inferred from a configured
* model) are capped at medium so the finding never overstates certainty.
*
* @param {number} aggregateTokens
* @param {'high'|'medium'} confidence
* @returns {'high'|'medium'|'low'}
*/
export function severityForForcedSchemas(aggregateTokens, confidence) {
const tok = typeof aggregateTokens === 'number' ? aggregateTokens : 0;
if (confidence === 'high') {
if (tok >= FORCED_SCHEMA_TOKENS_HIGH) return 'high';
if (tok >= FORCED_SCHEMA_TOKENS_MEDIUM) return 'medium';
return 'low';
}
// medium confidence: cap at medium
if (tok >= FORCED_SCHEMA_TOKENS_HIGH) return 'medium';
return 'low';
}
/**
* Assess whether MCP tool schemas are forced into the always-loaded prefix.
*
* @param {object} args
* @param {{ env?: object, permissions?: { deny?: string[] }, model?: string }} [args.settings]
* merged settings.json view (env block, permissions, model).
* @param {Array<{name:string, source?:string, enabled?:boolean, toolCount?:number,
* estimatedTokens?:number, alwaysLoad?:boolean}>} [args.mcpServers]
* @returns {{
* toolSearchDisabled: boolean, reason: string|null,
* confidence: 'high'|'medium'|null, thresholdMode: boolean,
* alwaysLoadServers: object[], affectedServers: object[],
* aggregateTokens: number, forcedUpfront: boolean,
* }}
*/
export function assessMcpDeferral({ settings = {}, mcpServers = [] } = {}) {
const s = settings || {};
const tsRaw = s.env && typeof s.env === 'object' ? s.env.ENABLE_TOOL_SEARCH : undefined;
const ts = String(tsRaw ?? '').trim().toLowerCase();
const denyTS = denyListDisablesToolSearch(s.permissions?.deny);
const haiku = isHaikuModel(s.model);
let toolSearchDisabled = false;
let reason = null;
let confidence = null;
let thresholdMode = false;
if (ts === 'false') {
toolSearchDisabled = true;
reason = 'enable-tool-search-false';
confidence = 'high';
} else if (denyTS) {
toolSearchDisabled = true;
reason = 'deny-tool-search';
confidence = 'high';
} else if (haiku) {
// Haiku lacks tool_reference support, so tool search cannot run even when
// ENABLE_TOOL_SEARCH=true. Medium confidence: the configured model can be
// switched at runtime (/model), which a static scan cannot observe.
toolSearchDisabled = true;
reason = 'haiku-model';
confidence = 'medium';
} else if (ts === 'true') {
toolSearchDisabled = false;
} else if (ts.startsWith('auto')) {
// Threshold mode (auto / auto:N): schemas load upfront only when they fit a
// percentage of the context window — not a clear always-load. Info, not a
// forced-upfront trigger.
thresholdMode = true;
}
const active = (Array.isArray(mcpServers) ? mcpServers : []).filter(
(m) => m && m.enabled !== false,
);
const alwaysLoadServers = active.filter((m) => m.alwaysLoad === true);
const affectedServers = toolSearchDisabled ? active : alwaysLoadServers;
const aggregateTokens = affectedServers.reduce(
(sum, m) => sum + (typeof m.estimatedTokens === 'number' ? m.estimatedTokens : 0),
0,
);
return {
toolSearchDisabled,
reason,
confidence,
thresholdMode,
alwaysLoadServers,
affectedServers,
aggregateTokens,
forcedUpfront: affectedServers.length > 0,
};
}
/**
* Merge project + local settings.json for the deferral check. Scoped to the
* audited path (NOT the user cascade) so the result stays deterministic and free
* of ambient HOME leakage mirrors Pattern G's project-local scoping. Local
* overrides project; env/permissions shallow-merge, deny lists concatenate.
*/
async function readMergedProjectSettings(repoPath) {
const merged = { env: {}, permissions: {}, model: undefined };
for (const rel of ['.claude/settings.json', '.claude/settings.local.json']) {
const content = await readTextFile(resolve(repoPath, rel));
if (!content) continue;
const parsed = parseJson(content);
if (!parsed || typeof parsed !== 'object') continue;
if (parsed.env && typeof parsed.env === 'object') Object.assign(merged.env, parsed.env);
if (parsed.permissions && typeof parsed.permissions === 'object') {
const deny = [
...(Array.isArray(merged.permissions.deny) ? merged.permissions.deny : []),
...(Array.isArray(parsed.permissions.deny) ? parsed.permissions.deny : []),
];
merged.permissions = { ...merged.permissions, ...parsed.permissions, deny };
}
if (typeof parsed.model === 'string') merged.model = parsed.model;
}
return merged;
}
/**
* IO wrapper: assess MCP deferral for a repo path. Scopes MCP servers to active
* project-local `.mcp.json` (plugin / ~/.claude.json servers are the manifest's
* concern). Pass `mcpServers` (e.g. an already-loaded activeConfig.mcpServers) to
* avoid a re-read; otherwise they are read via readActiveMcpServers.
*
* @param {string} repoPath
* @param {{ mcpServers?: object[] }} [opts]
*/
export async function assessMcpDeferralForRepo(repoPath, opts = {}) {
const all = Array.isArray(opts.mcpServers)
? opts.mcpServers
: await readActiveMcpServers(repoPath);
const projectLocal = all.filter((m) => m && m.enabled && m.source === '.mcp.json');
const settings = await readMergedProjectSettings(repoPath);
return assessMcpDeferral({ settings, mcpServers: projectLocal });
}

View file

@ -18,6 +18,9 @@ const VALID_SERVER_TYPES = new Set(['stdio', 'http', 'sse']);
// not a per-server .mcp.json field. Verified against code.claude.com/docs 2026-06-18.
const VALID_SERVER_FIELDS = new Set([
'type', 'command', 'args', 'env', 'url', 'headers', 'timeout',
// alwaysLoad: exempt a server from MCP tool-schema deferral (CC v2.1.121+).
// Verified against code.claude.com/docs/en/mcp.md#exempt-a-server-from-deferral 2026-06-23.
'alwaysLoad',
]);
// Match only bare ${IDENTIFIER} references. POSIX expansions like ${VAR%pattern}

View file

@ -28,6 +28,11 @@ import { finding, scannerResult } from './lib/output.mjs';
import { SEVERITY } from './lib/severity.mjs';
import { findImports, parseJson, parseFrontmatter } from './lib/yaml-parser.mjs';
import { estimateTokens, readActiveConfig, deriveLoadPattern } from './lib/active-config-reader.mjs';
import {
assessMcpDeferralForRepo,
severityForForcedSchemas,
DEFERRAL_DISCLOSURE,
} from './lib/mcp-deferral.mjs';
const SCANNER = 'TOK';
@ -580,6 +585,61 @@ export async function scan(targetPath, discovery) {
}));
}
// ── Pattern I: MCP tool-schema deferral (v5.10 B4, CA-TOK-006) ──
// By default MCP tool schemas are DEFERRED (names-only, ~120 tok); certain
// config signals force the FULL schemas into the always-loaded prefix every
// turn. Scope: project-local .mcp.json servers (mirrors Pattern G — plugin /
// global servers are the manifest's concern). Static config-file check only;
// runtime conditions (Vertex / ANTHROPIC_BASE_URL / runtime /model switch) are
// DISCLOSED (DEFERRAL_DISCLOSURE), not triggered, so the finding is deterministic.
const mcpForDeferral = (activeConfig && Array.isArray(activeConfig.mcpServers))
? activeConfig.mcpServers.filter(m => m && m.enabled && m.source === '.mcp.json')
: [];
if (mcpForDeferral.length > 0) {
const a = await assessMcpDeferralForRepo(targetPath, { mcpServers: activeConfig.mcpServers });
if (a.forcedUpfront) {
const conf = a.confidence || 'high';
const severity = severityForForcedSchemas(a.aggregateTokens, conf);
const names = a.affectedServers.map(m => m.name).join(', ');
const reasonText = {
'enable-tool-search-false': 'ENABLE_TOOL_SEARCH is set to "false" in settings',
'deny-tool-search': '"ToolSearch" is listed in permissions.deny',
'haiku-model': 'the configured model is a Haiku model (Haiku has no tool-search support)',
}[a.reason];
const description = a.toolSearchDisabled
? `Tool search is disabled (${reasonText}), so the full tool schemas of ` +
`${a.affectedServers.length} active project MCP server${a.affectedServers.length === 1 ? '' : 's'} ` +
`(~${a.aggregateTokens} tokens) load into the always-loaded prefix on every turn instead of ` +
'being deferred (tool names only, ~120 tokens total). Every schema token is re-sent each turn ' +
'whether or not a tool is used.'
: `${a.alwaysLoadServers.length} project MCP server${a.alwaysLoadServers.length === 1 ? '' : 's'} ` +
`marked alwaysLoad (${names}) load their full tool schemas (~${a.aggregateTokens} tokens) into ` +
'the always-loaded prefix on every turn regardless of tool search, instead of deferring them ' +
'(names only, ~120 tokens).';
const evidence =
`reason=${a.reason || 'alwaysLoad'}; confidence=${conf}; servers=${names}; ` +
`forced_schema_tokens~${a.aggregateTokens}${CALIBRATION_NOTE}. ${DEFERRAL_DISCLOSURE}`;
const recommendation = a.toolSearchDisabled
? 'Re-enable tool search so MCP schemas defer (names-only) by default: remove ' +
'ENABLE_TOOL_SEARCH="false" / the "ToolSearch" deny, or stop defaulting to a Haiku model. ' +
'Also disable unused servers via /mcp, and prefer CLI tools (gh / aws / gcloud) over MCP for ' +
'common operations — CLI adds zero context tokens until used.'
: 'Drop alwaysLoad on large-schema servers so they defer (names-only) until a tool is needed; ' +
'keep alwaysLoad only for small servers you call on most turns. Prefer CLI tools ' +
'(gh / aws / gcloud) over MCP for common operations.';
findings.push(finding({
scanner: SCANNER,
severity,
title: 'MCP tool schemas forced into the always-loaded prefix',
file: null,
evidence,
description,
recommendation,
category: 'token-efficiency',
}));
}
}
// ── Hotspots ranking ──
const hotspots = await buildHotspots(discovery, targetPath, activeConfig);