feat(tokens): MCP tool-schema deferral check + CLI-over-MCP lever (v5.10 B4)
By default Claude Code defers MCP tool schemas (names-only, ~120 tok; full
schemas on demand via tool search). CA-TOK-006 detects config-file signals that
force the FULL schemas into the always-loaded prefix every turn:
- settings.json env.ENABLE_TOOL_SEARCH="false" (high)
- "ToolSearch" in permissions.deny (high)
- configured model is a Haiku model (medium)
- per-server .mcp.json alwaysLoad:true (CC v2.1.121+) (high)
auto[:N] is threshold mode (info, not a trigger).
New engine lib/mcp-deferral.mjs: pure assessMcpDeferral({settings,mcpServers})
(unit-tested, no IO) + thin IO wrapper assessMcpDeferralForRepo shared by TOK and
GAP. Severity scales with aggregate forced-upfront tokens (medium-confidence
reasons cap at medium). feature-gap cliOverMcpLeverFinding fires only as a
companion to CA-TOK-006 (prefer gh/aws/gcloud over MCP for common ops).
Honest scoping (Verifiseringsplikt): triggers on config files ONLY — never
process.env shell vars. Vertex / custom ANTHROPIC_BASE_URL / runtime /model
switch are launch state (would flap snapshots machine-dependently), so they are
DISCLOSED in every finding, not triggered. Tool-level anthropic/alwaysLoad and
claude.ai connectors likewise disclosed. Mechanism verified 2026-06-23 against
code.claude.com/docs (context-window.md, mcp.md#configure-tool-search +
#exempt-a-server-from-deferral, costs.md); the prefix-cache invalidation claim
was NOT-CONFIRMED in docs and is not asserted.
alwaysLoad added to CA-MCP VALID_SERVER_FIELDS (no longer flagged as unknown).
active-config-reader surfaces per-server alwaysLoad. Byte-stable: CA-TOK-006
fires only on new conditions; frozen v5.0.0 + SC-5 snapshots untouched.
Tests 1215 -> 1239 (engine 16, integration 5, lever 2, mcp-field guard 1).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
parent
42e48514e4
commit
8f7e196046
13 changed files with 692 additions and 2 deletions
|
|
@ -788,6 +788,7 @@ export async function readActiveMcpServers(repoPath, claudeJsonSlice = null, plu
|
|||
toolCount,
|
||||
toolCountUnknown: detected.toolCountUnknown,
|
||||
estimatedTokens: estimateTokens(0, 'mcp', { toolCount: toolCount ?? 0 }),
|
||||
alwaysLoad: def?.alwaysLoad === true,
|
||||
});
|
||||
}
|
||||
|
||||
|
|
@ -817,6 +818,7 @@ async function collectMcpFromFile(path, source, disabled, out, repoPath) {
|
|||
toolCount,
|
||||
toolCountUnknown: detected.toolCountUnknown,
|
||||
estimatedTokens: estimateTokens(0, 'mcp', { toolCount: toolCount ?? 0 }),
|
||||
alwaysLoad: def?.alwaysLoad === true,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
|
|
|||
206
scanners/lib/mcp-deferral.mjs
Normal file
206
scanners/lib/mcp-deferral.mjs
Normal file
|
|
@ -0,0 +1,206 @@
|
|||
/**
|
||||
* MCP tool-schema deferral assessment (v5.10 B4).
|
||||
*
|
||||
* By default Claude Code DEFERS MCP tool schemas: only tool *names* enter the
|
||||
* always-loaded prefix (~120 tokens total) and full schemas load on demand via
|
||||
* tool search. Certain conditions force ALL full schemas into the always-loaded
|
||||
* prefix instead — paid on every turn whether or not a tool is used.
|
||||
*
|
||||
* This module is a STATIC, CONFIG-FILE assessment. It triggers only on signals
|
||||
* that live in config files a scanner can read deterministically:
|
||||
* - settings.json `env.ENABLE_TOOL_SEARCH` = "false" (HIGH confidence)
|
||||
* - settings.json `permissions.deny` contains "ToolSearch" (HIGH confidence)
|
||||
* - settings.json `model` is a Haiku model (MEDIUM confidence)
|
||||
* - a per-server `.mcp.json` `alwaysLoad: true` (HIGH confidence)
|
||||
*
|
||||
* It deliberately does NOT read process.env shell variables. Tool search is also
|
||||
* disabled on Vertex AI, with a custom ANTHROPIC_BASE_URL (non-first-party host),
|
||||
* or after a runtime `/model` switch to Haiku — but those are launch/runtime
|
||||
* state, not config files, so a static scan cannot see them without becoming
|
||||
* machine-dependent. They are disclosed (DEFERRAL_DISCLOSURE), never triggered.
|
||||
*
|
||||
* Mechanism verified 2026-06-23 against code.claude.com/docs:
|
||||
* context-window.md (MCP tools deferred, ~120 tok), mcp.md#configure-tool-search
|
||||
* + #exempt-a-server-from-deferral, costs.md (reduce MCP server overhead).
|
||||
*
|
||||
* The pure `assessMcpDeferral` takes already-parsed inputs so it is fully
|
||||
* unit-testable without file IO. `assessMcpDeferralForRepo` is the thin IO
|
||||
* wrapper shared by token-hotspots and feature-gap.
|
||||
*/
|
||||
|
||||
import { resolve } from 'node:path';
|
||||
import { readTextFile } from './file-discovery.mjs';
|
||||
import { parseJson } from './yaml-parser.mjs';
|
||||
import { readActiveMcpServers } from './active-config-reader.mjs';
|
||||
|
||||
// Aggregate forced-upfront schema cost (tokens) → severity ladder. MCP token
|
||||
// estimates are base 500 + ~200/tool (active-config-reader estimateTokens), so
|
||||
// these anchor on a couple of small servers (medium) vs a large one / several
|
||||
// (high). Heuristic, not measured — disclosed in every finding.
|
||||
export const FORCED_SCHEMA_TOKENS_MEDIUM = 1500;
|
||||
export const FORCED_SCHEMA_TOKENS_HIGH = 5000;
|
||||
|
||||
// Appended to every CA-TOK-006 finding: the launch/runtime conditions a static
|
||||
// config scan cannot see, so the user knows to check them manually.
|
||||
export const DEFERRAL_DISCLOSURE =
|
||||
'static config-file check: tool search is ALSO disabled (all MCP schemas forced ' +
|
||||
'upfront) on Vertex AI, with a custom ANTHROPIC_BASE_URL (non-first-party host), ' +
|
||||
'or after a runtime /model switch to a Haiku model — none visible to a static ' +
|
||||
'scan, so verify at launch. Tool-level "anthropic/alwaysLoad" set server-side is ' +
|
||||
'likewise invisible. Per-server alwaysLoad requires Claude Code v2.1.121+.';
|
||||
|
||||
/**
|
||||
* Does a permissions.deny list disable the ToolSearch tool? Matches a bare
|
||||
* "ToolSearch" tool name (with or without an argument suffix), same shape Claude
|
||||
* Code uses for built-in tool denies.
|
||||
*/
|
||||
function denyListDisablesToolSearch(deny) {
|
||||
if (!Array.isArray(deny)) return false;
|
||||
return deny.some((entry) => {
|
||||
if (typeof entry !== 'string') return false;
|
||||
const tool = entry.replace(/\(.*\)$/, '').trim();
|
||||
return tool === 'ToolSearch';
|
||||
});
|
||||
}
|
||||
|
||||
function isHaikuModel(model) {
|
||||
return typeof model === 'string' && /haiku/i.test(model);
|
||||
}
|
||||
|
||||
/**
|
||||
* Map severity for the forced-upfront aggregate. High-confidence reasons scale
|
||||
* with the token cost; medium-confidence reasons (inferred from a configured
|
||||
* model) are capped at medium so the finding never overstates certainty.
|
||||
*
|
||||
* @param {number} aggregateTokens
|
||||
* @param {'high'|'medium'} confidence
|
||||
* @returns {'high'|'medium'|'low'}
|
||||
*/
|
||||
export function severityForForcedSchemas(aggregateTokens, confidence) {
|
||||
const tok = typeof aggregateTokens === 'number' ? aggregateTokens : 0;
|
||||
if (confidence === 'high') {
|
||||
if (tok >= FORCED_SCHEMA_TOKENS_HIGH) return 'high';
|
||||
if (tok >= FORCED_SCHEMA_TOKENS_MEDIUM) return 'medium';
|
||||
return 'low';
|
||||
}
|
||||
// medium confidence: cap at medium
|
||||
if (tok >= FORCED_SCHEMA_TOKENS_HIGH) return 'medium';
|
||||
return 'low';
|
||||
}
|
||||
|
||||
/**
|
||||
* Assess whether MCP tool schemas are forced into the always-loaded prefix.
|
||||
*
|
||||
* @param {object} args
|
||||
* @param {{ env?: object, permissions?: { deny?: string[] }, model?: string }} [args.settings]
|
||||
* merged settings.json view (env block, permissions, model).
|
||||
* @param {Array<{name:string, source?:string, enabled?:boolean, toolCount?:number,
|
||||
* estimatedTokens?:number, alwaysLoad?:boolean}>} [args.mcpServers]
|
||||
* @returns {{
|
||||
* toolSearchDisabled: boolean, reason: string|null,
|
||||
* confidence: 'high'|'medium'|null, thresholdMode: boolean,
|
||||
* alwaysLoadServers: object[], affectedServers: object[],
|
||||
* aggregateTokens: number, forcedUpfront: boolean,
|
||||
* }}
|
||||
*/
|
||||
export function assessMcpDeferral({ settings = {}, mcpServers = [] } = {}) {
|
||||
const s = settings || {};
|
||||
const tsRaw = s.env && typeof s.env === 'object' ? s.env.ENABLE_TOOL_SEARCH : undefined;
|
||||
const ts = String(tsRaw ?? '').trim().toLowerCase();
|
||||
const denyTS = denyListDisablesToolSearch(s.permissions?.deny);
|
||||
const haiku = isHaikuModel(s.model);
|
||||
|
||||
let toolSearchDisabled = false;
|
||||
let reason = null;
|
||||
let confidence = null;
|
||||
let thresholdMode = false;
|
||||
|
||||
if (ts === 'false') {
|
||||
toolSearchDisabled = true;
|
||||
reason = 'enable-tool-search-false';
|
||||
confidence = 'high';
|
||||
} else if (denyTS) {
|
||||
toolSearchDisabled = true;
|
||||
reason = 'deny-tool-search';
|
||||
confidence = 'high';
|
||||
} else if (haiku) {
|
||||
// Haiku lacks tool_reference support, so tool search cannot run even when
|
||||
// ENABLE_TOOL_SEARCH=true. Medium confidence: the configured model can be
|
||||
// switched at runtime (/model), which a static scan cannot observe.
|
||||
toolSearchDisabled = true;
|
||||
reason = 'haiku-model';
|
||||
confidence = 'medium';
|
||||
} else if (ts === 'true') {
|
||||
toolSearchDisabled = false;
|
||||
} else if (ts.startsWith('auto')) {
|
||||
// Threshold mode (auto / auto:N): schemas load upfront only when they fit a
|
||||
// percentage of the context window — not a clear always-load. Info, not a
|
||||
// forced-upfront trigger.
|
||||
thresholdMode = true;
|
||||
}
|
||||
|
||||
const active = (Array.isArray(mcpServers) ? mcpServers : []).filter(
|
||||
(m) => m && m.enabled !== false,
|
||||
);
|
||||
const alwaysLoadServers = active.filter((m) => m.alwaysLoad === true);
|
||||
const affectedServers = toolSearchDisabled ? active : alwaysLoadServers;
|
||||
const aggregateTokens = affectedServers.reduce(
|
||||
(sum, m) => sum + (typeof m.estimatedTokens === 'number' ? m.estimatedTokens : 0),
|
||||
0,
|
||||
);
|
||||
|
||||
return {
|
||||
toolSearchDisabled,
|
||||
reason,
|
||||
confidence,
|
||||
thresholdMode,
|
||||
alwaysLoadServers,
|
||||
affectedServers,
|
||||
aggregateTokens,
|
||||
forcedUpfront: affectedServers.length > 0,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge project + local settings.json for the deferral check. Scoped to the
|
||||
* audited path (NOT the user cascade) so the result stays deterministic and free
|
||||
* of ambient HOME leakage — mirrors Pattern G's project-local scoping. Local
|
||||
* overrides project; env/permissions shallow-merge, deny lists concatenate.
|
||||
*/
|
||||
async function readMergedProjectSettings(repoPath) {
|
||||
const merged = { env: {}, permissions: {}, model: undefined };
|
||||
for (const rel of ['.claude/settings.json', '.claude/settings.local.json']) {
|
||||
const content = await readTextFile(resolve(repoPath, rel));
|
||||
if (!content) continue;
|
||||
const parsed = parseJson(content);
|
||||
if (!parsed || typeof parsed !== 'object') continue;
|
||||
if (parsed.env && typeof parsed.env === 'object') Object.assign(merged.env, parsed.env);
|
||||
if (parsed.permissions && typeof parsed.permissions === 'object') {
|
||||
const deny = [
|
||||
...(Array.isArray(merged.permissions.deny) ? merged.permissions.deny : []),
|
||||
...(Array.isArray(parsed.permissions.deny) ? parsed.permissions.deny : []),
|
||||
];
|
||||
merged.permissions = { ...merged.permissions, ...parsed.permissions, deny };
|
||||
}
|
||||
if (typeof parsed.model === 'string') merged.model = parsed.model;
|
||||
}
|
||||
return merged;
|
||||
}
|
||||
|
||||
/**
|
||||
* IO wrapper: assess MCP deferral for a repo path. Scopes MCP servers to active
|
||||
* project-local `.mcp.json` (plugin / ~/.claude.json servers are the manifest's
|
||||
* concern). Pass `mcpServers` (e.g. an already-loaded activeConfig.mcpServers) to
|
||||
* avoid a re-read; otherwise they are read via readActiveMcpServers.
|
||||
*
|
||||
* @param {string} repoPath
|
||||
* @param {{ mcpServers?: object[] }} [opts]
|
||||
*/
|
||||
export async function assessMcpDeferralForRepo(repoPath, opts = {}) {
|
||||
const all = Array.isArray(opts.mcpServers)
|
||||
? opts.mcpServers
|
||||
: await readActiveMcpServers(repoPath);
|
||||
const projectLocal = all.filter((m) => m && m.enabled && m.source === '.mcp.json');
|
||||
const settings = await readMergedProjectSettings(repoPath);
|
||||
return assessMcpDeferral({ settings, mcpServers: projectLocal });
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue