feat(scanners): model/effort routing becomes a lever, not a 25th dimension (C4)
New GAP finding CA-GAP-028: authored subagents exist and not one of them names `model:` or `effort:`, so every delegated task runs on the main conversation's model (`model` defaults to `inherit`). Cites BP-MODEL-001/002, landed in C1. `whats-active` and `manifest` now carry `model`/`effort` per agent. Shipped as a conditional LEVER rather than a 25th dimension, and the choice was made by measurement: as a t3 dimension the agent-less marketplace-medium fixture would count it vacuously-present, moving the denominators 41->42 and utilization 44->45 — which flips `segment` "Developing"->"Competent" in the frozen v5.0.0 posture baseline, a field strip-retired-gap.mjs does not mask. A lever never enters those denominators. The general rule is now an invariant in CLAUDE.md. One check across both axes, not one per axis: it fires only when neither is used anywhere, so a deliberate everything-on-one-model policy stays silent. Cost is recall, chosen for precision. Found by dogfooding, fixed red-first: `model: inherit` is the documented default spelled out, so it must not count as routing — otherwise a config opts out of the opportunity without changing anything real. Two pre-existing defects surfaced and closed on the way: - The humanizer guard asserted TRANSLATIONS.GAP.static EQUALS the dimension titles, which forbade humanizing any lever — all three existing levers fell through to the generic "feature opportunity" default, wrong for a budget lever. Guard now requires coverage of every emittable title, seen red against those three before the entries were written. - Two hand-written copies of the lever list (finding-codes guard, humanizer guard) merged into one exported LEVERS registry carrying code AND title. - suppression-validation pinned CA-GAP-028 as an unoccupied number; C4 claimed it. Fixed structurally with a derived first-free id, not by picking a new literal — same class as #60's "bump this again". Suite 1596/0. Frozen v5.0.0 snapshots untouched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pq3nye21RVYk4pZLeT8pGz
This commit is contained in:
parent
e861e63a7b
commit
9ae4be26d2
16 changed files with 512 additions and 31 deletions
|
|
@ -1,9 +1,10 @@
|
|||
/**
|
||||
* GAP Scanner — Feature Gap Scanner
|
||||
* Compares actual configuration against complete Claude Code feature register.
|
||||
* 25 gap dimensions across 4 tiers, plus a conditional disableBundledSkills
|
||||
* budget-lever check (remediation companion to SKL CA-SKL-002, fires only under
|
||||
* measured skill-listing pressure). Always runs with includeGlobal: true.
|
||||
* 24 gap dimensions across 4 tiers, plus four conditional levers (bundled-skills
|
||||
* budget, CLI-over-MCP, hook-output filtering, agent model/effort routing) which
|
||||
* fire only under a measured condition and are therefore NOT dimensions: they
|
||||
* stay out of the scoring denominators. Always runs with includeGlobal: true.
|
||||
* Finding IDs: CA-GAP-NNN
|
||||
*/
|
||||
|
||||
|
|
@ -115,6 +116,36 @@ const TIER_SEVERITY = {
|
|||
t4: SEVERITY.info,
|
||||
};
|
||||
|
||||
/**
|
||||
* Titles of the conditional levers — findings this scanner emits that are NOT
|
||||
* dimensions in GAP_CHECKS. They fire only under a measured condition, so they
|
||||
* carry no tier and never enter the scoring denominators (TIER_COUNTS /
|
||||
* TOTAL_DIMENSIONS) or the scoring TITLE_TO_ID map.
|
||||
*
|
||||
* Exported as the single source of both the code and the title: the
|
||||
* finding-code registry guard needs the codes, the humanizer coverage guard
|
||||
* needs the titles, and a hand-maintained copy of either list in a test is the
|
||||
* two-copies-drift class. One object so the two cannot disagree.
|
||||
*/
|
||||
export const LEVERS = {
|
||||
bundledSkills: {
|
||||
code: 'bundled-skills-lever',
|
||||
title: 'Bundled skills add to an over-budget skill listing',
|
||||
},
|
||||
cliOverMcp: {
|
||||
code: 'cli-over-mcp-lever',
|
||||
title: 'Prefer CLI over MCP for common operations',
|
||||
},
|
||||
filterHookOutput: {
|
||||
code: 'filter-hook-output-lever',
|
||||
title: 'Filter hook output before it enters context',
|
||||
},
|
||||
agentModelRouting: {
|
||||
code: 'agent-model-routing-lever',
|
||||
title: 'Subagents pin neither model nor effort',
|
||||
},
|
||||
};
|
||||
|
||||
/**
|
||||
* Lazily read and cache file content.
|
||||
* @param {CheckContext} ctx
|
||||
|
|
@ -177,8 +208,8 @@ export function bundledSkillsLeverFinding({ leverPulled, aggregate }) {
|
|||
return finding({
|
||||
scanner: SCANNER,
|
||||
severity: SEVERITY.low,
|
||||
code: 'bundled-skills-lever',
|
||||
title: 'Bundled skills add to an over-budget skill listing',
|
||||
code: LEVERS.bundledSkills.code,
|
||||
title: LEVERS.bundledSkills.title,
|
||||
description:
|
||||
`Your ${aggregate.scanned} active skills already carry ~${aggregate.aggregateTokens} tokens of ` +
|
||||
`description text, over the ${aggregate.budgetTokens}-token listing budget Claude Code allots the ` +
|
||||
|
|
@ -222,8 +253,8 @@ export function cliOverMcpLeverFinding({ assessment } = {}) {
|
|||
return finding({
|
||||
scanner: SCANNER,
|
||||
severity: SEVERITY.low,
|
||||
code: 'cli-over-mcp-lever',
|
||||
title: 'Prefer CLI over MCP for common operations',
|
||||
code: LEVERS.cliOverMcp.code,
|
||||
title: LEVERS.cliOverMcp.title,
|
||||
description:
|
||||
`Your active project MCP tool schemas (~${assessment.aggregateTokens} tokens) are forced into the ` +
|
||||
'always-loaded prefix every turn rather than deferred (see CA-TOK-006). CLI tools (gh, aws, gcloud, …) ' +
|
||||
|
|
@ -260,8 +291,8 @@ export function filterHookLeverFinding({ flaggedHooks } = {}) {
|
|||
return finding({
|
||||
scanner: SCANNER,
|
||||
severity: SEVERITY.info,
|
||||
code: 'filter-hook-output-lever',
|
||||
title: 'Filter hook output before it enters context',
|
||||
code: LEVERS.filterHookOutput.code,
|
||||
title: LEVERS.filterHookOutput.title,
|
||||
description:
|
||||
`${hooks.length} active hook${hooks.length === 1 ? '' : 's'} build hookSpecificOutput.additionalContext ` +
|
||||
"from un-grepped command output (see HKV advisory). That field enters Claude's context on every fire, " +
|
||||
|
|
@ -276,6 +307,101 @@ export function filterHookLeverFinding({ flaggedHooks } = {}) {
|
|||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Agent model/effort routing lever (C4) — cites BP-MODEL-001/002.
|
||||
*
|
||||
* A LEVER rather than a GAP_CHECKS dimension, and deliberately so. The question
|
||||
* "do your subagents route model/effort?" has no meaningful reading on a config
|
||||
* with no subagents — the `No custom subagents` dimension (t2_6) owns that case,
|
||||
* and firing here too would just double-report it. A dimension can only express
|
||||
* "not applicable" as "present", which would also inflate the utilization
|
||||
* denominator for every agent-less config.
|
||||
*
|
||||
* ONE check across BOTH axes, not two: it fires only when NEITHER `model:` nor
|
||||
* `effort:` appears on ANY authored agent. A deliberate all-on-one-model setup
|
||||
* therefore stays silent, which is the precision the opportunity framing needs.
|
||||
* The cost is recall — a config that pins `model:` everywhere but never uses
|
||||
* `effort:` gets no nudge. That trade is the v1 boundary, not an oversight.
|
||||
*
|
||||
* Pure and exported for unit testing.
|
||||
*
|
||||
* @param {{ agentCount: number, modelPinned: number, effortPinned: number }} counts
|
||||
* @returns {object|null} a GAP finding, or null when there is no opportunity
|
||||
*/
|
||||
export function agentModelRoutingLeverFinding({ agentCount, modelPinned, effortPinned }) {
|
||||
if (!agentCount) return null;
|
||||
if (modelPinned > 0 || effortPinned > 0) return null;
|
||||
|
||||
return finding({
|
||||
scanner: SCANNER,
|
||||
severity: SEVERITY.info,
|
||||
code: LEVERS.agentModelRouting.code,
|
||||
title: LEVERS.agentModelRouting.title,
|
||||
description:
|
||||
`All ${agentCount} of your subagents name neither a \`model:\` nor an \`effort:\` in their frontmatter. ` +
|
||||
'The `model` field defaults to `inherit`, so each one runs on the main conversation\'s model — omitting ' +
|
||||
'it is not a neutral default but a choice to pay the session\'s rate for every delegated task ' +
|
||||
'(BP-MODEL-001, https://code.claude.com/docs/en/sub-agents). Reasoning effort is a separate axis with ' +
|
||||
'its own frontmatter field and its own default, so a subagent can be routed on either or both ' +
|
||||
'(BP-MODEL-002, https://code.claude.com/docs/en/model-config).',
|
||||
evidence:
|
||||
`authored_agents=${agentCount}; model_pinned=${modelPinned}; effort_pinned=${effortPinned}; ` +
|
||||
'lever=agent frontmatter `model:` / `effort:` (plugin-bundled and fixture agents excluded)',
|
||||
recommendation:
|
||||
'Pin a cheaper `model:` on the subagents whose work is mechanical or read-only (search, extraction, ' +
|
||||
'summarisation) and leave the orchestrating session on the stronger model; pin a lower `effort:` on the ' +
|
||||
'same ones and reserve the high levels for work whose product is judgement. If running everything on one ' +
|
||||
'model is a deliberate policy, suppress this with `CA-GAP-028` in `.config-audit-ignore`.',
|
||||
category: 'model-fit',
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Count authored agents and how many pin each routing axis.
|
||||
* Frontmatter-only read; an unparseable or frontmatter-less file counts as an
|
||||
* agent that pins nothing, matching what Claude Code would load.
|
||||
* @param {CheckContext} ctx
|
||||
* @returns {Promise<{ agentCount: number, modelPinned: number, effortPinned: number }>}
|
||||
*/
|
||||
async function countAgentRouting(ctx) {
|
||||
let agentCount = 0;
|
||||
let modelPinned = 0;
|
||||
let effortPinned = 0;
|
||||
for (const file of ctx.files.filter(f => f.type === 'agent-md')) {
|
||||
agentCount++;
|
||||
const content = await getContent(ctx, file.absPath);
|
||||
if (!content) continue;
|
||||
const { frontmatter } = parseFrontmatter(content);
|
||||
if (!frontmatter) continue;
|
||||
if (isRoutingValue(frontmatter.model) && !isDefaultModel(frontmatter.model)) modelPinned++;
|
||||
if (isRoutingValue(frontmatter.effort)) effortPinned++;
|
||||
}
|
||||
return { agentCount, modelPinned, effortPinned };
|
||||
}
|
||||
|
||||
/**
|
||||
* True for a frontmatter value that actually names something. An empty or
|
||||
* whitespace-only `model:` is a no-op in Claude Code, so it must not read as a pin.
|
||||
* @param {*} v
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isRoutingValue(v) {
|
||||
return typeof v === 'string' ? v.trim().length > 0 : v != null && v !== false;
|
||||
}
|
||||
|
||||
/**
|
||||
* `inherit` IS the documented default for a subagent's `model` (BP-MODEL-001),
|
||||
* so writing it explicitly routes nothing — the agent still runs on the main
|
||||
* conversation's model. Spelling out a default must not buy silence, or a config
|
||||
* can opt out of the opportunity without changing a single thing about cost.
|
||||
* Effort has no documented sentinel of this kind, so it has no counterpart here.
|
||||
* @param {*} v
|
||||
* @returns {boolean}
|
||||
*/
|
||||
function isDefaultModel(v) {
|
||||
return typeof v === 'string' && v.trim().toLowerCase() === 'inherit';
|
||||
}
|
||||
|
||||
/** @type {GapCheck[]} */
|
||||
export const GAP_CHECKS = [
|
||||
// --- Tier 1: Foundation ---
|
||||
|
|
@ -608,6 +734,13 @@ export async function scan(targetPath, sharedDiscovery) {
|
|||
const hookLever = filterHookLeverFinding({ flaggedHooks });
|
||||
if (hookLever) findings.push(hookLever);
|
||||
|
||||
// Agent model/effort routing lever (C4) — fires only when authored agents
|
||||
// exist and not one of them uses either routing axis. Reads the SAME authored
|
||||
// set as the presence checks, so plugin-bundled and fixture agents cannot
|
||||
// make a machine look routed (M-BUG-13).
|
||||
const routingLever = agentModelRoutingLeverFinding(await countAgentRouting(ctx));
|
||||
if (routingLever) findings.push(routingLever);
|
||||
|
||||
const filesScanned = discovery.files.length;
|
||||
return scannerResult(SCANNER, 'ok', findings, filesScanned, Date.now() - start);
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue