Commit graph

12 commits

Author SHA1 Message Date
Kjell Tore Guttormsen
b7414303de feat(config-audit): --accurate-tokens API calibration (v5 N5) [skip-docs] 2026-05-01 09:15:02 +02:00
Kjell Tore Guttormsen
cd25c1e934 feat(config-audit): cross-plugin collision scanner COL (v5 N6) [skip-docs]
New COL scanner detects skill-name collisions across plugins and
between user-level skills (~/.claude/skills/) and plugin-bundled
skills. Skill identity is the directory basename — matches how
enumerateSkills resolves names.

Detection rules (per docs/v5-namespace-research.md, confidence: medium):
- Plugin-vs-plugin same skill name → severity low (CA-COL-001)
- User-vs-plugin same skill name → severity medium (CA-COL-001)
- Plugin-vs-built-in collisions: out of scope for v5.0.0 (insufficient
  verification — recorded for v5.0.1 follow-up).

Findings carry details.namespaces array with {source, name, path} for
every conflicting source — supports per-collision reporting downstream.

output.mjs: finding() helper now passes through optional `details`
field (scanner-specific structured payload).

scoring.mjs: COL → "Plugin Hygiene" (new area, 10 total). Posture test
updated from 9 → 10 area scores.

.gitignore: docs/v5-namespace-research.md is local-only (Step 22a
research output, gitignored per plan).

Fixture collision-plugins/fake-home/ has user skill `review` colliding
with plugin-a + plugin-b's `review` (medium severity), plus plugin-c's
unique `summarize` (no collision).

[skip-docs] reason: v5 plan fences off README/CLAUDE.md badge updates
to Session 5; Forgejo pre-commit-docs-gate hook requires this tag.

Tests: 617 → 625 (+8).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 07:46:15 +02:00
Kjell Tore Guttormsen
cc349d6fe1 feat(config-audit): disabled-in-schema scanner DIS (v5 N4) [skip-docs]
New DIS scanner detects tools that appear in BOTH permissions.deny
and permissions.allow within the same settings.json file. The deny
list wins, so allow entries are dead config but still load on every
turn and confuse intent.

Tool identity = bare name (everything before "("). `Bash(npm:*)` and
`Bash` are treated as the same tool, so a deny on `Bash` flags any
`Bash(...)` allow entry.

Severity: low. Wired into scan-orchestrator + scoring (area: Settings).
Fixture denied-tools-in-schema has Bash in both arrays; healthy-project
serves as the negative case.

[skip-docs] reason: v5 plan fences off README/CLAUDE.md badge updates
to Session 5; Forgejo pre-commit-docs-gate hook requires this tag.

Tests: 611 → 617 (+6).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 07:39:58 +02:00
Kjell Tore Guttormsen
65087e624f feat(config-audit): cache-prefix stability scanner CPS (v5 N3) [skip-docs]
New CPS scanner walks CLAUDE.md cascade and flags volatile content
between lines 31 and 150 — the cache-prefix window beyond TOK Pattern
A's top-30 territory. Volatile content anywhere in the cached prefix
forces a fresh cache write from that line down on every turn.

Volatile-pattern set extends TOK Pattern A with:
- shell-exec lines (! prefix) — common in CLAUDE.md to inject git/date
- ${VAR} substitutions — vary per-shell, defeat cache reuse

Severity: medium per finding. Skips lines 1-30 to avoid duplicating
Pattern A's range; CPS' value is in the 31-150 zone.

Wired into scan-orchestrator + scoring SCANNER_AREA_MAP. CPS shares
the "Token Efficiency" area with TOK; scoreByArea now deduplicates by
area name and combines counts across scanners contributing to the
same area, so the 9-area scorecard contract holds.

Fixtures volatile-mid-section/{volatile-line-60, volatile-line-200}
verify both positive (line 60) and out-of-window (line 200) cases.

[skip-docs] reason: v5 plan fences off README/CLAUDE.md badge updates
to Session 5; Forgejo pre-commit-docs-gate hook requires this tag.

Tests: 604 → 611 (+7).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 07:37:54 +02:00
Kjell Tore Guttormsen
1422daf895 feat(config-audit): MCP tool-count detection with manifest fallback (v5 M1) [skip-docs]
readActiveMcpServers now resolves tool count via:
  1. In-config tools array
  2. Cached tools/list at \$HOME/.claude/config-audit/mcp-cache/<name>.json
  3. node_modules/<pkg>/package.json (resolved from npx <pkg>)
  4. Fallback: { toolCount: null, toolCountUnknown: true }

estimateTokens uses detected toolCount (heavy server > light server).

New fixture: mcp-tool-heavy/ with mocked node_modules/mcp-heavy/package.json (20 tools).

576 → 580 tests, all green.
2026-05-01 07:02:08 +02:00
Kjell Tore Guttormsen
ce7c42f517 fix(config-audit): MCP token callers use 'mcp' kind (v5 F2)
Two MCP enumeration paths in readActiveMcpServers now pass kind='mcp'
to estimateTokens with optional toolCount derived from def.tools array
(populated when callers cache MCP discovery — Step 14 wires that up).

Hook callers keep kind='item' (no schema overhead).

Visible effect: every active MCP server jumps from estimatedTokens=15
to >= 500 (or higher when toolCount is known). The whats-active output
and TOK hotspots now reflect actual MCP cost.

Tests: assert mcpServers[].estimatedTokens >= 500 in fixture.
2026-05-01 06:22:54 +02:00
Kjell Tore Guttormsen
48d560a209 feat(config-audit): add 'mcp' kind to estimateTokens (v5 F2)
Differentiate MCP servers from generic 'item' (flat 15) — they actually
cost 500+ tokens per turn for protocol metadata and tool schemas.

estimateTokens(bytes, 'mcp', {toolCount}) returns max of:
- 500 token floor (base overhead)
- ceil(bytes / 3.5) (json-rate when bytes known)
- 500 + toolCount * 200 (when tool count is detected; Step 14 wires this)

Caller-side migration in next commit (Step 5).

Tests: +4 cases for mcp kind.
2026-05-01 06:21:30 +02:00
Kjell Tore Guttormsen
a65c7f4080 feat(config-audit): severity-weighted scoreByArea (v5 F3)
Replace count-based pass-rate with severity-weighted penalty:
- penalty = sum(count[s] * WEIGHTS[s])
- maxBudget = max(10, findingCount * 4)
- passRate = max(0, 100 - penalty / maxBudget * 100)

A few lows no longer crater an area's grade; a single high or critical
consumes a large fraction of budget. Mirrors the operator intuition that
severity, not count, is the signal.

BREAKING (intentional): scoring semantics differ from v4 for non-clean
configs. Add scoringVersion: 'v5' to the returned struct so consumers
can detect the version. baseline-all-a remains all-A (no critical/high
on that fixture).

Tests: +6 cases for severity weighting; existing "many findings" test
updated to use highs (where v5 still drops the grade as expected).
2026-05-01 06:20:08 +02:00
Kjell Tore Guttormsen
e5efc2ff64 feat(config-audit): export WEIGHTS from severity.mjs (v5 F3 prep)
Promote WEIGHTS const to named export with Object.freeze for downstream
use in scoring.mjs (severity-weighted scoreByArea, F3).

Tests: +2 cases asserting WEIGHTS shape.
2026-05-01 06:16:28 +02:00
Kjell Tore Guttormsen
4b385bf456 feat(config-audit): wire TOK into posture scorecard as 8th quality area (Token Efficiency) 2026-04-19 22:45:12 +02:00
Kjell Tore Guttormsen
4f1cc7e0b7 feat(config-audit): v3.1.0 — /config-audit whats-active inventory command
New read-only command that shows everything Claude Code actually loads for a
given repo — plugins, skills, MCP servers, hooks, CLAUDE.md cascade — with
source attribution (user/project/plugin) and rough token estimates. Helps
identify candidates for disabling without guessing.

Added:
- scanners/lib/active-config-reader.mjs — pure async helper: readActiveConfig,
  detectGitRoot, walkClaudeMdCascade, readClaudeJsonProjectSlice (longest-prefix
  matching for .claude.json projects), enumeratePlugins, enumerateSkills,
  readActiveHooks, readActiveMcpServers, estimateTokens (markdown 4 c/tok,
  json 3.5 c/tok, frontmatter cap 150 tokens, item flat 15)
- scanners/whats-active.mjs — thin CLI shim: --json, --output-file, --verbose,
  --suggest-disables
- commands/whats-active.md — renders tables via Read tool; honors UX rules
- tests/lib/active-config-reader.test.mjs — 36 tests, all green (integration
  fixture built in tmpdir with fake HOME, .claude.json prefix matching,
  plugin discovery, hook/MCP merge from all scopes)

Verified:
- Performance budget: <2s wall-clock (smoke test: 102ms on real repo)
- Token estimates within ±20% of hand-computed values
- Read-only: no writeFile/mkdir/unlink in production code
- Self-audit: Plugin Health scanner reports 0 findings (Grade A)
- Full test suite: 522 tests, 512 pass (10 pre-existing conflict-detector
  failures on main — unrelated to this change, reproducible on clean HEAD)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 21:50:20 +02:00
Kjell Tore Guttormsen
f93d6abdae feat: initial open marketplace with llm-security, config-audit, ultraplan-local 2026-04-06 18:47:49 +02:00