Commit graph

27 commits

Author SHA1 Message Date
Kjell Tore Guttormsen
0420b8cc4a feat(config-audit): /config-audit manifest command (v5 N2) [skip-docs]
New scanners/manifest.mjs CLI + commands/manifest.md slash command.
Reads activeConfig and produces a flat, ranked list of every token
source (CLAUDE.md cascade entries, plugins, skills, MCP servers, hooks)
sorted DESC by estimated_tokens.

CLAUDE.md per-file tokens are derived by distributing
claudeMd.estimatedTokens across the cascade proportional to bytes.

Tests cover both real-config (plugin root) and fixture (rich-repo with
patched HOME containing 2 plugins + 3 skills + .mcp.json) paths, plus
error handling (nonexistent path → exit 3, --output-file).

Builds on readActiveConfig from M1 (v5 alpha.2).

[skip-docs] reason: v5 plan fences off README/CLAUDE.md badge updates
to Session 5; Forgejo pre-commit-docs-gate hook requires this tag on
feat commits without doc changes.

Tests: 593 → 604 (+11).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 07:32:54 +02:00
Kjell Tore Guttormsen
b2407a09b3 feat(config-audit): CA-TOK-005 MCP tool-schema budget (v5 N1) [skip-docs]
Adds detectMcpToolBudget detection block in TOK scanner. Tiered severity
per project-local .mcp.json server based on toolCount:
- < 20: no finding
- 20-49: low
- 50-99: medium
- 100+: high
- null (manifest unparseable): low + "tool count unknown" message

Scoped to source==='.mcp.json' to keep findings actionable for the
audited path; plugin/user-level MCP servers are surfaced by the
manifest scanner (Step 19 / N2).

5 fixtures (mcp-budget/{14,25,60,120,unknown}-tools) use inline `tools`
arrays in .mcp.json — no node_modules needed for these tests.

Tests assert title+severity (not exact ID) since TOK IDs are sequential
per scan, not semantic per pattern.

[skip-docs] reason: v5 plan fences off README/CLAUDE.md badge updates
to Session 5; Forgejo pre-commit-docs-gate hook requires this tag on
feat commits without doc changes.

Tests: 586 → 593 (+7).

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
2026-05-01 07:29:57 +02:00
Kjell Tore Guttormsen
3c79f95e9a feat(config-audit): self-audit --check-readme flag (v5 F6) [skip-docs]
Filesystem counts are the source of truth; README badges parsed via
line-anchored substring (badge/<kind>-<N>-...). Emits readmeCheck object
with counts/badges/mismatches.

CLI: node scanners/self-audit.mjs --check-readme [--json]
API: runSelfAudit({ checkReadme: true }) → result.readmeCheck
Helper: checkReadmeBadges(pluginDir) for per-fixture testing

New fixture: readme-desynced/ (commands/foo + bar, README claims 1).

Note: alpha phase does NOT require result.readmeCheck.passed === true.
Self-test of real plugin currently fails (scanners 10 vs 9, tests 31 vs 543);
will be reconciled in Session 5 Step 28 (README sync).

582 → 586 tests, all green.
2026-05-01 07:09:26 +02:00
Kjell Tore Guttormsen
910567d661 feat(config-audit): HKV flags verbose hook output (v5 M5) [skip-docs]
Static heuristic — counts console.log / process.stdout.write lines per
referenced hook script. > 50 → low CA-HKV-NNN finding.

New fixtures:
- hooks-verbose/ (61 verbose lines → triggers)
- hooks-quiet/ (5 lines → no finding)

580 → 582 tests, all green.
2026-05-01 07:05:45 +02:00
Kjell Tore Guttormsen
7181862644 chore(config-audit): allow fake node_modules in tests/fixtures (v5 M1) [skip-docs]
The mcp-tool-heavy fixture relies on node_modules/mcp-heavy/package.json
being committed so the v5 M1 tool-count detection test runs deterministically.
Add an unignore rule for tests/fixtures/**/node_modules/.
2026-05-01 07:02:54 +02:00
Kjell Tore Guttormsen
1422daf895 feat(config-audit): MCP tool-count detection with manifest fallback (v5 M1) [skip-docs]
readActiveMcpServers now resolves tool count via:
  1. In-config tools array
  2. Cached tools/list at \$HOME/.claude/config-audit/mcp-cache/<name>.json
  3. node_modules/<pkg>/package.json (resolved from npx <pkg>)
  4. Fallback: { toolCount: null, toolCountUnknown: true }

estimateTokens uses detected toolCount (heavy server > light server).

New fixture: mcp-tool-heavy/ with mocked node_modules/mcp-heavy/package.json (20 tools).

576 → 580 tests, all green.
2026-05-01 07:02:08 +02:00
Kjell Tore Guttormsen
9a44df22ac feat(config-audit): TOK flags skill description > 500 chars (v5 M2) [skip-docs]
- New Pattern F in TOK: low-severity finding when SKILL.md description > 500 chars
- Scoped to discovery.files (project-local) — activeConfig.skills walk would
  pull in user/plugin skills out of project scope
- New fixtures: skill-bloated (594-char desc) + skill-tight (46-char baseline)

574 → 576 tests, all green.
2026-05-01 06:58:42 +02:00
Kjell Tore Guttormsen
25ca6139b4 feat(config-audit): TOK flags CLAUDE.md cascade > 10k tokens (v5 M4) [skip-docs]
- New Pattern E in TOK: emits medium finding when activeConfig.claudeMd.estimatedTokens > 10_000
- Uses cascade tokens, file count, and calibration note as evidence
- New fixtures: large-cascade (37k bytes / 14475 cascade tokens) + small-cascade (5k baseline)

572 → 574 tests, all green.
2026-05-01 06:53:12 +02:00
Kjell Tore Guttormsen
9330124f5c feat(config-audit): flag additionalDirectories > 2 (v5 M6) [skip-docs]
- Add 'additionalDirectories' to KNOWN_KEYS
- Emit low severity finding when length > 2
- New fixtures: additional-dirs-many (3 entries) + additional-dirs-ok (2)

569 → 572 tests, all green.
2026-05-01 06:50:24 +02:00
Kjell Tore Guttormsen
58d6b5b9ea feat(config-audit): recalibrate TOK severities for tokens/turn (v5 F7) [skip-docs]
- Pattern A (cache-breaking volatile top): medium → high
- Pattern B (redundant permissions): low → medium
- Pattern C (deep @import chain): medium → low
- Add calibration_note evidence on every TOK finding
- Table-driven severity tests (identify by title, IDs are sequential)

563 → 569 tests, all green. Doc sweep deferred to Session 5 (Step 28).
2026-05-01 06:47:32 +02:00
Kjell Tore Guttormsen
2810ee6f62 feat(config-audit): remove TOK Pattern D detectSonnetEra (v5 F5)
Pattern D was the v4 sonnet-era signature: 'config is structurally
clean but uses no Opus-4.7-specific features'. Two problems:
- It triggered on any minimal config that happened to lack skills/MCP
- The advice was generic and not actionable

The hotspots ranking and per-pattern findings (A/B/C) cover the same
ground with concrete, file-anchored signal. Dropping the noise.

BREAKING (intentional): scanners no longer emit the sonnet-era info
finding. Suppression entries and downstream tooling that reference
the v4 finding ID should be updated. Doc sweep follows in Step 8b.

Tests: sonnet-era fixture now asserts zero findings.
2026-05-01 06:31:43 +02:00
Kjell Tore Guttormsen
0d8a9af3d6 fix(config-audit): remove TOK dead take + hotspot padding (v5 F4)
The buildHotspots padding loop and unused 'take' variable were dead
code from the v3 hotspots-min contract. Replaced with a clean
ranked.slice(0, HOTSPOTS_MAX). Tiny fixtures may now return fewer
than 3 hotspots, which is the honest answer; the contract now only
asserts <= 10.

Tests: +2 cases — every hotspot.source is unique (no padding); length
never exceeds HOTSPOTS_MAX.
2026-05-01 06:29:33 +02:00
Kjell Tore Guttormsen
34669d596c feat(config-audit): TOK consumes readActiveConfig (v5 F1)
Removes the v4 'void readActiveConfig' placeholder and wires the
active-config snapshot into the TOK scanner.

Per-turn behavior changes:
- Each enabled MCP server becomes its own hotspot entry (richer than
  the parent .mcp.json file alone)
- total_estimated_tokens now includes MCP server cost
- result.activeConfig exposes a small summary
  (claudeMdEstimatedTokens, mcpServerCount, pluginCount, skillCount)

Failures of readActiveConfig are non-fatal — the scanner falls back
to the discovery-only path used in v4.

Tests: +3 cases on the new tok-active-config fixture
(.mcp.json with 2 servers, CLAUDE.md, plugin skeleton).
2026-05-01 06:27:34 +02:00
Kjell Tore Guttormsen
ce7c42f517 fix(config-audit): MCP token callers use 'mcp' kind (v5 F2)
Two MCP enumeration paths in readActiveMcpServers now pass kind='mcp'
to estimateTokens with optional toolCount derived from def.tools array
(populated when callers cache MCP discovery — Step 14 wires that up).

Hook callers keep kind='item' (no schema overhead).

Visible effect: every active MCP server jumps from estimatedTokens=15
to >= 500 (or higher when toolCount is known). The whats-active output
and TOK hotspots now reflect actual MCP cost.

Tests: assert mcpServers[].estimatedTokens >= 500 in fixture.
2026-05-01 06:22:54 +02:00
Kjell Tore Guttormsen
48d560a209 feat(config-audit): add 'mcp' kind to estimateTokens (v5 F2)
Differentiate MCP servers from generic 'item' (flat 15) — they actually
cost 500+ tokens per turn for protocol metadata and tool schemas.

estimateTokens(bytes, 'mcp', {toolCount}) returns max of:
- 500 token floor (base overhead)
- ceil(bytes / 3.5) (json-rate when bytes known)
- 500 + toolCount * 200 (when tool count is detected; Step 14 wires this)

Caller-side migration in next commit (Step 5).

Tests: +4 cases for mcp kind.
2026-05-01 06:21:30 +02:00
Kjell Tore Guttormsen
a65c7f4080 feat(config-audit): severity-weighted scoreByArea (v5 F3)
Replace count-based pass-rate with severity-weighted penalty:
- penalty = sum(count[s] * WEIGHTS[s])
- maxBudget = max(10, findingCount * 4)
- passRate = max(0, 100 - penalty / maxBudget * 100)

A few lows no longer crater an area's grade; a single high or critical
consumes a large fraction of budget. Mirrors the operator intuition that
severity, not count, is the signal.

BREAKING (intentional): scoring semantics differ from v4 for non-clean
configs. Add scoringVersion: 'v5' to the returned struct so consumers
can detect the version. baseline-all-a remains all-A (no critical/high
on that fixture).

Tests: +6 cases for severity weighting; existing "many findings" test
updated to use highs (where v5 still drops the grade as expected).
2026-05-01 06:20:08 +02:00
Kjell Tore Guttormsen
e5efc2ff64 feat(config-audit): export WEIGHTS from severity.mjs (v5 F3 prep)
Promote WEIGHTS const to named export with Object.freeze for downstream
use in scoring.mjs (severity-weighted scoreByArea, F3).

Tests: +2 cases asserting WEIGHTS shape.
2026-05-01 06:16:28 +02:00
Kjell Tore Guttormsen
cbc889f603 feat(config-audit): add token-hotspots CLI (node scanners/token-hotspots-cli.mjs) 2026-04-19 22:46:25 +02:00
Kjell Tore Guttormsen
295a6289b4 test(config-audit): extend grade-stability test to assert Token Efficiency A/B on baseline 2026-04-19 22:45:34 +02:00
Kjell Tore Guttormsen
4b385bf456 feat(config-audit): wire TOK into posture scorecard as 8th quality area (Token Efficiency) 2026-04-19 22:45:12 +02:00
Kjell Tore Guttormsen
712058c387 test(config-audit): add token-hotspots scanner test suite (red) 2026-04-19 22:40:44 +02:00
Kjell Tore Guttormsen
5a4f29fd14 test(config-audit): add marketplace-small/medium/large scanner fixtures 2026-04-19 22:36:33 +02:00
Kjell Tore Guttormsen
94ce70186c test(config-audit): add Opus 4.7 pattern fixtures (cache, redundant, imports, sonnet-era) 2026-04-19 22:34:41 +02:00
Kjell Tore Guttormsen
350cebc39c test(config-audit): add baseline-all-a fixture + grade-stability regression test 2026-04-19 22:32:40 +02:00
Kjell Tore Guttormsen
a9fb328584 fix(config-audit): complete conflict-project fixture for CNF cross-scope tests 2026-04-19 22:29:46 +02:00
Kjell Tore Guttormsen
4f1cc7e0b7 feat(config-audit): v3.1.0 — /config-audit whats-active inventory command
New read-only command that shows everything Claude Code actually loads for a
given repo — plugins, skills, MCP servers, hooks, CLAUDE.md cascade — with
source attribution (user/project/plugin) and rough token estimates. Helps
identify candidates for disabling without guessing.

Added:
- scanners/lib/active-config-reader.mjs — pure async helper: readActiveConfig,
  detectGitRoot, walkClaudeMdCascade, readClaudeJsonProjectSlice (longest-prefix
  matching for .claude.json projects), enumeratePlugins, enumerateSkills,
  readActiveHooks, readActiveMcpServers, estimateTokens (markdown 4 c/tok,
  json 3.5 c/tok, frontmatter cap 150 tokens, item flat 15)
- scanners/whats-active.mjs — thin CLI shim: --json, --output-file, --verbose,
  --suggest-disables
- commands/whats-active.md — renders tables via Read tool; honors UX rules
- tests/lib/active-config-reader.test.mjs — 36 tests, all green (integration
  fixture built in tmpdir with fake HOME, .claude.json prefix matching,
  plugin discovery, hook/MCP merge from all scopes)

Verified:
- Performance budget: <2s wall-clock (smoke test: 102ms on real repo)
- Token estimates within ±20% of hand-computed values
- Read-only: no writeFile/mkdir/unlink in production code
- Self-audit: Plugin Health scanner reports 0 findings (Grade A)
- Full test suite: 522 tests, 512 pass (10 pre-existing conflict-detector
  failures on main — unrelated to this change, reproducible on clean HEAD)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-04-14 21:50:20 +02:00
Kjell Tore Guttormsen
f93d6abdae feat: initial open marketplace with llm-security, config-audit, ultraplan-local 2026-04-06 18:47:49 +02:00