test(snapshots): make byte/snapshot tests hermetic + re-seed baseline
The COL collision-scanner and the CLAUDE.md cascade resolve ~/.claude from process.env.HOME (active-config-reader). Snapshot/byte CLIs were spawned with the developer's real HOME, so they picked up installed plugins/skills and the user CLAUDE.md — making the v5.0.0 + default-output snapshots machine- and time-dependent. They were seeded 2026-05-01 with COL=1 (a real ~/.claude skill collision) and drifted to COL=0 after the polyrepo split: 26 pre-existing failures unrelated to Batch 1. Fix (test-only, no production change): - tests/helpers/hermetic-home.mjs — empty temp HOME, mirroring the pattern collision.test.mjs already uses for the COL unit test. - 7 harnesses spawn CLIs (or call lint()) under the hermetic HOME, so output depends only on committed fixtures. Determinism verified across runs. - Re-seeded all snapshots under hermetic HOME via SEED_SNAPSHOT/UPDATE_SNAPSHOT (added a SEED guard to the frozen v5.0.0 byte tests). Snapshots now reflect the fixture alone (COL=0, fixture-only activeConfig counts). - Also re-seeded the unused env-aware snapshots (manifest/whats-active/ plugin-health), which had baked dozens of real ~/.claude skill/plugin names into the committed repo — privacy cleanup. Full suite: 812/812 green, stable across 3 runs.
This commit is contained in:
parent
4b94da0f11
commit
8216fb4175
20 changed files with 326 additions and 3386 deletions
|
|
@ -1,4 +1,4 @@
|
|||
{
|
||||
"kind": "text",
|
||||
"payload": "`[CML] CLAUDE.md Linter`: 0 finding(s) (0ms)\n `[SET] Settings Validator`: 0 finding(s) (0ms)\n `[HKV] Hook Validator`: 0 finding(s) (0ms)\n `[RUL] Rules Validator`: 0 finding(s) (0ms)\n `[MCP] MCP Config Validator`: 0 finding(s) (0ms)\n `[IMP] Import Resolver`: 0 finding(s) (0ms)\n `[CNF] Conflict Detector`: 0 finding(s) (0ms)\n `[GAP] Feature Gap Scanner`: 17 finding(s) (0ms)\n `[TOK] Token Hotspots`: 1 finding(s) (0ms)\n `[CPS] Cache-Prefix Stability`: 0 finding(s) (0ms)\n `[DIS] Disabled-In-Schema`: 1 finding(s) (0ms)\n `[COL] Plugin Skill Collision`: 1 finding(s) (0ms)\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n Configuration health\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n Health: A (97/100) — Healthy setup, only minor polish needed\n 9 areas reviewed\n\n Area scores\n ───────────\n `CLAUDE.md` ........... A (100) `Settings` ............ A (90)\n `Hooks` ............... A (100) `Rules` ............... A (100)\n `MCP` ................. A (100) `Imports` ............. A (100)\n `Conflicts` ........... A (100) `Token Efficiency` .... A (90)\n `Plugin Hygiene` ...... A (90)\n\n 17 ways you could get more out of Claude Code — see /config-audit feature-gap\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
"payload": "`[CML] CLAUDE.md Linter`: 1 finding(s) (0ms)\n `[SET] Settings Validator`: 0 finding(s) (0ms)\n `[HKV] Hook Validator`: 0 finding(s) (0ms)\n `[RUL] Rules Validator`: 0 finding(s) (0ms)\n `[MCP] MCP Config Validator`: 0 finding(s) (0ms)\n `[IMP] Import Resolver`: 0 finding(s) (0ms)\n `[CNF] Conflict Detector`: 0 finding(s) (0ms)\n `[GAP] Feature Gap Scanner`: 17 finding(s) (0ms)\n `[TOK] Token Hotspots`: 1 finding(s) (0ms)\n `[CPS] Cache-Prefix Stability`: 0 finding(s) (0ms)\n `[DIS] Disabled-In-Schema`: 1 finding(s) (0ms)\n `[COL] Plugin Skill Collision`: 0 finding(s) (0ms)\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n Configuration health\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n Health: A (97/100) — Healthy setup, only minor polish needed\n 9 areas reviewed\n\n Area scores\n ───────────\n `CLAUDE.md` ........... A (90) `Settings` ............ A (90)\n `Hooks` ............... A (100) `Rules` ............... A (100)\n `MCP` ................. A (100) `Imports` ............. A (100)\n `Conflicts` ........... A (100) `Token Efficiency` .... A (90)\n `Plugin Hygiene` ...... A (100)\n\n 17 ways you could get more out of Claude Code — see /config-audit feature-gap\n\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
}
|
||||
|
|
|
|||
|
|
@ -13,12 +13,29 @@
|
|||
"status": "ok",
|
||||
"files_scanned": 1,
|
||||
"duration_ms": 0,
|
||||
"findings": [],
|
||||
"findings": [
|
||||
{
|
||||
"id": "CA-CML-001",
|
||||
"scanner": "CML",
|
||||
"severity": "low",
|
||||
"title": "Your instructions file is missing common sections",
|
||||
"description": "Sections like Project Overview, Commands, and Conventions help Claude apply your guidance consistently across tasks.",
|
||||
"file": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/CLAUDE.md",
|
||||
"line": null,
|
||||
"evidence": "Present sections: Marketplace Medium, Plugins, Commands, Conventions",
|
||||
"category": null,
|
||||
"recommendation": "Add the missing sections noted in the details.",
|
||||
"autoFixable": false,
|
||||
"userImpactCategory": "Configuration mistake",
|
||||
"userActionLanguage": "Optional cleanup",
|
||||
"relevanceContext": "test-fixture-no-impact"
|
||||
}
|
||||
],
|
||||
"counts": {
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 0,
|
||||
"low": 1,
|
||||
"info": 0
|
||||
}
|
||||
},
|
||||
|
|
@ -432,69 +449,53 @@
|
|||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "mcp:sadhguru-wisdom (plugin:sadhguru-wisdom)",
|
||||
"estimated_tokens": 500,
|
||||
"rank": 2,
|
||||
"recommendations": [
|
||||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "mcp:vegnorm-rag (plugin:vegnormalene)",
|
||||
"estimated_tokens": 500,
|
||||
"rank": 3,
|
||||
"recommendations": [
|
||||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "CLAUDE.md",
|
||||
"estimated_tokens": 116,
|
||||
"rank": 4,
|
||||
"rank": 2,
|
||||
"recommendations": [
|
||||
"Move volatile top-of-file content to the bottom or extract to an @import-ed file.",
|
||||
"Split overlong CLAUDE.md into focused @imports (≤200 lines each)."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/CLAUDE.md"
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/CLAUDE.md"
|
||||
},
|
||||
{
|
||||
"source": "hooks/hooks.json",
|
||||
"estimated_tokens": 81,
|
||||
"rank": 3,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/hooks/hooks.json"
|
||||
},
|
||||
{
|
||||
"source": ".claude/settings.json",
|
||||
"estimated_tokens": 59,
|
||||
"rank": 4,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json"
|
||||
},
|
||||
{
|
||||
"source": ".mcp.json",
|
||||
"estimated_tokens": 53,
|
||||
"rank": 5,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/hooks/hooks.json"
|
||||
},
|
||||
{
|
||||
"source": ".claude/settings.json",
|
||||
"estimated_tokens": 59,
|
||||
"rank": 6,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json"
|
||||
},
|
||||
{
|
||||
"source": ".mcp.json",
|
||||
"estimated_tokens": 53,
|
||||
"rank": 7,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/.mcp.json"
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/.mcp.json"
|
||||
}
|
||||
],
|
||||
"total_estimated_tokens": 1809,
|
||||
"total_estimated_tokens": 809,
|
||||
"activeConfig": {
|
||||
"claudeMdEstimatedTokens": "<ANCESTOR_DERIVED>",
|
||||
"mcpServerCount": 3,
|
||||
"pluginCount": 41,
|
||||
"skillCount": 65
|
||||
"mcpServerCount": 1,
|
||||
"pluginCount": 0,
|
||||
"skillCount": 0
|
||||
}
|
||||
},
|
||||
{
|
||||
|
|
@ -523,7 +524,7 @@
|
|||
"severity": "low",
|
||||
"title": "A tool is in both the let-in list and the shut-out list",
|
||||
"description": "When a tool is in both lists, the shut-out always wins, so the let-in entry does nothing. It looks like the tool is approved, but it isn't.",
|
||||
"file": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json",
|
||||
"file": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json",
|
||||
"line": null,
|
||||
"evidence": "Read: allow=\"Read(src/**)\" + deny=\"Read(./.env)\"",
|
||||
"category": "permissions-hygiene",
|
||||
|
|
@ -545,45 +546,14 @@
|
|||
{
|
||||
"scanner": "COL",
|
||||
"status": "ok",
|
||||
"files_scanned": 65,
|
||||
"files_scanned": 0,
|
||||
"duration_ms": 0,
|
||||
"findings": [
|
||||
{
|
||||
"id": "CA-COL-001",
|
||||
"scanner": "COL",
|
||||
"severity": "low",
|
||||
"title": "Two plugins both define a skill with the same name",
|
||||
"description": "When two plugins offer the same skill name, only one wins, and which one is hard to predict.",
|
||||
"file": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/okr/skills/okr-offentlig-sektor/SKILL.md",
|
||||
"line": null,
|
||||
"evidence": "name=\"okr-offentlig-sektor\"; plugins=okr,okr",
|
||||
"category": "plugin-hygiene",
|
||||
"recommendation": "Rename the skill in one of the plugins, or disable the one you don't use.",
|
||||
"autoFixable": false,
|
||||
"userImpactCategory": "Conflict",
|
||||
"userActionLanguage": "Optional cleanup",
|
||||
"relevanceContext": "affects-everyone",
|
||||
"details": {
|
||||
"namespaces": [
|
||||
{
|
||||
"source": "plugin:okr",
|
||||
"name": "okr-offentlig-sektor",
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/okr/skills/okr-offentlig-sektor/SKILL.md"
|
||||
},
|
||||
{
|
||||
"source": "plugin:okr",
|
||||
"name": "okr-offentlig-sektor",
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-privat/plugins/okr/skills/okr-offentlig-sektor/SKILL.md"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"findings": [],
|
||||
"counts": {
|
||||
"critical": 0,
|
||||
"high": 0,
|
||||
"medium": 0,
|
||||
"low": 1,
|
||||
"low": 0,
|
||||
"info": 0
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
"status": "ok",
|
||||
"files_scanned": 2,
|
||||
"duration_ms": 0,
|
||||
"total_estimated_tokens": 1809,
|
||||
"total_estimated_tokens": 809,
|
||||
"hotspots": [
|
||||
{
|
||||
"source": "mcp:memory (.mcp.json)",
|
||||
|
|
@ -15,61 +15,45 @@
|
|||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "mcp:sadhguru-wisdom (plugin:sadhguru-wisdom)",
|
||||
"estimated_tokens": 500,
|
||||
"rank": 2,
|
||||
"recommendations": [
|
||||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "mcp:vegnorm-rag (plugin:vegnormalene)",
|
||||
"estimated_tokens": 500,
|
||||
"rank": 3,
|
||||
"recommendations": [
|
||||
"Review whether this source needs to load on every turn."
|
||||
]
|
||||
},
|
||||
{
|
||||
"source": "CLAUDE.md",
|
||||
"estimated_tokens": 116,
|
||||
"rank": 4,
|
||||
"rank": 2,
|
||||
"recommendations": [
|
||||
"Move volatile top-of-file content to the bottom or extract to an @import-ed file.",
|
||||
"Split overlong CLAUDE.md into focused @imports (≤200 lines each)."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/CLAUDE.md"
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/CLAUDE.md"
|
||||
},
|
||||
{
|
||||
"source": "hooks/hooks.json",
|
||||
"estimated_tokens": 81,
|
||||
"rank": 3,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/hooks/hooks.json"
|
||||
},
|
||||
{
|
||||
"source": ".claude/settings.json",
|
||||
"estimated_tokens": 59,
|
||||
"rank": 4,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json"
|
||||
},
|
||||
{
|
||||
"source": ".mcp.json",
|
||||
"estimated_tokens": 53,
|
||||
"rank": 5,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/hooks/hooks.json"
|
||||
},
|
||||
{
|
||||
"source": ".claude/settings.json",
|
||||
"estimated_tokens": 59,
|
||||
"rank": 6,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/.claude/settings.json"
|
||||
},
|
||||
{
|
||||
"source": ".mcp.json",
|
||||
"estimated_tokens": 53,
|
||||
"rank": 7,
|
||||
"recommendations": [
|
||||
"Deduplicate overlapping entries — each duplicate inflates the per-turn schema payload.",
|
||||
"Move rarely-used permissions to a project-local override."
|
||||
],
|
||||
"path": "/Users/ktg/.claude/plugins/marketplaces/ktg-plugin-marketplace/plugins/config-audit/tests/fixtures/marketplace-medium/.mcp.json"
|
||||
"path": "/Users/ktg/repos/ktg-plugin-marketplace/config-audit/tests/fixtures/marketplace-medium/.mcp.json"
|
||||
}
|
||||
],
|
||||
"findings": [
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue