open/ktg-plugin-marketplace
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions
ktg-plugin-marketplace/plugins/config-audit/docs/v5.1.0-test-audit.md
Kjell Tore Guttormsen 2397ffb5e4 chore(humanizer): pre-flight snapshots + test audit for v5.1.0 
Wave 0 / Step 0 of the v5.1.0 plain-language UX humanizer plan.

Captures v5.0.0 baseline output for all 8 CLIs at
tests/snapshots/v5.0.0/ — these snapshots are immutable references
for SC-6 (--json byte-equal) and SC-7 (--raw byte-equal) tests in
later waves.

- 5 CLIs captured via --output-file: scan-orchestrator, posture,
  token-hotspots-cli, manifest, whats-active
- 3 CLIs captured via stdout redirect (no --output-file support):
  drift-cli (after baseline seed), fix-cli, plugin-health-scanner
- Posture stderr scorecard captured separately for SC-7 stderr-mode
  comparison

docs/v5.1.0-test-audit.md classifies all 42 .title references in
7 known test files: 34 will break under humanization (literal
string equality / substring), 8 are safe (test fixtures or error
formatting). This document is the change list for Step 4.

Project: .claude/projects/2026-05-01-config-audit-ux-redesign/
2026-05-01 16:47:13 +02:00

8 KiB
Raw Blame History

v5.1.0 Title-String Assertion Audit

Generated by Wave 0 / Step 0 pre-flight on 2026-05-01.

This document is the authoritative change list for Step 4 (replace title-string assertions with ID-based or shape-based assertions). Step 5 cannot wire the humanizer until every "WILL BREAK" entry below is converted.

Classification key

  • (a) shape-only — checks existence, type, or test-fixture input; not affected by humanization.
  • (b) literal-string WILL BREAK — exact equality or substring match against scanner-produced title prose. Humanization rewrites these strings; the assertion must be re-anchored to finding.id, finding.scanner, or finding.evidence.
  • (c) ID-based — already anchored on finding.id or scanner prefix. No change needed.

Audit summary

Test file Matches Will break (b) Safe (a/c)
tests/lib/output.test.mjs 1 0 1
tests/scanners/feature-gap-scanner.test.mjs 6 6 0
tests/scanners/hook-validator.test.mjs 12 9 3
tests/lib/diff-engine.test.mjs 2 0 2
tests/scanners/fix-engine.test.mjs 1 0 1
tests/scanners/plugin-health-scanner.test.mjs 9 8 1
tests/scanners/settings-validator.test.mjs 11 11 0
Total 42 34 8

Per-file findings

tests/lib/output.test.mjs

Line Code Class Action
46 assert.strictEqual(f.title, 'Test') (a) shape-only None — 'Test' is the test's own input to finding() constructor, not a scanner-produced title.

tests/scanners/feature-gap-scanner.test.mjs

Line Code Class Action
45 f.title === 'No CLAUDE.md file' (b) WILL BREAK Replace with f.id === '<GAP-ID-for-no-CLAUDE.md>'. Anchor on ID.
49 f.title === 'No MCP servers configured' (b) WILL BREAK Replace with ID anchor.
53 f.title === 'No hooks configured' (b) WILL BREAK Replace with ID anchor.
96 f.title === 'No hooks configured' (b) WILL BREAK Replace with ID anchor.
100 f.title === 'No MCP servers configured' (b) WILL BREAK Replace with ID anchor.
150 f.title === 'No CLAUDE.md file' (b) WILL BREAK Replace with ID anchor.

Implementation note for Step 4: look up the actual GAP finding IDs via grep -n "title:" scanners/feature-gap-scanner.mjs and substitute. For shape only: assert.ok(f.id.startsWith('CA-GAP-')) is acceptable when the test only cares that a GAP finding fired.

tests/scanners/hook-validator.test.mjs

Line Code Class Action
30 serious.map(f => f.title).join(', ') (a) shape-only None — title used only for error-message formatting in failed assert; not the assertion itself.
49 f.title === 'Unknown hook event' (b) WILL BREAK Replace with ID anchor.
54 f.title.includes('Matcher must be a string') (b) WILL BREAK Replace with ID anchor or .evidence.includes(...).
59 f.title === 'Invalid hook handler type' (b) WILL BREAK Replace with ID anchor.
64 f.title.includes('timeout') (b) WILL BREAK Replace with ID anchor.
69 f.title === 'Unknown hook event' (b) WILL BREAK Replace with ID anchor.
80 /verbose hook output/i.test(x.title || '') (b) WILL BREAK Replace with ID anchor.
81 result.findings.map(x => x.title).join(' | ') (a) shape-only Used only in error-message formatting. None.
91 /verbose hook output/i.test(x.title || '') (b) WILL BREAK Replace with ID anchor.
92 f?.title (a) shape-only Used only in error-message formatting. None.

tests/lib/diff-engine.test.mjs

Line Code Class Action
66 diff.newFindings[0].title === 'New issue' (a) shape-only None — 'New issue' is the test's synthetic finding input, not scanner-produced.
78 diff.resolvedFindings[0].title === 'Old issue' (a) shape-only None — synthetic test input.

tests/scanners/fix-engine.test.mjs

Line Code Class Action
62 assert.ok(m.title, 'Manual finding should have title') (a) shape-only None — pure existence check.

tests/scanners/plugin-health-scanner.test.mjs

Line Code Class Action
52 f.title.includes('Missing required field') (b) WILL BREAK Replace with ID anchor or f.evidence.includes(...).
59 f.title.includes('missing') && f.title.includes('section') (b) WILL BREAK Replace with ID anchor on the missing-section finding.
68 f.title.includes('Missing required field') (b) WILL BREAK Replace with ID anchor.
75 f.title === 'Missing CLAUDE.md' (b) WILL BREAK Replace with ID anchor.
82 f.title === 'Command missing frontmatter' (b) WILL BREAK Replace with ID anchor.
90 f.title.startsWith('Agent missing frontmatter field:') (b) WILL BREAK Replace with ID anchor + f.evidence.includes(...) for the field name (humanizer preserves evidence).
93 missingAgent.map(f => f.title).join(', ') (a) shape-only Used only in error-message formatting. None.
102 result.findings[0].title === 'No plugins found' (b) WILL BREAK Replace with ID anchor.
125 assert.ok(f.title) (a) shape-only None — pure existence check.

tests/scanners/settings-validator.test.mjs

Line Code Class Action
49 f.title === 'Unknown settings key' (b) WILL BREAK Replace with ID anchor (likely CA-SET-001 or similar — verify).
54 f.title === 'Deprecated settings key' (b) WILL BREAK Replace with ID anchor.
59 f.title === 'Type mismatch in settings' (b) WILL BREAK Replace with ID anchor.
64 f.title === 'Invalid effortLevel value' (b) WILL BREAK Replace with ID anchor.
69 f.title.includes('array instead of object') (b) WILL BREAK Replace with ID anchor.
74 f.title.includes('array instead of object') (b) WILL BREAK Replace with ID anchor.
86 f.title === 'Unknown settings key' && /additionalDirectories/.test(f.evidence) (b) WILL BREAK Keep evidence regex; replace title check with ID anchor.
96 /additionalDirectories/i.test(x.title || '') (b) WILL BREAK Replace with ID anchor + evidence regex (additionalDirectories likely appears in evidence already).
98 f?.title (a) shape-only — but inside breaking assertion Will become moot after line 96 is fixed.
106 /additionalDirectories/i.test(x.title || '') (b) WILL BREAK Replace with ID anchor + evidence regex.
107 result.findings.map(x => x.title).join(' | ') (a) shape-only Error-message formatting only. None.

Step 4 implementation guidance

  1. For each (b) WILL BREAK row, look up the actual finding ID from the corresponding scanner source:
    • grep -n "id: 'CA-GAP-" scanners/feature-gap-scanner.mjs
    • grep -n "id: 'CA-HKV-" scanners/hook-validator.mjs
    • grep -n "id: 'CA-PLH-" scanners/plugin-health-scanner.mjs
    • grep -n "id: 'CA-SET-" scanners/settings-validator.mjs
  2. Replace the title check with f.id === '<exact-id>'. If the test cares about a sub-variant (e.g., a specific deprecated key), pair the ID anchor with an f.evidence.includes(...) substring check — humanizer preserves evidence exactly.
  3. For broad categorical checks ("any GAP finding fired"), use f.id.startsWith('CA-GAP-').
  4. For tests that capture f.title only inside assert failure-message templates (class (a)): leave them. Humanization changes the displayed string but the assertion still anchors on f.id.
  5. Re-run node --test 'tests/**/*.test.mjs' after changes; expect zero regressions before proceeding to Step 5.

Total scope for Step 4

  • 6 test files require code changes (output.test.mjs and diff-engine.test.mjs are clean).
  • 34 distinct assertions to convert.
  • Estimated effort: 1–2 hours including ID lookup and verification.