open/ktg-plugin-marketplace
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions

chore(humanizer): pre-flight snapshots + test audit for v5.1.0

Browse source 
Wave 0 / Step 0 of the v5.1.0 plain-language UX humanizer plan.

Captures v5.0.0 baseline output for all 8 CLIs at
tests/snapshots/v5.0.0/ — these snapshots are immutable references
for SC-6 (--json byte-equal) and SC-7 (--raw byte-equal) tests in
later waves.

- 5 CLIs captured via --output-file: scan-orchestrator, posture,
  token-hotspots-cli, manifest, whats-active
- 3 CLIs captured via stdout redirect (no --output-file support):
  drift-cli (after baseline seed), fix-cli, plugin-health-scanner
- Posture stderr scorecard captured separately for SC-7 stderr-mode
  comparison

docs/v5.1.0-test-audit.md classifies all 42 .title references in
7 known test files: 34 will break under humanization (literal
string equality / substring), 8 are safe (test fixtures or error
formatting). This document is the change list for Step 4.

Project: .claude/projects/2026-05-01-config-audit-ux-redesign/
This commit is contained in:
Kjell Tore Guttormsen 2026-05-01 16:47:13 +02:00
commit 2397ffb5e4
10 changed files with 5134 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

121
plugins/config-audit/docs/v5.1.0-test-audit.md Normal file
View file

@ -0,0 +1,121 @@
# v5.1.0 Title-String Assertion Audit
Generated by Wave 0 / Step 0 pre-flight on 2026-05-01.
This document is the authoritative change list for **Step 4** (replace title-string assertions with ID-based or shape-based assertions). Step 5 cannot wire the humanizer until every "WILL BREAK" entry below is converted.
## Classification key
- **(a) shape-only** — checks existence, type, or test-fixture input; not affected by humanization.
- **(b) literal-string WILL BREAK** — exact equality or substring match against scanner-produced title prose. Humanization rewrites these strings; the assertion must be re-anchored to `finding.id`, `finding.scanner`, or `finding.evidence`.
- **(c) ID-based** — already anchored on `finding.id` or scanner prefix. No change needed.
## Audit summary
| Test file | Matches | Will break (b) | Safe (a/c) |
|-----------|---------|----------------|------------|
| `tests/lib/output.test.mjs` | 1 | 0 | 1 |
| `tests/scanners/feature-gap-scanner.test.mjs` | 6 | 6 | 0 |
| `tests/scanners/hook-validator.test.mjs` | 12 | 9 | 3 |
| `tests/lib/diff-engine.test.mjs` | 2 | 0 | 2 |
| `tests/scanners/fix-engine.test.mjs` | 1 | 0 | 1 |
| `tests/scanners/plugin-health-scanner.test.mjs` | 9 | 8 | 1 |
| `tests/scanners/settings-validator.test.mjs` | 11 | 11 | 0 |
| **Total** | **42** | **34** | **8** |
## Per-file findings
### `tests/lib/output.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 46 | `assert.strictEqual(f.title, 'Test')` | (a) shape-only | None — `'Test'` is the test's own input to `finding()` constructor, not a scanner-produced title. |
### `tests/scanners/feature-gap-scanner.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 45 | `f.title === 'No CLAUDE.md file'` | (b) WILL BREAK | Replace with `f.id === '<GAP-ID-for-no-CLAUDE.md>'`. Anchor on ID. |
| 49 | `f.title === 'No MCP servers configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 53 | `f.title === 'No hooks configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 96 | `f.title === 'No hooks configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 100 | `f.title === 'No MCP servers configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 150 | `f.title === 'No CLAUDE.md file'` | (b) WILL BREAK | Replace with ID anchor. |
> **Implementation note for Step 4:** look up the actual GAP finding IDs via `grep -n "title:" scanners/feature-gap-scanner.mjs` and substitute. For shape only: `assert.ok(f.id.startsWith('CA-GAP-'))` is acceptable when the test only cares that a GAP finding fired.
### `tests/scanners/hook-validator.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 30 | `serious.map(f => f.title).join(', ')` | (a) shape-only | None — title used only for error-message formatting in failed assert; not the assertion itself. |
| 49 | `f.title === 'Unknown hook event'` | (b) WILL BREAK | Replace with ID anchor. |
| 54 | `f.title.includes('Matcher must be a string')` | (b) WILL BREAK | Replace with ID anchor or `.evidence.includes(...)`. |
| 59 | `f.title === 'Invalid hook handler type'` | (b) WILL BREAK | Replace with ID anchor. |
| 64 | `f.title.includes('timeout')` | (b) WILL BREAK | Replace with ID anchor. |
| 69 | `f.title === 'Unknown hook event'` | (b) WILL BREAK | Replace with ID anchor. |
| 80 | `/verbose hook output/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor. |
| 81 | `result.findings.map(x => x.title).join(' \| ')` | (a) shape-only | Used only in error-message formatting. None. |
| 91 | `/verbose hook output/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor. |
| 92 | `f?.title` | (a) shape-only | Used only in error-message formatting. None. |
### `tests/lib/diff-engine.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 66 | `diff.newFindings[0].title === 'New issue'` | (a) shape-only | None — `'New issue'` is the test's synthetic finding input, not scanner-produced. |
| 78 | `diff.resolvedFindings[0].title === 'Old issue'` | (a) shape-only | None — synthetic test input. |
### `tests/scanners/fix-engine.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 62 | `assert.ok(m.title, 'Manual finding should have title')` | (a) shape-only | None — pure existence check. |
### `tests/scanners/plugin-health-scanner.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 52 | `f.title.includes('Missing required field')` | (b) WILL BREAK | Replace with ID anchor or `f.evidence.includes(...)`. |
| 59 | `f.title.includes('missing') && f.title.includes('section')` | (b) WILL BREAK | Replace with ID anchor on the missing-section finding. |
| 68 | `f.title.includes('Missing required field')` | (b) WILL BREAK | Replace with ID anchor. |
| 75 | `f.title === 'Missing CLAUDE.md'` | (b) WILL BREAK | Replace with ID anchor. |
| 82 | `f.title === 'Command missing frontmatter'` | (b) WILL BREAK | Replace with ID anchor. |
| 90 | `f.title.startsWith('Agent missing frontmatter field:')` | (b) WILL BREAK | Replace with ID anchor + `f.evidence.includes(...)` for the field name (humanizer preserves evidence). |
| 93 | `missingAgent.map(f => f.title).join(', ')` | (a) shape-only | Used only in error-message formatting. None. |
| 102 | `result.findings[0].title === 'No plugins found'` | (b) WILL BREAK | Replace with ID anchor. |
| 125 | `assert.ok(f.title)` | (a) shape-only | None — pure existence check. |
### `tests/scanners/settings-validator.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 49 | `f.title === 'Unknown settings key'` | (b) WILL BREAK | Replace with ID anchor (likely `CA-SET-001` or similar — verify). |
| 54 | `f.title === 'Deprecated settings key'` | (b) WILL BREAK | Replace with ID anchor. |
| 59 | `f.title === 'Type mismatch in settings'` | (b) WILL BREAK | Replace with ID anchor. |
| 64 | `f.title === 'Invalid effortLevel value'` | (b) WILL BREAK | Replace with ID anchor. |
| 69 | `f.title.includes('array instead of object')` | (b) WILL BREAK | Replace with ID anchor. |
| 74 | `f.title.includes('array instead of object')` | (b) WILL BREAK | Replace with ID anchor. |
| 86 | `f.title === 'Unknown settings key' && /additionalDirectories/.test(f.evidence)` | (b) WILL BREAK | Keep evidence regex; replace title check with ID anchor. |
| 96 | `/additionalDirectories/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor + evidence regex (additionalDirectories likely appears in evidence already). |
| 98 | `f?.title` | (a) shape-only — but inside breaking assertion | Will become moot after line 96 is fixed. |
| 106 | `/additionalDirectories/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor + evidence regex. |
| 107 | `result.findings.map(x => x.title).join(' \| ')` | (a) shape-only | Error-message formatting only. None. |
## Step 4 implementation guidance
1. For each (b) WILL BREAK row, look up the actual finding ID from the corresponding scanner source:
- `grep -n "id: 'CA-GAP-" scanners/feature-gap-scanner.mjs`
- `grep -n "id: 'CA-HKV-" scanners/hook-validator.mjs`
- `grep -n "id: 'CA-PLH-" scanners/plugin-health-scanner.mjs`
- `grep -n "id: 'CA-SET-" scanners/settings-validator.mjs`
2. Replace the title check with `f.id === '<exact-id>'`. If the test cares about a sub-variant (e.g., a specific deprecated key), pair the ID anchor with an `f.evidence.includes(...)` substring check — humanizer preserves `evidence` exactly.
3. For broad categorical checks ("any GAP finding fired"), use `f.id.startsWith('CA-GAP-')`.
4. For tests that capture `f.title` only inside `assert` failure-message templates (class (a)): leave them. Humanization changes the displayed string but the assertion still anchors on `f.id`.
5. Re-run `node --test 'tests/**/*.test.mjs'` after changes; expect zero regressions before proceeding to Step 5.
## Total scope for Step 4
- **6 test files** require code changes (`output.test.mjs` and `diff-engine.test.mjs` are clean).
- **34 distinct assertions** to convert.
- Estimated effort: 1–2 hours including ID lookup and verification.