open/ktg-plugin-marketplace
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity Actions
ktg-plugin-marketplace/plugins/config-audit/docs/v5.1.0-test-audit.md
Kjell Tore Guttormsen 2397ffb5e4 chore(humanizer): pre-flight snapshots + test audit for v5.1.0 
Wave 0 / Step 0 of the v5.1.0 plain-language UX humanizer plan.

Captures v5.0.0 baseline output for all 8 CLIs at
tests/snapshots/v5.0.0/ — these snapshots are immutable references
for SC-6 (--json byte-equal) and SC-7 (--raw byte-equal) tests in
later waves.

- 5 CLIs captured via --output-file: scan-orchestrator, posture,
  token-hotspots-cli, manifest, whats-active
- 3 CLIs captured via stdout redirect (no --output-file support):
  drift-cli (after baseline seed), fix-cli, plugin-health-scanner
- Posture stderr scorecard captured separately for SC-7 stderr-mode
  comparison

docs/v5.1.0-test-audit.md classifies all 42 .title references in
7 known test files: 34 will break under humanization (literal
string equality / substring), 8 are safe (test fixtures or error
formatting). This document is the change list for Step 4.

Project: .claude/projects/2026-05-01-config-audit-ux-redesign/
2026-05-01 16:47:13 +02:00

121 lines
8 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# v5.1.0 Title-String Assertion Audit
Generated by Wave 0 / Step 0 pre-flight on 2026-05-01.
This document is the authoritative change list for **Step 4** (replace title-string assertions with ID-based or shape-based assertions). Step 5 cannot wire the humanizer until every "WILL BREAK" entry below is converted.
## Classification key
- **(a) shape-only** — checks existence, type, or test-fixture input; not affected by humanization.
- **(b) literal-string WILL BREAK** — exact equality or substring match against scanner-produced title prose. Humanization rewrites these strings; the assertion must be re-anchored to `finding.id`, `finding.scanner`, or `finding.evidence`.
- **(c) ID-based** — already anchored on `finding.id` or scanner prefix. No change needed.
## Audit summary
| Test file | Matches | Will break (b) | Safe (a/c) |
|-----------|---------|----------------|------------|
| `tests/lib/output.test.mjs` | 1 | 0 | 1 |
| `tests/scanners/feature-gap-scanner.test.mjs` | 6 | 6 | 0 |
| `tests/scanners/hook-validator.test.mjs` | 12 | 9 | 3 |
| `tests/lib/diff-engine.test.mjs` | 2 | 0 | 2 |
| `tests/scanners/fix-engine.test.mjs` | 1 | 0 | 1 |
| `tests/scanners/plugin-health-scanner.test.mjs` | 9 | 8 | 1 |
| `tests/scanners/settings-validator.test.mjs` | 11 | 11 | 0 |
| **Total** | **42** | **34** | **8** |
## Per-file findings
### `tests/lib/output.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 46 | `assert.strictEqual(f.title, 'Test')` | (a) shape-only | None — `'Test'` is the test's own input to `finding()` constructor, not a scanner-produced title. |
### `tests/scanners/feature-gap-scanner.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 45 | `f.title === 'No CLAUDE.md file'` | (b) WILL BREAK | Replace with `f.id === '<GAP-ID-for-no-CLAUDE.md>'`. Anchor on ID. |
| 49 | `f.title === 'No MCP servers configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 53 | `f.title === 'No hooks configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 96 | `f.title === 'No hooks configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 100 | `f.title === 'No MCP servers configured'` | (b) WILL BREAK | Replace with ID anchor. |
| 150 | `f.title === 'No CLAUDE.md file'` | (b) WILL BREAK | Replace with ID anchor. |
> **Implementation note for Step 4:** look up the actual GAP finding IDs via `grep -n "title:" scanners/feature-gap-scanner.mjs` and substitute. For shape only: `assert.ok(f.id.startsWith('CA-GAP-'))` is acceptable when the test only cares that a GAP finding fired.
### `tests/scanners/hook-validator.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 30 | `serious.map(f => f.title).join(', ')` | (a) shape-only | None — title used only for error-message formatting in failed assert; not the assertion itself. |
| 49 | `f.title === 'Unknown hook event'` | (b) WILL BREAK | Replace with ID anchor. |
| 54 | `f.title.includes('Matcher must be a string')` | (b) WILL BREAK | Replace with ID anchor or `.evidence.includes(...)`. |
| 59 | `f.title === 'Invalid hook handler type'` | (b) WILL BREAK | Replace with ID anchor. |
| 64 | `f.title.includes('timeout')` | (b) WILL BREAK | Replace with ID anchor. |
| 69 | `f.title === 'Unknown hook event'` | (b) WILL BREAK | Replace with ID anchor. |
| 80 | `/verbose hook output/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor. |
| 81 | `result.findings.map(x => x.title).join(' \| ')` | (a) shape-only | Used only in error-message formatting. None. |
| 91 | `/verbose hook output/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor. |
| 92 | `f?.title` | (a) shape-only | Used only in error-message formatting. None. |
### `tests/lib/diff-engine.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 66 | `diff.newFindings[0].title === 'New issue'` | (a) shape-only | None — `'New issue'` is the test's synthetic finding input, not scanner-produced. |
| 78 | `diff.resolvedFindings[0].title === 'Old issue'` | (a) shape-only | None — synthetic test input. |
### `tests/scanners/fix-engine.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 62 | `assert.ok(m.title, 'Manual finding should have title')` | (a) shape-only | None — pure existence check. |
### `tests/scanners/plugin-health-scanner.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 52 | `f.title.includes('Missing required field')` | (b) WILL BREAK | Replace with ID anchor or `f.evidence.includes(...)`. |
| 59 | `f.title.includes('missing') && f.title.includes('section')` | (b) WILL BREAK | Replace with ID anchor on the missing-section finding. |
| 68 | `f.title.includes('Missing required field')` | (b) WILL BREAK | Replace with ID anchor. |
| 75 | `f.title === 'Missing CLAUDE.md'` | (b) WILL BREAK | Replace with ID anchor. |
| 82 | `f.title === 'Command missing frontmatter'` | (b) WILL BREAK | Replace with ID anchor. |
| 90 | `f.title.startsWith('Agent missing frontmatter field:')` | (b) WILL BREAK | Replace with ID anchor + `f.evidence.includes(...)` for the field name (humanizer preserves evidence). |
| 93 | `missingAgent.map(f => f.title).join(', ')` | (a) shape-only | Used only in error-message formatting. None. |
| 102 | `result.findings[0].title === 'No plugins found'` | (b) WILL BREAK | Replace with ID anchor. |
| 125 | `assert.ok(f.title)` | (a) shape-only | None — pure existence check. |
### `tests/scanners/settings-validator.test.mjs`
| Line | Code | Class | Action |
|------|------|-------|--------|
| 49 | `f.title === 'Unknown settings key'` | (b) WILL BREAK | Replace with ID anchor (likely `CA-SET-001` or similar — verify). |
| 54 | `f.title === 'Deprecated settings key'` | (b) WILL BREAK | Replace with ID anchor. |
| 59 | `f.title === 'Type mismatch in settings'` | (b) WILL BREAK | Replace with ID anchor. |
| 64 | `f.title === 'Invalid effortLevel value'` | (b) WILL BREAK | Replace with ID anchor. |
| 69 | `f.title.includes('array instead of object')` | (b) WILL BREAK | Replace with ID anchor. |
| 74 | `f.title.includes('array instead of object')` | (b) WILL BREAK | Replace with ID anchor. |
| 86 | `f.title === 'Unknown settings key' && /additionalDirectories/.test(f.evidence)` | (b) WILL BREAK | Keep evidence regex; replace title check with ID anchor. |
| 96 | `/additionalDirectories/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor + evidence regex (additionalDirectories likely appears in evidence already). |
| 98 | `f?.title` | (a) shape-only — but inside breaking assertion | Will become moot after line 96 is fixed. |
| 106 | `/additionalDirectories/i.test(x.title \|\| '')` | (b) WILL BREAK | Replace with ID anchor + evidence regex. |
| 107 | `result.findings.map(x => x.title).join(' \| ')` | (a) shape-only | Error-message formatting only. None. |
## Step 4 implementation guidance
1. For each (b) WILL BREAK row, look up the actual finding ID from the corresponding scanner source:
- `grep -n "id: 'CA-GAP-" scanners/feature-gap-scanner.mjs`
- `grep -n "id: 'CA-HKV-" scanners/hook-validator.mjs`
- `grep -n "id: 'CA-PLH-" scanners/plugin-health-scanner.mjs`
- `grep -n "id: 'CA-SET-" scanners/settings-validator.mjs`
2. Replace the title check with `f.id === '<exact-id>'`. If the test cares about a sub-variant (e.g., a specific deprecated key), pair the ID anchor with an `f.evidence.includes(...)` substring check — humanizer preserves `evidence` exactly.
3. For broad categorical checks ("any GAP finding fired"), use `f.id.startsWith('CA-GAP-')`.
4. For tests that capture `f.title` only inside `assert` failure-message templates (class (a)): leave them. Humanization changes the displayed string but the assertion still anchors on `f.id`.
5. Re-run `node --test 'tests/**/*.test.mjs'` after changes; expect zero regressions before proceeding to Step 5.
## Total scope for Step 4
- **6 test files** require code changes (`output.test.mjs` and `diff-engine.test.mjs` are clean).
- **34 distinct assertions** to convert.
- Estimated effort: 1–2 hours including ID lookup and verification.
Reference in a new issue View git blame Copy permalink