8b146bf489
Step 9 of v5.1.0 humanizer Wave 4. Adds tests/scenario-read-test.mjs
runner, tests/scenario-read-test.test.mjs wrapper, and 5 scenario
fixtures in tests/scenarios/ that feed deterministic raw findings
through humanizeFinding and assert the humanized
title/description/recommendation match brief-owner-approved regex
patterns encoding the ground-truth what/why/whatNext answers.
Corpus selection (per brief criteria):
- 01-tok-cascade.json - TOK/CPS category (token efficiency)
- 02-cps-volatile.json - TOK/CPS category (cache prefix stability)
- 03-cnf-conflict.json - CNF category (conflicts)
- 04-gap-no-claude-md.json - GAP category (feature gap)
- 05-set-invalid-json.json - SET category, AND its v5.0.0 title +
description carry tier1 'invalid' (the brief criterion 'one finding
whose v5.0.0 description uses a forbidden word').
Runner mechanics:
- Loads scenarios matching ^\\d{2}-[a-z0-9-]+\\.json$ in sorted order.
- Calls humanizeFinding(scannerInput) and matches each humanized field
against its declared pattern (case-insensitive regex).
- Verifies humanizer-added structural fields (userImpactCategory,
userActionLanguage, relevanceContext) are non-empty strings.
- Per session decision (1a) acceptance is deterministic regex matching
without a runtime human approval gate.
Wrapper adds 3 tests: scenario-match (binds runner to node --test),
category-coverage (TOK/CPS, CNF, GAP, SET all present), and
tier1-presence (at least one v5.0.0 title or description contains a
tier1 forbidden word).
Tests: 736 to 739 (+3 SC-4 tests). Full suite passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
29 lines
1.5 KiB
JSON
29 lines
1.5 KiB
JSON
{
|
|
"_meta": {
|
|
"comment": "Scenario 05: SET invalid JSON in settings file. Covers the settings category AND the brief criterion 'one finding whose v5.0.0 description contains a tier1 forbidden word' — both the v5.0.0 title and description carry tier1 'invalid'. Humanizer rewrites to plain language."
|
|
},
|
|
"findingId": "CA-SET-001",
|
|
"scannerInput": {
|
|
"id": "CA-SET-001",
|
|
"scanner": "SET",
|
|
"severity": "critical",
|
|
"title": "Invalid JSON in settings file",
|
|
"description": ".claude/settings.json contains invalid JSON and will be ignored by Claude Code.",
|
|
"file": ".claude/settings.json",
|
|
"line": null,
|
|
"evidence": "Unexpected token } in JSON at position 187",
|
|
"recommendation": "Fix JSON syntax errors. Use a JSON validator.",
|
|
"category": null,
|
|
"autoFixable": false
|
|
},
|
|
"expectedHumanized": {
|
|
"titlePattern": "settings file isn'?t readable as JSON",
|
|
"descriptionPattern": "Claude Code can'?t parse the file|settings are skipped",
|
|
"recommendationPattern": "Open the file and fix the JSON syntax"
|
|
},
|
|
"groundTruth": {
|
|
"what": "Your settings file can't be read as JSON, so none of the settings inside take effect.",
|
|
"why": "Claude Code parses the settings file once at startup. When that parse fails, all settings inside are skipped silently, and you get the defaults.",
|
|
"whatNext": "Open the file and fix the JSON syntax shown in the details (often a missing comma, an extra trailing comma, or an unterminated quote)."
|
|
}
|
|
}
|