feat(humanizer): scenario read-test corpus + runner (SC-4) [skip-docs]
Step 9 of v5.1.0 humanizer Wave 4. Adds tests/scenario-read-test.mjs
runner, tests/scenario-read-test.test.mjs wrapper, and 5 scenario
fixtures in tests/scenarios/ that feed deterministic raw findings
through humanizeFinding and assert the humanized
title/description/recommendation match brief-owner-approved regex
patterns encoding the ground-truth what/why/whatNext answers.
Corpus selection (per brief criteria):
- 01-tok-cascade.json - TOK/CPS category (token efficiency)
- 02-cps-volatile.json - TOK/CPS category (cache prefix stability)
- 03-cnf-conflict.json - CNF category (conflicts)
- 04-gap-no-claude-md.json - GAP category (feature gap)
- 05-set-invalid-json.json - SET category, AND its v5.0.0 title +
description carry tier1 'invalid' (the brief criterion 'one finding
whose v5.0.0 description uses a forbidden word').
Runner mechanics:
- Loads scenarios matching ^\\d{2}-[a-z0-9-]+\\.json$ in sorted order.
- Calls humanizeFinding(scannerInput) and matches each humanized field
against its declared pattern (case-insensitive regex).
- Verifies humanizer-added structural fields (userImpactCategory,
userActionLanguage, relevanceContext) are non-empty strings.
- Per session decision (1a) acceptance is deterministic regex matching
without a runtime human approval gate.
Wrapper adds 3 tests: scenario-match (binds runner to node --test),
category-coverage (TOK/CPS, CNF, GAP, SET all present), and
tier1-presence (at least one v5.0.0 title or description contains a
tier1 forbidden word).
Tests: 736 to 739 (+3 SC-4 tests). Full suite passes.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This commit is contained in:
parent
c5c937e94e
commit
8b146bf489
7 changed files with 373 additions and 0 deletions
29
plugins/config-audit/tests/scenarios/02-cps-volatile.json
Normal file
29
plugins/config-audit/tests/scenarios/02-cps-volatile.json
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
{
|
||||
"_meta": {
|
||||
"comment": "Scenario 02: CPS volatile content inside cached prefix. Covers the CPS half of the TOK/CPS category. Tests that the humanizer explains cache-prefix-stability in user-facing language."
|
||||
},
|
||||
"findingId": "CA-CPS-001",
|
||||
"scannerInput": {
|
||||
"id": "CA-CPS-001",
|
||||
"scanner": "CPS",
|
||||
"severity": "medium",
|
||||
"title": "Volatile content inside cached prefix breaks reuse",
|
||||
"description": "Volatile pattern matched at .claude/CLAUDE.md:42 (timestamp). Lines 31-150 form the cache prefix.",
|
||||
"file": ".claude/CLAUDE.md",
|
||||
"line": 42,
|
||||
"evidence": "Pattern: timestamp; window: 31-150",
|
||||
"recommendation": "Move volatile content (timestamps, session state) below line 150 or to a separate file.",
|
||||
"category": null,
|
||||
"autoFixable": false
|
||||
},
|
||||
"expectedHumanized": {
|
||||
"titlePattern": "Content that changes between turns sits in the part Claude tries to reuse",
|
||||
"descriptionPattern": "fresh read every time|slows responses",
|
||||
"recommendationPattern": "Move the changing content|150 lines"
|
||||
},
|
||||
"groundTruth": {
|
||||
"what": "Content that changes between turns is inside the part of the file Claude tries to reuse.",
|
||||
"why": "Claude saves space by reusing the start of your instructions across turns. When that area changes, every turn re-reads the whole start, which slows responses.",
|
||||
"whatNext": "Move the changing content (timestamps, session notes) below the first 150 lines, or out of the file entirely."
|
||||
}
|
||||
}
|
||||
Loading…
Add table
Add a link
Reference in a new issue