feat(engine)!: a skip says which of the two things it is
`notChecked` merged "deliberately not judged" with "could not run". 0.7.0 took
SKIP out of the severity lattice, which fixed "clean repos look skipped"; this
fixes the same defect one level down, where it was saying so more quietly.
Measured across 19 clones, 35 of 39 skips are deliberate — so
`portfolio-optimiser — OK · 11 not checked` announced eleven unread checks when
all eleven were fixture and out-of-repo links.
Every SKIP finding now carries `skip: 'byDesign' | 'notRun'`, set at the
emission site rather than looked up from its code: VERSION-TAG is emitted at
SKIP with no tags and at ERROR with the wrong one, so a code→kind map has to
re-derive a reason the emitter already had. Untagged falls to `notRun`, the
loud side. Results carry `skips: { byDesign, notRun }`; `notChecked` stays a
number and stays the total, because a consumer doing `notChecked > 0` against
an object gets a silent false.
The summary line names only `notRun` — the deliberate ones keep their own
`NOT JUDGED` heading in the detail output. Landing the split in `--json` alone
would have left the symptom exactly where it was.
VERSION-UNAVAILABLE stays `notRun` by decision, not by default: it is arguably
"no subject to judge", the shape checkReadmeLanguage answers with OK, but
re-levelling it moves a repo's status and is a second behaviour change.
Verified: 169 tests. 19 clones swept with both engines offline — 220 judged
findings, every status, notChecked and buckets identical, no finding differing
beyond the new tag. The exhaustiveness test scans the engine SOURCE, not a
finding set: a corpus test only sees the sites it triggers, and the next
emission site added would take the reader's default invisibly.
BREAKING CHANGE: the summary line's coverage qualifier reads `· N not run` and
counts only un-runnable skips, where it read `· N not checked` over all of
them. `--json` gains `skips`; `notChecked` is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHaQTcaD3w2C6PqVsAmrcv
This commit is contained in:
parent
1a647a591f
commit
f4017bcfe1
8 changed files with 346 additions and 38 deletions
17
CLAUDE.md
17
CLAUDE.md
|
|
@ -39,6 +39,21 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
clean", so coverage rides its own axis: `notChecked` in `--json`, a `· N not
|
||||
checked` qualifier on the summary line. Absent `notChecked` means an older
|
||||
engine and prints the pre-0.7.0 line — it is not zero.
|
||||
- **A skip is one of TWO facts, and merging them says the same wrong thing more
|
||||
quietly.** `byDesign` (the check saw it and declined — out-of-repo links,
|
||||
fixture paths; it can never become a verdict, nobody has an action) versus
|
||||
`notRun` (a re-run or an operator action resolves it). Measured across 19
|
||||
clones, 35 of 39 skips were the first kind, which is why `portfolio-optimiser
|
||||
— OK · 11 not checked` read as eleven unread checks. The kind is set at the
|
||||
EMISSION site, never looked up from the code: `VERSION-TAG` is emitted at
|
||||
`SKIP` with no tags and at `ERROR` with the wrong one, so a code→kind map has
|
||||
to re-derive a reason the emission site already had. Untagged falls to
|
||||
`notRun`, the loud side. The summary line names only `notRun`; the deliberate
|
||||
ones keep their own `NOT JUDGED` heading, because the wanted side effect is
|
||||
exposure, not silence — landing the split in `--json` alone would have left
|
||||
the symptom untouched. The test that holds this is a scan of the engine
|
||||
SOURCE, not of a finding set: a corpus test only sees the sites it manages to
|
||||
trigger, and the next site added would take the reader's default invisibly.
|
||||
- **When a check fires many times in one repo, suspect the CHECK.** The first
|
||||
link pass produced ~30 findings against `llm-security` and all were noise —
|
||||
regexes in code spans, `file:` URLs, relative paths resolved against the wrong
|
||||
|
|
@ -134,7 +149,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
## Commands
|
||||
|
||||
```bash
|
||||
npm test # 147 tests
|
||||
npm test # 169 tests
|
||||
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
|
||||
node scripts/repo-standard-check.mjs --offline # no network call
|
||||
node scripts/repo-standard-check.mjs --json # machine output
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue