feat(engine)!: "no version to compare against" is a verdict, not a skip

`VERSION-UNAVAILABLE` (SKIP/notRun) becomes `VERSION-NONE` (OK). A repo that
claims no version in any of the four places one can be written down has not
left a check un-run — the check ran, saw all four, and found no subject. That
is the shape `checkReadmeLanguage` has always answered with OK.

0.8.0 deferred this and recorded the reason in the engine: re-levelling "moves
a repo's status". Measured false across 19 clones — an added OK cannot worsen
the worst *judged* finding, and all three repos emitting the code already read
OK. Status moves only for a repo whose entire finding set was skips.

OK cannot bless a real gap here: no class requires a version file, and a
`plugin` missing its manifest is an independent FILE-MISSING ERROR.

Renamed, not just re-levelled: `-UNAVAILABLE` is this engine's naming for a
notRun skip, and one OK-level `-UNAVAILABLE` would mislead exactly the reader
this change serves. A source scan keeps the old name out.

A/B sweep, both engines offline, 19 clones: 223 judged findings, every status
and buckets identical; 3 conversions (notChecked 14→13, 3→2, 14→13), byDesign
untouched. 170 tests. `--refresh`: register 21 = forge 21.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CTNGvxzCLZp3UXfibYG4kb
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 22:05:21 +02:00
commit 1dfd408ffc
8 changed files with 99 additions and 21 deletions

View file

@ -54,6 +54,21 @@ would recreate, in data, exactly the drift this plugin exists to remove.
the symptom untouched. The test that holds this is a scan of the engine
SOURCE, not of a finding set: a corpus test only sees the sites it manages to
trigger, and the next site added would take the reader's default invisibly.
- **"Nothing to judge" is an `OK`, not a skip — and it is a THIRD shape, not a
third bucket.** A check that ran, saw everything, and found no subject has
produced a verdict: there is nothing here to be wrong. `checkReadmeLanguage`
already answered that way for a README with no prose; `VERSION-NONE` (0.9.0,
was `VERSION-UNAVAILABLE`/`notRun`) does for a repo that claims no version.
The deferral that kept it a skip for one release was recorded in the engine as
"re-levelling moves a repo's status" — measured false: an added `OK` cannot
worsen the worst *judged* finding, and all three affected repos already read
`OK`. Status moves only for a repo whose *entire* finding set was skips. The
lesson is not about this code: **a recorded decision that is wrong is worse
than no record**, because the next session re-derives the measurement instead
of reading it. And the guard against `OK` blessing a real gap is that the gap
is caught elsewhere — no class requires a version file, and a `plugin` missing
its manifest is an independent `FILE-MISSING` `ERROR`. Without that, this
would have been "SKIP is never a pass" one level up.
- **When a check fires many times in one repo, suspect the CHECK.** The first
link pass produced ~30 findings against `llm-security` and all were noise —
regexes in code spans, `file:` URLs, relative paths resolved against the wrong
@ -149,7 +164,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
## Commands
```bash
npm test # 169 tests
npm test # 170 tests
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
node scripts/repo-standard-check.mjs --offline # no network call
node scripts/repo-standard-check.mjs --json # machine output