feat(engine)!: a skip says which of the two things it is

`notChecked` merged "deliberately not judged" with "could not run". 0.7.0 took
SKIP out of the severity lattice, which fixed "clean repos look skipped"; this
fixes the same defect one level down, where it was saying so more quietly.
Measured across 19 clones, 35 of 39 skips are deliberate — so
`portfolio-optimiser — OK · 11 not checked` announced eleven unread checks when
all eleven were fixture and out-of-repo links.

Every SKIP finding now carries `skip: 'byDesign' | 'notRun'`, set at the
emission site rather than looked up from its code: VERSION-TAG is emitted at
SKIP with no tags and at ERROR with the wrong one, so a code→kind map has to
re-derive a reason the emitter already had. Untagged falls to `notRun`, the
loud side. Results carry `skips: { byDesign, notRun }`; `notChecked` stays a
number and stays the total, because a consumer doing `notChecked > 0` against
an object gets a silent false.

The summary line names only `notRun` — the deliberate ones keep their own
`NOT JUDGED` heading in the detail output. Landing the split in `--json` alone
would have left the symptom exactly where it was.

VERSION-UNAVAILABLE stays `notRun` by decision, not by default: it is arguably
"no subject to judge", the shape checkReadmeLanguage answers with OK, but
re-levelling it moves a repo's status and is a second behaviour change.

Verified: 169 tests. 19 clones swept with both engines offline — 220 judged
findings, every status, notChecked and buckets identical, no finding differing
beyond the new tag. The exhaustiveness test scans the engine SOURCE, not a
finding set: a corpus test only sees the sites it triggers, and the next
emission site added would take the reader's default invisibly.

BREAKING CHANGE: the summary line's coverage qualifier reads `· N not run` and
counts only un-runnable skips, where it read `· N not checked` over all of
them. `--json` gains `skips`; `notChecked` is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHaQTcaD3w2C6PqVsAmrcv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:52:00 +02:00
commit f4017bcfe1
8 changed files with 346 additions and 38 deletions

View file

@ -10,7 +10,7 @@ checks that surface in one repository and reports what it finds.
*AI-generated: all code produced by Claude Code through dialog-driven development.*
![Version](https://img.shields.io/badge/version-0.7.1-blue)
![Version](https://img.shields.io/badge/version-0.8.0-blue)
![Platform](https://img.shields.io/badge/platform-Claude_Code_Plugin-purple)
![Skills](https://img.shields.io/badge/skills-1-orange)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
@ -80,26 +80,43 @@ Findings carry two independent things: a **level** (`ERROR`, `WARN`, `SKIP`,
- **missing** — an expected artefact is absent
- **weakening** — present and working, but it reads as amateur
The process exits 1 on any `ERROR`. A `SKIP` means the check could not run — an
unreachable forge, an untagged repo, a link that leaves the repository. It is
not a pass, and the output prints those separately under a heading that says so.
The process exits 1 on any `ERROR`. A `SKIP` is never a pass and never a
severity — it is the absence of a verdict, printed apart from the judged
findings.
A repository's own `status` is the worst of the findings that were *judged*, and
`SKIP` only when nothing was. A skip is not a severity — it is the absence of a
verdict, so it cannot outrank twelve checks that did run. Coverage rides beside
it on its own axis: `notChecked` in `--json`, and a qualifier on the summary
line, so a pass that had checks it could not run never prints unqualified.
`SKIP` only when nothing was. A skip cannot outrank twelve checks that did run.
Coverage rides beside it on its own axis, and since 0.8.0 that axis has two
values, because a skip is one of two unrelated facts:
- **not judged, by design** — the check saw the thing and declined. A link that
leaves the repository; a dead link inside a fixture path. It can never become
a verdict, and nobody has an action.
- **could not run** — a re-run or an operator action turns it into a verdict.
An unreachable catalog, an unregistered repo, a repo with no tags.
Merging them made clean repositories read as unread. Measured across 19 clones,
35 of 39 skips were the first kind, so `portfolio-optimiser — OK · 11 not
checked` announced eleven unread checks when all eleven were links the gate
declines to judge on purpose.
The summary line names only what someone has an action on:
```
✓ okr [plugin] — OK · 1 not checked (repo-standard v0.7.0 @10ad125)
· stranger — SKIP · 1 not checked (repo-standard v0.7.0 @10ad125)
✓ portfolio-optimiser [standalone] — OK (repo-standard v0.8.0 @<sha>)
✗ some-plugin [plugin] — ERROR · 2 not run (repo-standard v0.8.0 @<sha>)
· stranger — SKIP · 1 not run (repo-standard v0.8.0 @<sha>)
```
The count is **findings**, not checks — one per un-judged item, the same list the
detail output prints under `NOT CHECKED`. A repo with many links leaving it
scores a high `notChecked` without a single check having failed to run, which is
why the next thing this number needs is to distinguish *deliberately not judged*
from *could not run*.
The deliberate ones are not silenced: they keep their own `NOT JUDGED` heading
in the detail output, beside `NOT CHECKED`. A decision nobody can see reads
exactly like a check that silently stopped running.
`--json` carries `skips: { byDesign, notRun }` beside `status`. `notChecked` is
unchanged, and still the numeric total of both — the counts are **findings**,
not checks, one per un-judged item. Absent `skips` means an engine older than
0.8.0 and absent `notChecked` one older than 0.7.0; neither absence is zero.
### Traits — a second axis
@ -239,7 +256,7 @@ distinct from "an older engine that never tried".
npm test
```
147 tests over the pure classifiers. The reference fixtures are measured false
169 tests over the pure classifiers. The reference fixtures are measured false
positives, each with its expected verdict — the six that produced the
three-outcome reference rule, plus the noise sources found by running the gate
against a real repository: regexes inside code spans that are markdown links to