feat(engine)!: a skip says which of the two things it is

`notChecked` merged "deliberately not judged" with "could not run". 0.7.0 took
SKIP out of the severity lattice, which fixed "clean repos look skipped"; this
fixes the same defect one level down, where it was saying so more quietly.
Measured across 19 clones, 35 of 39 skips are deliberate — so
`portfolio-optimiser — OK · 11 not checked` announced eleven unread checks when
all eleven were fixture and out-of-repo links.

Every SKIP finding now carries `skip: 'byDesign' | 'notRun'`, set at the
emission site rather than looked up from its code: VERSION-TAG is emitted at
SKIP with no tags and at ERROR with the wrong one, so a code→kind map has to
re-derive a reason the emitter already had. Untagged falls to `notRun`, the
loud side. Results carry `skips: { byDesign, notRun }`; `notChecked` stays a
number and stays the total, because a consumer doing `notChecked > 0` against
an object gets a silent false.

The summary line names only `notRun` — the deliberate ones keep their own
`NOT JUDGED` heading in the detail output. Landing the split in `--json` alone
would have left the symptom exactly where it was.

VERSION-UNAVAILABLE stays `notRun` by decision, not by default: it is arguably
"no subject to judge", the shape checkReadmeLanguage answers with OK, but
re-levelling it moves a repo's status and is a second behaviour change.

Verified: 169 tests. 19 clones swept with both engines offline — 220 judged
findings, every status, notChecked and buckets identical, no finding differing
beyond the new tag. The exhaustiveness test scans the engine SOURCE, not a
finding set: a corpus test only sees the sites it triggers, and the next
emission site added would take the reader's default invisibly.

BREAKING CHANGE: the summary line's coverage qualifier reads `· N not run` and
counts only un-runnable skips, where it read `· N not checked` over all of
them. `--json` gains `skips`; `notChecked` is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHaQTcaD3w2C6PqVsAmrcv
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 21:52:00 +02:00
commit f4017bcfe1
8 changed files with 346 additions and 38 deletions

View file

@ -30,16 +30,25 @@ could not run), `OK`. Exit 1 on any ERROR. Add `--offline` to skip the two
network calls, `--json` for machine output, `--refresh` to compare the bundled
register against the live org listing.
**Never report a `SKIP` as a pass.** A SKIP means the gate could not see enough
to judge — an unreachable forge, an untagged repo, a link leaving the
repository. Say which.
**Never report a `SKIP` as a pass.** A SKIP means the gate did not judge the
thing. Say which, and say which KIND — since 0.8.0 there are two, and they are
not interchangeable:
- **`NOT CHECKED` (`skips.notRun`)** — the gate could not see enough. An
unreachable forge or catalog, an unregistered repo, a repo with no tags. A
re-run or an operator action turns it into a verdict, so it belongs in your
report even when the status is green.
- **`NOT JUDGED` (`skips.byDesign`)** — the gate saw it and declined on
purpose. A link leaving the repository, a dead link inside a fixture path.
Nobody has an action. Reporting these as outstanding work is how a clean repo
gets rewritten for nothing.
That applies hardest when the repo's own status is green. Since 0.7.0 a `SKIP`
no longer outranks an `OK` in the status line, so a repo reads
`OK · 3 not checked` rather than `SKIP`. The number is the qualifier: three
findings nobody judged. Report the status and the number together, never the
status alone, and name what went unjudged — the detail output lists them under
`NOT CHECKED`.
no longer outranks an `OK` in the status line, so a repo reads `OK · 3 not run`
rather than `SKIP`. The summary line carries only the `notRun` count — a repo
whose every skip was deliberate prints an unqualified verdict. **The line's
silence is not proof there was nothing to say:** read the `NOT JUDGED` block, or
`skips.byDesign` in `--json`, before you tell anyone a repo was fully judged.
**Confirm the version before you trust a green run.** Measured 2026-08-04:
`${CLAUDE_PLUGIN_ROOT}` resolved to a cached `0.1.1` in a live session while