feat(engine)!: a skip says which of the two things it is
`notChecked` merged "deliberately not judged" with "could not run". 0.7.0 took
SKIP out of the severity lattice, which fixed "clean repos look skipped"; this
fixes the same defect one level down, where it was saying so more quietly.
Measured across 19 clones, 35 of 39 skips are deliberate — so
`portfolio-optimiser — OK · 11 not checked` announced eleven unread checks when
all eleven were fixture and out-of-repo links.
Every SKIP finding now carries `skip: 'byDesign' | 'notRun'`, set at the
emission site rather than looked up from its code: VERSION-TAG is emitted at
SKIP with no tags and at ERROR with the wrong one, so a code→kind map has to
re-derive a reason the emitter already had. Untagged falls to `notRun`, the
loud side. Results carry `skips: { byDesign, notRun }`; `notChecked` stays a
number and stays the total, because a consumer doing `notChecked > 0` against
an object gets a silent false.
The summary line names only `notRun` — the deliberate ones keep their own
`NOT JUDGED` heading in the detail output. Landing the split in `--json` alone
would have left the symptom exactly where it was.
VERSION-UNAVAILABLE stays `notRun` by decision, not by default: it is arguably
"no subject to judge", the shape checkReadmeLanguage answers with OK, but
re-levelling it moves a repo's status and is a second behaviour change.
Verified: 169 tests. 19 clones swept with both engines offline — 220 judged
findings, every status, notChecked and buckets identical, no finding differing
beyond the new tag. The exhaustiveness test scans the engine SOURCE, not a
finding set: a corpus test only sees the sites it triggers, and the next
emission site added would take the reader's default invisibly.
BREAKING CHANGE: the summary line's coverage qualifier reads `· N not run` and
counts only un-runnable skips, where it read `· N not checked` over all of
them. `--json` gains `skips`; `notChecked` is unchanged.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHaQTcaD3w2C6PqVsAmrcv
This commit is contained in:
parent
1a647a591f
commit
f4017bcfe1
8 changed files with 346 additions and 38 deletions
49
README.md
49
README.md
|
|
@ -10,7 +10,7 @@ checks that surface in one repository and reports what it finds.
|
|||
|
||||
*AI-generated: all code produced by Claude Code through dialog-driven development.*
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||

|
||||
|
|
@ -80,26 +80,43 @@ Findings carry two independent things: a **level** (`ERROR`, `WARN`, `SKIP`,
|
|||
- **missing** — an expected artefact is absent
|
||||
- **weakening** — present and working, but it reads as amateur
|
||||
|
||||
The process exits 1 on any `ERROR`. A `SKIP` means the check could not run — an
|
||||
unreachable forge, an untagged repo, a link that leaves the repository. It is
|
||||
not a pass, and the output prints those separately under a heading that says so.
|
||||
The process exits 1 on any `ERROR`. A `SKIP` is never a pass and never a
|
||||
severity — it is the absence of a verdict, printed apart from the judged
|
||||
findings.
|
||||
|
||||
A repository's own `status` is the worst of the findings that were *judged*, and
|
||||
`SKIP` only when nothing was. A skip is not a severity — it is the absence of a
|
||||
verdict, so it cannot outrank twelve checks that did run. Coverage rides beside
|
||||
it on its own axis: `notChecked` in `--json`, and a qualifier on the summary
|
||||
line, so a pass that had checks it could not run never prints unqualified.
|
||||
`SKIP` only when nothing was. A skip cannot outrank twelve checks that did run.
|
||||
|
||||
Coverage rides beside it on its own axis, and since 0.8.0 that axis has two
|
||||
values, because a skip is one of two unrelated facts:
|
||||
|
||||
- **not judged, by design** — the check saw the thing and declined. A link that
|
||||
leaves the repository; a dead link inside a fixture path. It can never become
|
||||
a verdict, and nobody has an action.
|
||||
- **could not run** — a re-run or an operator action turns it into a verdict.
|
||||
An unreachable catalog, an unregistered repo, a repo with no tags.
|
||||
|
||||
Merging them made clean repositories read as unread. Measured across 19 clones,
|
||||
35 of 39 skips were the first kind, so `portfolio-optimiser — OK · 11 not
|
||||
checked` announced eleven unread checks when all eleven were links the gate
|
||||
declines to judge on purpose.
|
||||
|
||||
The summary line names only what someone has an action on:
|
||||
|
||||
```
|
||||
✓ okr [plugin] — OK · 1 not checked (repo-standard v0.7.0 @10ad125)
|
||||
· stranger — SKIP · 1 not checked (repo-standard v0.7.0 @10ad125)
|
||||
✓ portfolio-optimiser [standalone] — OK (repo-standard v0.8.0 @<sha>)
|
||||
✗ some-plugin [plugin] — ERROR · 2 not run (repo-standard v0.8.0 @<sha>)
|
||||
· stranger — SKIP · 1 not run (repo-standard v0.8.0 @<sha>)
|
||||
```
|
||||
|
||||
The count is **findings**, not checks — one per un-judged item, the same list the
|
||||
detail output prints under `NOT CHECKED`. A repo with many links leaving it
|
||||
scores a high `notChecked` without a single check having failed to run, which is
|
||||
why the next thing this number needs is to distinguish *deliberately not judged*
|
||||
from *could not run*.
|
||||
The deliberate ones are not silenced: they keep their own `NOT JUDGED` heading
|
||||
in the detail output, beside `NOT CHECKED`. A decision nobody can see reads
|
||||
exactly like a check that silently stopped running.
|
||||
|
||||
`--json` carries `skips: { byDesign, notRun }` beside `status`. `notChecked` is
|
||||
unchanged, and still the numeric total of both — the counts are **findings**,
|
||||
not checks, one per un-judged item. Absent `skips` means an engine older than
|
||||
0.8.0 and absent `notChecked` one older than 0.7.0; neither absence is zero.
|
||||
|
||||
### Traits — a second axis
|
||||
|
||||
|
|
@ -239,7 +256,7 @@ distinct from "an older engine that never tried".
|
|||
npm test
|
||||
```
|
||||
|
||||
147 tests over the pure classifiers. The reference fixtures are measured false
|
||||
169 tests over the pure classifiers. The reference fixtures are measured false
|
||||
positives, each with its expected verdict — the six that produced the
|
||||
three-outcome reference rule, plus the noise sources found by running the gate
|
||||
against a real repository: regexes inside code spans that are markdown links to
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue