feat(engine)!: SKIP stops outranking OK, coverage gets its own axis
A repo's `status` is now the worst JUDGED finding, and `SKIP` only when nothing was judged. `SKIP` used to rank between `OK` and `WARN`, so one un-runnable check spoke for every check that ran: 0 ERROR, 0 WARN and a dozen OK headlined as "skipped". Five repos in org-ops census 05, `okr` among them with the most OK in the org, reading as unread. "`SKIP` is never a pass" survives in the half of the rule that carries it — an unregistered repo, or an empty finding set, still says SKIP, because there is nothing else to be worst of. Fixing the status alone would have traded "clean repos look skipped" for "skipped checks look clean". So `notChecked` rides beside it: in --json, and as a `· N not checked` qualifier on the summary line. Absent means an older engine, not zero. Measured across all 21 local clones from ONE saved sweep, so before and after come from the same findings rather than two sweeps of a moving org: 343 findings before, 343 after. 8 repos moved, every one SKIP -> OK. 1 ERROR and 3 WARN before, 1 ERROR and 3 WARN after — the counts that decide whether a repo needs work did not move. BREAKING: consumers reading `.status` see a changed value domain. 147 -> 154 tests.
This commit is contained in:
parent
5884a64e54
commit
10ad1254ab
7 changed files with 162 additions and 14 deletions
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "repo-standard",
|
||||
"version": "0.6.0",
|
||||
"version": "0.7.0",
|
||||
"description": "Per-repo gate for the open/ presentation standard: README first screen, install block, files required by the repo's class, and dead repo references.",
|
||||
"author": {
|
||||
"name": "Kjell Tore Guttormsen"
|
||||
|
|
|
|||
39
CHANGELOG.md
39
CHANGELOG.md
|
|
@ -4,6 +4,45 @@ All notable changes to this project are documented here.
|
|||
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
|
||||
versioning is [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.7.0] — 2026-08-09
|
||||
|
||||
### Changed
|
||||
|
||||
- **A repo's `status` is the worst *judged* finding, and `SKIP` only when
|
||||
nothing was judged.** `SKIP` used to rank between `OK` and `WARN`, so one
|
||||
un-runnable check spoke for every check that ran: a repo with 0 ERROR, 0 WARN
|
||||
and a dozen `OK` headlined as "skipped". Reported by org-ops against five
|
||||
repos in census 05 — `ktg-plugin-marketplace`, `okr`,
|
||||
`playground-design-system`, `portfolio-optimiser`, `commons` — with `okr`
|
||||
holding the most `OK` in the org and reading as unread. A skip is not a
|
||||
severity; it is the absence of a verdict, and it cannot be the worst of a set
|
||||
that contains real ones. "`SKIP` is never a pass" survives intact in the half
|
||||
of the rule that keeps it: an unregistered repo, or an empty finding set,
|
||||
still says `SKIP`, because there is nothing else to be worst of.
|
||||
|
||||
Measured across all 21 local clones from one saved sweep, so before and after
|
||||
are computed from the *same* findings rather than from two sweeps of a moving
|
||||
org: 343 findings before, 343 after, none added or removed. 8 repos moved,
|
||||
every one `SKIP` → `OK`. The counts that decide whether a repo needs work did
|
||||
not move at all — 1 `ERROR` before and after, 3 `WARN` before and after.
|
||||
|
||||
Consumers reading `.status` see a changed value domain: a repo that reported
|
||||
`SKIP` in censuses 03–07 may report `OK` from 08 on with an identical finding
|
||||
set. Raw census files already carry `engineVersion` and `engineCommit`, which
|
||||
is what distinguishes the two readings.
|
||||
|
||||
### Added
|
||||
|
||||
- `notChecked` beside `status`, in `--json` and as a `· N not checked` qualifier
|
||||
on the summary line. Fixing the status alone would have traded "clean repos
|
||||
look skipped" for "skipped checks look clean", which is the worse direction —
|
||||
so coverage gets its own axis rather than being folded into the verdict, the
|
||||
same split the engine already makes between a finding's level and its bucket.
|
||||
The count is the number of `SKIP` findings, which is what the existing
|
||||
`NOT CHECKED — these are not passes` section has always listed. An **absent**
|
||||
`notChecked` means an older engine and prints the pre-0.7.0 line; it is not
|
||||
zero.
|
||||
|
||||
## [0.6.0] — 2026-08-09
|
||||
|
||||
### Added
|
||||
|
|
|
|||
12
CLAUDE.md
12
CLAUDE.md
|
|
@ -28,7 +28,17 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
not belong here. It is measured where the org is enumerated.
|
||||
- **It records, it does not fix.** Findings first, remediation afterwards.
|
||||
Patching while measuring is how the inconsistency it detects was produced.
|
||||
- **`SKIP` is never a pass.** A check that could not run says so and names why.
|
||||
- **`SKIP` is never a pass — and never a severity either.** A check that could
|
||||
not run says so and names why. But a skip is the ABSENCE of a verdict, so it
|
||||
cannot be the worst of a set that contains real ones: a repo's `status` is the
|
||||
worst *judged* finding, and `SKIP` only when nothing was judged (unregistered,
|
||||
or an empty finding set). It used to outrank `OK`, which made five repos with
|
||||
0 ERROR, 0 WARN and a dozen OK headline as "skipped" in org-ops census 05 —
|
||||
`okr`, with the most OK in the org, read as unread. Fixing it in the status
|
||||
alone would have traded that for the worse direction, "skipped checks look
|
||||
clean", so coverage rides its own axis: `notChecked` in `--json`, a `· N not
|
||||
checked` qualifier on the summary line. Absent `notChecked` means an older
|
||||
engine and prints the pre-0.7.0 line — it is not zero.
|
||||
- **When a check fires many times in one repo, suspect the CHECK.** The first
|
||||
link pass produced ~30 findings against `llm-security` and all were noise —
|
||||
regexes in code spans, `file:` URLs, relative paths resolved against the wrong
|
||||
|
|
|
|||
15
README.md
15
README.md
|
|
@ -10,7 +10,7 @@ checks that surface in one repository and reports what it finds.
|
|||
|
||||
*AI-generated: all code produced by Claude Code through dialog-driven development.*
|
||||
|
||||

|
||||

|
||||

|
||||

|
||||

|
||||
|
|
@ -84,6 +84,17 @@ The process exits 1 on any `ERROR`. A `SKIP` means the check could not run — a
|
|||
unreachable forge, an untagged repo, a link that leaves the repository. It is
|
||||
not a pass, and the output prints those separately under a heading that says so.
|
||||
|
||||
A repository's own `status` is the worst of the findings that were *judged*, and
|
||||
`SKIP` only when nothing was. A skip is not a severity — it is the absence of a
|
||||
verdict, so it cannot outrank twelve checks that did run. Coverage rides beside
|
||||
it on its own axis: `notChecked` in `--json`, and a qualifier on the summary
|
||||
line, so a pass that had checks it could not run never prints unqualified.
|
||||
|
||||
```
|
||||
✓ okr [plugin] — OK · 1 not checked (repo-standard v0.7.0 @5884a64)
|
||||
· stranger — SKIP · 1 not checked (repo-standard v0.7.0 @5884a64)
|
||||
```
|
||||
|
||||
### Traits — a second axis
|
||||
|
||||
Class is structural. A **trait** is about what the code does, which no remote can
|
||||
|
|
@ -186,7 +197,7 @@ The header and the `--json` output carry the engine's version *and* the commit
|
|||
it ran from:
|
||||
|
||||
```
|
||||
✓ voyage [plugin] — OK (repo-standard v0.6.0 @a7276e6)
|
||||
✓ voyage [plugin] — OK (repo-standard v0.7.0 @a7276e6)
|
||||
```
|
||||
|
||||
Both, because they answer different questions. A stale plugin cache once served
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
{
|
||||
"name": "repo-standard",
|
||||
"version": "0.6.0",
|
||||
"version": "0.7.0",
|
||||
"private": true,
|
||||
"type": "module",
|
||||
"engines": {
|
||||
|
|
|
|||
|
|
@ -54,7 +54,13 @@ export function readEngineCommit(dir = join(HERE, '..')) {
|
|||
}
|
||||
}
|
||||
|
||||
const LEVELS = ['OK', 'SKIP', 'WARN', 'ERROR'];
|
||||
// The JUDGEMENT lattice, and `SKIP` is deliberately not in it. A skip is not a
|
||||
// severity — it is the absence of a verdict, so it cannot be the worst of a set
|
||||
// that contains real ones. It used to sit between OK and WARN here, which meant
|
||||
// a repo with 0 ERROR, 0 WARN and a dozen OK headlined as "skipped": five repos
|
||||
// in org-ops census 05, `okr` among them with the most OK in the org. Coverage
|
||||
// is carried on its own axis instead — see `notCheckedOf`.
|
||||
const LEVELS = ['OK', 'WARN', 'ERROR'];
|
||||
|
||||
// Findings carry a level AND a bucket, and the two are independent axes.
|
||||
// The level says how sure and how loud; the bucket says what KIND of problem it
|
||||
|
|
@ -893,12 +899,25 @@ export function checkInternalLinks({ files, present }) {
|
|||
return findings;
|
||||
}
|
||||
|
||||
// Worst of the judged findings; `SKIP` only when there is nothing to be worst
|
||||
// OF. That second half is what keeps "`SKIP` is never a pass" true: an
|
||||
// unregistered repo, or an empty finding set, still says so plainly. What the
|
||||
// rule no longer does is let one un-runnable check speak for twelve that ran.
|
||||
export function levelOf(findings) {
|
||||
let worst = 'OK';
|
||||
let worst = null;
|
||||
for (const f of findings ?? []) {
|
||||
if (LEVELS.indexOf(f.level) > LEVELS.indexOf(worst)) worst = f.level;
|
||||
if (f.level === 'SKIP') continue;
|
||||
if (worst === null || LEVELS.indexOf(f.level) > LEVELS.indexOf(worst)) worst = f.level;
|
||||
}
|
||||
return worst;
|
||||
return worst ?? 'SKIP';
|
||||
}
|
||||
|
||||
// The coverage axis, counted rather than left for each consumer to re-derive
|
||||
// from `findings`. Same reason `buckets` is precomputed beside `status`: a
|
||||
// number nobody can see reads exactly like a check that silently stopped
|
||||
// running.
|
||||
export function notCheckedOf(findings) {
|
||||
return (findings ?? []).filter((f) => f.level === 'SKIP').length;
|
||||
}
|
||||
|
||||
export function bucketsOf(findings) {
|
||||
|
|
@ -920,6 +939,7 @@ export function classifyRepo(
|
|||
klass: null,
|
||||
traits: [],
|
||||
status: 'SKIP',
|
||||
notChecked: 1,
|
||||
buckets: { broken: 0, missing: 0, weakening: 0 },
|
||||
findings: [{
|
||||
level: 'SKIP',
|
||||
|
|
@ -947,7 +967,15 @@ export function classifyRepo(
|
|||
...checkDescription(description, register),
|
||||
];
|
||||
|
||||
return { name, klass, traits, status: levelOf(findings), buckets: bucketsOf(findings), findings };
|
||||
return {
|
||||
name,
|
||||
klass,
|
||||
traits,
|
||||
status: levelOf(findings),
|
||||
notChecked: notCheckedOf(findings),
|
||||
buckets: bucketsOf(findings),
|
||||
findings,
|
||||
};
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------- I/O shell
|
||||
|
|
@ -1155,11 +1183,17 @@ const BUCKET_TITLE = {
|
|||
// engine is visible, not just correctable in hindsight.
|
||||
const MARK = { OK: '✓', WARN: '!', ERROR: '✗', SKIP: '·' };
|
||||
|
||||
// Both axes on the one line a sweep actually reads. Letting `status` mean
|
||||
// judgement fixed "clean repos look skipped"; printing a bare OK next to a
|
||||
// check that never ran would trade it for "skipped checks look clean", which is
|
||||
// the worse direction. Absent `notChecked` is neither zero nor a crash — a
|
||||
// result from before this axis existed prints the old line, not "undefined".
|
||||
export function headerLine(result, engineVersion, engineCommit = null) {
|
||||
const klass = result.klass ? ` [${result.klass}]` : '';
|
||||
const traits = result.traits?.length ? ` {${result.traits.join(', ')}}` : '';
|
||||
const sha = engineCommit ? ` @${String(engineCommit).slice(0, 7)}` : '';
|
||||
return `${MARK[result.status]} ${result.name}${klass}${traits} — ${result.status} (repo-standard v${engineVersion}${sha})`;
|
||||
const coverage = result.notChecked > 0 ? ` · ${result.notChecked} not checked` : '';
|
||||
return `${MARK[result.status]} ${result.name}${klass}${traits} — ${result.status}${coverage} (repo-standard v${engineVersion}${sha})`;
|
||||
}
|
||||
|
||||
// `engineCommit` is always present, null when underivable: an ABSENT key means
|
||||
|
|
|
|||
|
|
@ -567,17 +567,49 @@ test('every registered repo names a class that actually exists', () => {
|
|||
|
||||
// -------------------------------------------------------------- aggregation
|
||||
|
||||
test('levelOf ranks ERROR above WARN above SKIP above OK', () => {
|
||||
test('levelOf ranks ERROR above WARN above OK', () => {
|
||||
assert.equal(levelOf([{ level: 'OK' }, { level: 'WARN' }, { level: 'ERROR' }]), 'ERROR');
|
||||
assert.equal(levelOf([{ level: 'OK' }, { level: 'WARN' }]), 'WARN');
|
||||
assert.equal(levelOf([{ level: 'OK' }, { level: 'SKIP' }]), 'SKIP');
|
||||
assert.equal(levelOf([{ level: 'OK' }]), 'OK');
|
||||
assert.equal(levelOf([]), 'OK');
|
||||
});
|
||||
|
||||
// The defect this replaced: SKIP outranked OK, so a repo with 0 ERROR, 0 WARN
|
||||
// and a dozen OK headlined as "skipped". Reported by org-ops against five repos
|
||||
// in census 05; `okr` has the most OK in the org and read as unread. SKIP is not
|
||||
// a severity — it is the absence of a judgement, and it cannot be the worst of a
|
||||
// set that contains real ones.
|
||||
test('a SKIP does not outrank an OK — it is absence of judgement, not a severity', () => {
|
||||
assert.equal(levelOf([{ level: 'OK' }, { level: 'SKIP' }]), 'OK');
|
||||
assert.equal(levelOf([{ level: 'SKIP' }, { level: 'WARN' }]), 'WARN');
|
||||
assert.equal(levelOf([{ level: 'SKIP' }, { level: 'ERROR' }]), 'ERROR');
|
||||
});
|
||||
|
||||
// The other half of the same rule, and the half that keeps "`SKIP` is never a
|
||||
// pass" true: when there is nothing else to be worst OF, SKIP stands.
|
||||
test('SKIP still wins when nothing was judged at all', () => {
|
||||
assert.equal(levelOf([{ level: 'SKIP' }]), 'SKIP');
|
||||
assert.equal(levelOf([{ level: 'SKIP' }, { level: 'SKIP' }]), 'SKIP');
|
||||
});
|
||||
|
||||
// An empty finding set means no check ran, which is the same state as an
|
||||
// all-SKIP set — not a clean bill. It cannot occur for a registered repo, so
|
||||
// this pins a deliberate choice rather than an observed case.
|
||||
test('no findings at all is SKIP, not OK — nothing ran', () => {
|
||||
assert.equal(levelOf([]), 'SKIP');
|
||||
});
|
||||
|
||||
test('an unregistered repo is SKIP, not a pass — the gate refuses to guess a class', () => {
|
||||
const r = classifyRepo({ name: 'stranger', files: {}, present: [], description: null }, REGISTER);
|
||||
assert.equal(r.status, 'SKIP');
|
||||
assert.equal(r.notChecked, 1);
|
||||
});
|
||||
|
||||
// Coverage is its own axis, carried explicitly rather than left for each
|
||||
// consumer to re-derive from `findings`. An exemption nobody can see reads
|
||||
// exactly like a check that silently stopped running.
|
||||
test('notChecked counts the skipped checks beside the status, on every result', () => {
|
||||
const r = classifyRepo({ name: 'stranger', files: {}, present: [], description: null }, REGISTER);
|
||||
assert.equal(typeof r.notChecked, 'number');
|
||||
});
|
||||
|
||||
test('a fully compliant plugin repo classifies OK', () => {
|
||||
|
|
@ -1483,6 +1515,28 @@ test('headerLine falls back to the version alone when the commit is underivable'
|
|||
assert.doesNotMatch(line, /null|undefined/);
|
||||
});
|
||||
|
||||
// Letting `status` mean judgement fixes "clean repos look skipped"; printing an
|
||||
// unqualified OK beside a check that never ran would trade it for "skipped
|
||||
// checks look clean", which is the worse direction. The one line org-ops reads
|
||||
// has to carry both axes.
|
||||
test('headerLine qualifies a pass that had checks it could not run', () => {
|
||||
const line = headerLine({ name: 'okr', klass: 'plugin', traits: [], status: 'OK', notChecked: 1 }, '0.7.0');
|
||||
assert.match(line, /OK/);
|
||||
assert.match(line, /1 not checked/);
|
||||
});
|
||||
|
||||
test('headerLine says nothing about coverage when every check ran', () => {
|
||||
const line = headerLine({ name: 'okr', klass: 'plugin', traits: [], status: 'OK', notChecked: 0 }, '0.7.0');
|
||||
assert.doesNotMatch(line, /not checked/);
|
||||
});
|
||||
|
||||
// An older result object has no `notChecked` at all. Absent is not zero and not
|
||||
// a crash — it prints the pre-0.7.0 line rather than the word "undefined".
|
||||
test('headerLine tolerates a result from before the coverage axis existed', () => {
|
||||
const line = headerLine({ name: 'voyage', klass: 'plugin', traits: [], status: 'OK' }, '0.7.0');
|
||||
assert.doesNotMatch(line, /not checked|undefined|NaN/);
|
||||
});
|
||||
|
||||
test('withEngineVersion stamps the commit next to the version', () => {
|
||||
const stamped = withEngineVersion({ name: 'x', status: 'OK', findings: [] }, '0.5.0', 'a7276e6f0d1e2b3c');
|
||||
assert.equal(stamped.engineVersion, '0.5.0');
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue