fix(catalog): release gate can no longer be green without measuring anything
check-versions.mjs reported "12 SKIP, exit 0" on a fresh clone with no sibling plugin repos — a run that verified nothing was indistinguishable from a clean run. Any SKIP now fails the gate by default (--allow-skip opts in explicitly), and the summary always reports "verified N/M". Also adds a homepage-resolves check on plugin.json (dead-link ERROR, verified live against voyage's stale pre-polyrepo URL); the field stays optional since 11 of 12 plugins don't carry it yet. gateOutcome is split out as a pure function so both directions (all-SKIP fails, fully-verified-clean still passes) are unit-tested without needing a real sibling-repo layout on disk.
This commit is contained in:
parent
4509bb6096
commit
0b0bf24137
3 changed files with 210 additions and 13 deletions
24
CLAUDE.md
24
CLAUDE.md
|
|
@ -92,10 +92,28 @@ their own Forgejo repositories under `https://git.fromaitochitta.com/open/`.
|
|||
change. For each plugin it checks (against the sibling repo) that the catalog `ref` resolves to a
|
||||
real git tag (ERROR if dangling — breaks install), that `plugin.json` version == README
|
||||
version-badge (ERROR), that the catalog README's per-plugin `` `vX.Y.Z` `` label == the catalog
|
||||
`ref` (ERROR — the human-facing doc must not misstate the installed version), and that the catalog
|
||||
`ref` matches `plugin.json` version (WARN — catalog lags or an unreleased bump). Exit 1 on any
|
||||
ERROR; `--strict` also fails on WARN. Pure-function core covered by
|
||||
`ref` (ERROR — the human-facing doc must not misstate the installed version), that the catalog
|
||||
`ref` matches `plugin.json` version (WARN — catalog lags or an unreleased bump), and that
|
||||
`plugin.json`'s `homepage`, if present, actually resolves (ERROR if dead — a live `curl` check,
|
||||
synchronous by design, see the comment at `checkHomepage` for why it must never become async).
|
||||
Exit 1 on any ERROR; `--strict` also fails on WARN. Pure-function core covered by
|
||||
`scripts/check-versions.test.mjs` (`node --test scripts/check-versions.test.mjs`).
|
||||
- **`homepage` is OPTIONAL, not required** (decided 2026-08-26): measured that 11 of 12
|
||||
plugins omit the field. Requiring it would flip those 11 to ERROR and, via
|
||||
`release-plugin.mjs`'s ERROR-set pre-flight (catalog-wide), block every release in the
|
||||
marketplace until each of those 11 sibling repos re-tags — a marketplace-wide freeze
|
||||
delivered by a script fix. The gate only fires when the field is present AND resolves
|
||||
to a definitive 4xx/5xx; a network failure (timeout/DNS) reads as "not checked", never
|
||||
as a confirmed dead link.
|
||||
- **SKIP is UNVERIFIED, not clean, and now blocks the gate by default.** Measured 2026-08-18: a
|
||||
fresh clone with zero sibling repos printed `12 plugins — 0 OK, 0 WARN, 0 ERROR, 12 SKIP` at
|
||||
**exit 0** — a run that verified nothing was indistinguishable from a run that verified
|
||||
everything and found it clean (Verifiseringsloven ansikt 4, in the org's own release gate).
|
||||
Any SKIP now fails the run unless `--allow-skip` is passed explicitly, and the summary line
|
||||
always reports the denominator: `— verified N/M`. `runGate`'s `hasError`/`hasWarn`/`failed`
|
||||
fields are unchanged in meaning; the new SKIP-based failure is a separate `unverified` field
|
||||
folded into `failed` — `release-plugin.mjs`'s pre-flight (`preflightErrors`) reads the
|
||||
ERROR set directly off `results`, never `failed`, so it is untouched by this.
|
||||
- **Stat-badge mirroring (part of the same gate):** each plugin block in the catalog README ends in a
|
||||
stat line (`7 agents · 16 scanners · 21 commands · 1398 tests · [Full documentation →]`). The gate
|
||||
compares every number on that line against the plugin's own shields badge for the same axis, and
|
||||
|
|
|
|||
|
|
@ -7,10 +7,17 @@
|
|||
// - plugin.json version must equal the README version-badge → else ERROR (internal)
|
||||
// - catalog README per-plugin label must equal catalog ref → else ERROR (doc misstates install)
|
||||
// - catalog ref should equal plugin.json version → else WARN (catalog lags / unreleased bump)
|
||||
// - plugin.json homepage, if present, must resolve → else ERROR (dead link); field is OPTIONAL
|
||||
// - sibling repo missing → SKIP
|
||||
//
|
||||
// Exit 1 if any ERROR (or any WARN under --strict); 0 otherwise.
|
||||
// Usage: node scripts/check-versions.mjs [--strict]
|
||||
// SKIP is UNVERIFIED, not clean — a plugin that could not be checked at all is not the
|
||||
// same thing as a plugin that was checked and found fine. A run with any SKIP therefore
|
||||
// exits 1 by default (fixes: 2026-08-18 a fresh clone with zero sibling repos printed
|
||||
// "12 SKIP, exit 0" — a run that verified NOTHING read as green). Pass --allow-skip to
|
||||
// accept partial verification explicitly; the summary line always reports "verified N/M".
|
||||
//
|
||||
// Exit 1 if any ERROR, any SKIP (unless --allow-skip), or any WARN under --strict; 0 otherwise.
|
||||
// Usage: node scripts/check-versions.mjs [--strict] [--allow-skip]
|
||||
import { readFileSync, existsSync } from 'node:fs';
|
||||
import { execFileSync } from 'node:child_process';
|
||||
import { join, dirname } from 'node:path';
|
||||
|
|
@ -136,7 +143,7 @@ export function extractCatalogStats(readmeText, name) {
|
|||
|
||||
// Pure classifier — all I/O is resolved into the input shape before this is called.
|
||||
// tags === null means "repo not inspected" (missing locally); [] means "no tags".
|
||||
export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags, catalogLabel = null, statBadges = null, catalogStats = null }) {
|
||||
export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags, catalogLabel = null, statBadges = null, catalogStats = null, homepage = null, homepageReachable = null }) {
|
||||
if (pluginVersion === null && tags === null) {
|
||||
return { name, status: 'SKIP', findings: [{ level: 'SKIP', msg: 'plugin repo not found locally — cannot verify' }] };
|
||||
}
|
||||
|
|
@ -192,6 +199,18 @@ export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, t
|
|||
}
|
||||
}
|
||||
|
||||
// 6. plugin.json homepage must resolve when present — the first link an agent follows
|
||||
// from a manifest. The field stays OPTIONAL (measured 2026-08-18: 11 of 12 plugins omit
|
||||
// it) — a deliberate policy call, not an oversight. Making it required would flip those
|
||||
// 11 to ERROR and, via release-plugin.mjs's ERROR-set pre-flight (catalog-wide), block
|
||||
// every release in the marketplace until each of those 11 sibling repos re-tags. That is
|
||||
// a marketplace-wide freeze delivered by a script fix, not a gate improvement.
|
||||
// `homepageReachable === null` means "not checked" (e.g. network unreachable) and must
|
||||
// NOT read as either pass or fail.
|
||||
if (homepage !== null && homepageReachable === false) {
|
||||
findings.push({ level: 'ERROR', msg: `plugin.json homepage ${homepage} does not resolve (dead link)` });
|
||||
}
|
||||
|
||||
const status = findings.some(f => f.level === 'ERROR') ? 'ERROR'
|
||||
: findings.some(f => f.level === 'WARN') ? 'WARN'
|
||||
: 'OK';
|
||||
|
|
@ -219,6 +238,30 @@ function gitTags(repoDir) {
|
|||
}
|
||||
}
|
||||
|
||||
// Synchronous on purpose (curl via execFileSync, not fetch): inspectPlugin/runGate must
|
||||
// stay synchronous because release-plugin.mjs's applyRelease calls `io.runGate(catalogDir)`
|
||||
// without awaiting it. Turning runGate async would make that call site silently receive a
|
||||
// Promise — preflightErrors reads `gateResult?.results ?? []`, so a Promise (no `.results`)
|
||||
// resolves to an empty error list and the pre-flight gate stops blocking ANY release,
|
||||
// forever, without a single visible failure. That is the exact "unmeasured reads as green"
|
||||
// defect this order exists to fix, one level away — so this stays synchronous.
|
||||
// Returns true/false only on a definitive HTTP response; null ("not checked") on any
|
||||
// network failure — timeouts and DNS errors must not read as a confirmed dead link.
|
||||
function checkHomepage(url) {
|
||||
if (!url) return null;
|
||||
try {
|
||||
const out = execFileSync(
|
||||
'curl', ['-s', '-o', '/dev/null', '-w', '%{http_code}', '--max-time', '5', '-L', url],
|
||||
{ encoding: 'utf8' },
|
||||
).trim();
|
||||
const code = Number(out);
|
||||
if (!Number.isFinite(code) || code === 0) return null;
|
||||
return code < 400;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
// I/O shell — resolve one plugin's observed state from disk, then classify.
|
||||
export function inspectPlugin(catalogDir, plugin) {
|
||||
const name = plugin.name;
|
||||
|
|
@ -230,10 +273,13 @@ export function inspectPlugin(catalogDir, plugin) {
|
|||
}
|
||||
|
||||
let pluginVersion = null;
|
||||
let homepage = null;
|
||||
try {
|
||||
const pj = JSON.parse(readFileSync(join(repoDir, '.claude-plugin', 'plugin.json'), 'utf8'));
|
||||
pluginVersion = pj.version ?? null;
|
||||
homepage = pj.homepage ?? null;
|
||||
} catch { /* leave null → flagged */ }
|
||||
const homepageReachable = checkHomepage(homepage);
|
||||
|
||||
let worktreeReadme = null;
|
||||
try {
|
||||
|
|
@ -255,22 +301,44 @@ export function inspectPlugin(catalogDir, plugin) {
|
|||
catalogStats = extractCatalogStats(catalogReadme, name);
|
||||
} catch { /* no catalog README → label + stat checks skipped */ }
|
||||
|
||||
return classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags: gitTags(repoDir), catalogLabel, statBadges, catalogStats });
|
||||
return classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags: gitTags(repoDir), catalogLabel, statBadges, catalogStats, homepage, homepageReachable });
|
||||
}
|
||||
|
||||
export function runGate(catalogDir, { strict = false } = {}) {
|
||||
const mkt = JSON.parse(readFileSync(join(catalogDir, '.claude-plugin', 'marketplace.json'), 'utf8'));
|
||||
const results = (mkt.plugins || []).map(p => inspectPlugin(catalogDir, p));
|
||||
// Pure — takes classified results, not I/O. Split out of runGate so both directions of
|
||||
// the SKIP-vs-green fix are unit-testable without needing a real sibling-repo layout on
|
||||
// disk: a run that verified NOTHING
|
||||
// (all SKIP) must fail even with zero ERROR/WARN, and a fully-verified clean run must
|
||||
// still pass — a fix that only reddens the former without proving the latter would be
|
||||
// unverified in the other direction.
|
||||
export function gateOutcome(results, { strict = false, allowSkip = false } = {}) {
|
||||
const total = results.length;
|
||||
const skipped = results.filter(r => r.status === 'SKIP').length;
|
||||
const hasError = results.some(r => r.status === 'ERROR');
|
||||
const hasWarn = results.some(r => r.status === 'WARN');
|
||||
return { results, hasError, hasWarn, failed: hasError || (strict && hasWarn) };
|
||||
const unverified = skipped > 0 && !allowSkip;
|
||||
return {
|
||||
total,
|
||||
skipped,
|
||||
verified: total - skipped,
|
||||
hasError,
|
||||
hasWarn,
|
||||
unverified,
|
||||
failed: hasError || (strict && hasWarn) || unverified,
|
||||
};
|
||||
}
|
||||
|
||||
export function runGate(catalogDir, { strict = false, allowSkip = false } = {}) {
|
||||
const mkt = JSON.parse(readFileSync(join(catalogDir, '.claude-plugin', 'marketplace.json'), 'utf8'));
|
||||
const results = (mkt.plugins || []).map(p => inspectPlugin(catalogDir, p));
|
||||
return { results, ...gateOutcome(results, { strict, allowSkip }) };
|
||||
}
|
||||
|
||||
const __filename = fileURLToPath(import.meta.url);
|
||||
if (process.argv[1] && pathToFileURL(process.argv[1]).href === import.meta.url) {
|
||||
const strict = process.argv.includes('--strict');
|
||||
const allowSkip = process.argv.includes('--allow-skip');
|
||||
const catalogDir = join(dirname(__filename), '..');
|
||||
const { results, failed } = runGate(catalogDir, { strict });
|
||||
const { results, failed, skipped, total, verified } = runGate(catalogDir, { strict, allowSkip });
|
||||
|
||||
const icon = { OK: '✓', WARN: '⚠', ERROR: '✗', SKIP: '–' };
|
||||
for (const r of results) {
|
||||
|
|
@ -279,6 +347,10 @@ if (process.argv[1] && pathToFileURL(process.argv[1]).href === import.meta.url)
|
|||
}
|
||||
const count = (s) => results.filter(r => r.status === s).length;
|
||||
console.log('');
|
||||
console.log(`${results.length} plugins — ${count('OK')} OK, ${count('WARN')} WARN, ${count('ERROR')} ERROR, ${count('SKIP')} SKIP${strict ? ' (strict)' : ''}`);
|
||||
console.log(`${results.length} plugins — ${count('OK')} OK, ${count('WARN')} WARN, ${count('ERROR')} ERROR, ${count('SKIP')} SKIP${strict ? ' (strict)' : ''} — verified ${verified}/${total}`);
|
||||
if (skipped > 0 && !allowSkip) {
|
||||
console.log(`✗ ${skipped} plugin(s) UNVERIFIED (sibling repo not found locally) — this is NOT a green run.`);
|
||||
console.log(` Clone the missing sibling repos next to this one, or pass --allow-skip to accept partial verification explicitly.`);
|
||||
}
|
||||
process.exit(failed ? 1 : 0);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,6 +11,7 @@ import {
|
|||
extractStatBadges,
|
||||
pickStatSource,
|
||||
extractCatalogStats,
|
||||
gateOutcome,
|
||||
} from './check-versions.mjs';
|
||||
|
||||
test('normalizeVersion strips a leading v', () => {
|
||||
|
|
@ -292,3 +293,109 @@ test('pickStatSource falls back to the working tree only when the ref cannot be
|
|||
test('an empty README at the ref is a real read, not a failed one', () => {
|
||||
assert.equal(pickStatSource({ atRef: '', atWorktree: 'badge/scanners-22-cyan' }), '');
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// gateOutcome — the fix for the UNMEASURED-reads-as-green bug found 2026-08-18.
|
||||
// A fresh clone with no sibling repos cloned produced "12 plugins — 0 OK, 0
|
||||
// WARN, 0 ERROR, 12 SKIP, exit 0": a run that verified NOTHING was
|
||||
// indistinguishable from a run that verified everything and found it clean.
|
||||
// Both directions must hold, or a fix that only
|
||||
// reddens the all-SKIP case could just as easily redden (or silently pass) the
|
||||
// all-verified case too.
|
||||
|
||||
test('gateOutcome: all-SKIP (nothing verified) fails even with zero ERROR/WARN', () => {
|
||||
const results = [{ status: 'SKIP' }, { status: 'SKIP' }, { status: 'SKIP' }];
|
||||
const o = gateOutcome(results);
|
||||
assert.equal(o.hasError, false);
|
||||
assert.equal(o.skipped, 3);
|
||||
assert.equal(o.verified, 0);
|
||||
assert.equal(o.total, 3);
|
||||
assert.equal(o.failed, true, 'a run that verified nothing must not report green');
|
||||
});
|
||||
|
||||
test('gateOutcome: fully-verified all-OK run stays green (the fix must not break the normal case)', () => {
|
||||
const results = [{ status: 'OK' }, { status: 'OK' }, { status: 'WARN' }];
|
||||
const o = gateOutcome(results);
|
||||
assert.equal(o.skipped, 0);
|
||||
assert.equal(o.verified, 3);
|
||||
assert.equal(o.failed, false, 'no ERROR, no SKIP, non-strict → still green');
|
||||
});
|
||||
|
||||
test('gateOutcome: ERROR still fails regardless of skip count', () => {
|
||||
const o = gateOutcome([{ status: 'ERROR' }, { status: 'SKIP' }]);
|
||||
assert.equal(o.failed, true);
|
||||
});
|
||||
|
||||
test('gateOutcome: --allow-skip lets a partially-unverified run pass deliberately', () => {
|
||||
const results = [{ status: 'OK' }, { status: 'SKIP' }];
|
||||
const withoutFlag = gateOutcome(results);
|
||||
const withFlag = gateOutcome(results, { allowSkip: true });
|
||||
assert.equal(withoutFlag.failed, true, 'default: unverified plugin blocks green');
|
||||
assert.equal(withFlag.failed, false, 'explicit opt-in: caller accepts partial verification');
|
||||
});
|
||||
|
||||
test('gateOutcome: strict mode still fails on WARN even with everything verified', () => {
|
||||
const o = gateOutcome([{ status: 'WARN' }], { strict: true });
|
||||
assert.equal(o.failed, true);
|
||||
});
|
||||
|
||||
test('gateOutcome: verified/total denominator is reported for the summary line', () => {
|
||||
const o = gateOutcome([{ status: 'OK' }, { status: 'OK' }, { status: 'SKIP' }]);
|
||||
assert.equal(o.total, 3);
|
||||
assert.equal(o.verified, 2);
|
||||
assert.equal(o.skipped, 1);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Manifest `homepage` must resolve when present (order finding 2 — voyage's
|
||||
// manifest pointed at a pre-polyrepo path that 404s). The field stays OPTIONAL
|
||||
// (11 of 12 plugins omit it) — this is a deliberate policy call made in this
|
||||
// session, not an oversight: making it required would flip 11 plugins to ERROR
|
||||
// and, via release-plugin.mjs's ERROR-set pre-flight, block every release in
|
||||
// the marketplace until all 11 sibling repos re-tag. See check-versions.mjs
|
||||
// for the same note at the check site.
|
||||
|
||||
test('homepage present and reachable → no finding (known-positive)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'voyage', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], homepage: 'https://git.fromaitochitta.com/open/voyage', homepageReachable: true,
|
||||
});
|
||||
assert.equal(r.status, 'OK');
|
||||
assert.ok(!r.findings.some(f => /homepage/.test(f.msg)));
|
||||
});
|
||||
|
||||
test('homepage present and dead → ERROR (known-negative, the voyage 404 case)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'voyage', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'],
|
||||
homepage: 'https://git.fromaitochitta.com/open/ktg-plugin-marketplace/src/branch/main/plugins/voyage',
|
||||
homepageReachable: false,
|
||||
});
|
||||
assert.equal(r.status, 'ERROR');
|
||||
assert.ok(r.findings.some(f => f.level === 'ERROR' && /homepage/.test(f.msg) && /does not resolve/.test(f.msg)));
|
||||
});
|
||||
|
||||
test('homepage absent → no finding (field is optional, not every plugin must carry it)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'config-audit', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], homepage: null, homepageReachable: null,
|
||||
});
|
||||
assert.equal(r.status, 'OK');
|
||||
assert.ok(!r.findings.some(f => /homepage/.test(f.msg)));
|
||||
});
|
||||
|
||||
test('homepage present but unchecked (network unreachable) → not silently treated as reachable, but does not false-fail', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'voyage', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], homepage: 'https://git.fromaitochitta.com/open/voyage', homepageReachable: null,
|
||||
});
|
||||
assert.equal(r.status, 'OK', 'unmeasured must not be reported as a confirmed dead link');
|
||||
assert.ok(!r.findings.some(f => /homepage/.test(f.msg)));
|
||||
});
|
||||
|
||||
test('homepage omitted entirely (legacy callers) → check skipped, stays OK', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'x', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0', tags: ['v1.0.0'],
|
||||
});
|
||||
assert.equal(r.status, 'OK');
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue