feat(check-versions): gate the catalog's stat lines against plugin badges [skip-docs]
[skip-docs]: the user-facing doc here is CLAUDE.md, not README.md. This is a
maintainer-side consistency gate — the catalog landing page should not describe
it — and CLAUDE.md gains a full paragraph covering the rule, the measured
badge-less gap, and the never-hand-edit-a-stat-line instruction. README.md
changes by exactly one number because the gate found it wrong.
The catalog restates each plugin's counts in a per-plugin stat line, and those
numbers rot silently: nothing compared them to anything. Measured across all 11
plugin READMEs today, config-audit's line claimed 1410 tests while the plugin's
own badge said 1441.
Rule is PER-AXIS, not per-plugin. For each number on a catalog stat line, if the
plugin carries a shields badge for that axis, they must agree (ERROR otherwise);
if it carries no such badge, the axis is skipped. This was measured, not assumed:
21 axis-pairs are badge-covered, but 14 axes across 8 of the 11 plugins are
badge-less (voyage 4, ms-ai-architect 3, repo-mailbox 2, ai-psychosis 2,
linkedin-studio 1, claude-design 1, graceful-handoff 1). A per-plugin exception
list — the original sketch, scoped around voyage alone — would have had to name 8
of 11 repos and be hand-edited for every new axis.
Stated plainly rather than hidden: those 14 axes stay ungated. repo-mailbox's two
were 6 and 251 against a true 8 and 374 (fixed in e9054f8).
Parsing is measured against the real corpus: shields' `--` literal-hyphen escape,
`_`/`%20` spaces, zero as a real count, non-numeric badges (version/platform/
license) excluded, parenthetical asides read as their own axis ("5 skills (389
docs)"), doc/docs/references/reference_docs/knowledge_docs folded onto one axis,
and `N+` read as a lower bound.
Only the catalog's config-audit number changed; the plugin badge is the source.
Tests 106 -> 117 (+11). Gate green at 11 OK / 0 WARN / 0 ERROR.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RDSiMcgLMpEETwtkc86Nym
This commit is contained in:
parent
e9054f8452
commit
fc95cbd300
4 changed files with 255 additions and 7 deletions
10
CLAUDE.md
10
CLAUDE.md
|
|
@ -37,3 +37,13 @@ their own Forgejo repositories under `https://git.fromaitochitta.com/open/`.
|
|||
`ref` matches `plugin.json` version (WARN — catalog lags or an unreleased bump). Exit 1 on any
|
||||
ERROR; `--strict` also fails on WARN. Pure-function core covered by
|
||||
`scripts/check-versions.test.mjs` (`node --test scripts/check-versions.test.mjs`).
|
||||
- **Stat-badge mirroring (part of the same gate):** each plugin block in the catalog README ends in a
|
||||
stat line (`7 agents · 16 scanners · 21 commands · 1441 tests · [Full documentation →]`). The gate
|
||||
compares every number on that line against the plugin's own shields badge for the same axis, and
|
||||
ERRORs when they disagree — the catalog must not overstate a plugin. **The rule is per-AXIS, not
|
||||
per-plugin:** an axis the plugin does not badge is skipped silently, so there is no exception list
|
||||
to maintain. Measured 2026-08-02: 21 axis-pairs are badge-covered, and **14 axes across 8 of the 11
|
||||
plugins are badge-less** and therefore ungated — those numbers still rot and need a human pass
|
||||
(repo-mailbox's read `6 CLI scripts` / `251 selftest checks` against a true 8 / 374). `N+` in the
|
||||
catalog is read as a lower bound, not an equality. **Never hand-edit a stat line to silence the
|
||||
gate** — the plugin's badge is the source for every stat number; fix the catalog to match it.
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ Configuration intelligence for Claude Code. Claude reads instructions from 7+ fi
|
|||
|
||||
Key commands: `/config-audit posture`, `/config-audit feature-gap`, `/config-audit fix`, `/config-audit whats-active`, `/config-audit tokens`
|
||||
|
||||
7 agents · 16 scanners · 21 commands · 1410 tests · [Full documentation →](https://git.fromaitochitta.com/open/config-audit)
|
||||
7 agents · 16 scanners · 21 commands · 1441 tests · [Full documentation →](https://git.fromaitochitta.com/open/config-audit)
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -39,9 +39,80 @@ export function extractCatalogLabel(readmeText, name) {
|
|||
return null;
|
||||
}
|
||||
|
||||
// --- stat-badge mirroring -------------------------------------------------
|
||||
//
|
||||
// The catalog restates each plugin's counts in a per-plugin stat line
|
||||
// ("7 agents · 16 scanners · 21 commands · 1410 tests · [Full documentation →]").
|
||||
// Those numbers rot: measured 2026-08-02, config-audit's line said 1410 tests
|
||||
// while the plugin's own badge said 1441. This mirrors catalog line -> plugin badge.
|
||||
//
|
||||
// The rule is PER-AXIS, not per-plugin. An axis is gated only when the plugin
|
||||
// carries a badge for it; a badge-less axis is skipped silently. That is not a
|
||||
// concession to voyage (the one plugin with no stat badges at all) — 14 axes
|
||||
// across 8 of the 11 plugins are badge-less, so a per-plugin exception list
|
||||
// would have had to name most of the marketplace and be edited by hand forever.
|
||||
// The cost is stated plainly: those 14 numbers stay ungated and need a human
|
||||
// pass (repo-mailbox's two were 6 and 251 against a true 8 and 374 when measured).
|
||||
|
||||
const AXIS_SYNONYM = { 'reference doc': 'reference', 'knowledge doc': 'reference', doc: 'reference' };
|
||||
|
||||
// Fold a badge label or a stat-line noun onto a shared axis name.
|
||||
export function normalizeAxis(raw) {
|
||||
let s = String(raw || '').replace(/--/g, '-').replace(/%20/g, ' ').replace(/_/g, ' ')
|
||||
.toLowerCase().replace(/\s+/g, ' ').trim();
|
||||
s = s.replace(/\s+files?$/, '');
|
||||
const words = s.split(' ');
|
||||
const last = words[words.length - 1];
|
||||
if (last.length > 3 && last.endsWith('s') && !last.endsWith('ss')) words[words.length - 1] = last.slice(0, -1);
|
||||
s = words.join(' ');
|
||||
return AXIS_SYNONYM[s] || s;
|
||||
}
|
||||
|
||||
// Plugin README -> Map(axis -> count). Only badges whose VALUE is an integer are
|
||||
// stat axes; version/platform/license carry text and are ignored. Shields escapes
|
||||
// a literal hyphen as `--`, so the label/value/color split must ignore doubled ones.
|
||||
export function extractStatBadges(readmeText) {
|
||||
const out = new Map();
|
||||
for (const m of String(readmeText || '').matchAll(/img\.shields\.io\/badge\/([^)\s]+)/g)) {
|
||||
const parts = m[1].split(/(?<!-)-(?!-)/);
|
||||
if (parts.length < 3) continue;
|
||||
if (!/^\d+$/.test(parts[1])) continue;
|
||||
out.set(normalizeAxis(parts[0]), Number(parts[1]));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Pull "<n> <axis>" pairs out of one `·`-separated segment, including any
|
||||
// parenthetical aside ("5 skills (389 docs)" is two axes; "(+1 helper)" is one).
|
||||
function readStatSegment(segment, out) {
|
||||
const asides = [...segment.matchAll(/\(([^)]*)\)/g)].map(a => a[1]);
|
||||
const main = segment.replace(/\([^)]*\)/g, ' ');
|
||||
for (const piece of [main, ...asides]) {
|
||||
const m = /(\d+)(\+?)\s+([A-Za-z][A-Za-z -]*)/.exec(piece);
|
||||
if (!m) continue;
|
||||
const axis = normalizeAxis(m[3]);
|
||||
if (!axis || out.has(axis)) continue;
|
||||
out.set(axis, m[2] === '+' ? { atLeast: Number(m[1]) } : Number(m[1]));
|
||||
}
|
||||
}
|
||||
|
||||
// Catalog README -> Map(axis -> count | {atLeast}). Reads ONLY the stat line of the
|
||||
// named plugin's block, so prose numbers above it can never be mistaken for counts.
|
||||
export function extractCatalogStats(readmeText, name) {
|
||||
const lines = String(readmeText || '').split('\n');
|
||||
const out = new Map();
|
||||
let inBlock = false;
|
||||
for (const line of lines) {
|
||||
if (line.startsWith('### [')) inBlock = line.includes(`/open/${name})`);
|
||||
if (!inBlock || !line.includes('[Full documentation')) continue;
|
||||
for (const seg of line.split('[Full documentation')[0].split('·')) readStatSegment(seg, out);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
// Pure classifier — all I/O is resolved into the input shape before this is called.
|
||||
// tags === null means "repo not inspected" (missing locally); [] means "no tags".
|
||||
export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags, catalogLabel = null }) {
|
||||
export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags, catalogLabel = null, statBadges = null, catalogStats = null }) {
|
||||
if (pluginVersion === null && tags === null) {
|
||||
return { name, status: 'SKIP', findings: [{ level: 'SKIP', msg: 'plugin repo not found locally — cannot verify' }] };
|
||||
}
|
||||
|
|
@ -77,6 +148,26 @@ export function classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, t
|
|||
findings.push({ level: 'WARN', msg: `catalog ref ${catalogRef} != plugin.json version ${pluginVersion} (${reason})` });
|
||||
}
|
||||
|
||||
// 5. catalog stat line vs the plugin's own stat badges — per-axis, badge-gated.
|
||||
// An axis the plugin does not badge is SKIPPED (no claim to mirror). An axis the
|
||||
// badge carries but the catalog does not restate is likewise nothing: the catalog
|
||||
// chooses what to show. Only a stated number contradicting a badged one is a defect.
|
||||
if (statBadges !== null && catalogStats !== null) {
|
||||
for (const [axis, stated] of catalogStats) {
|
||||
if (!statBadges.has(axis)) continue;
|
||||
const badge = statBadges.get(axis);
|
||||
const approx = stated !== null && typeof stated === 'object';
|
||||
const bad = approx ? badge < stated.atLeast : badge !== stated;
|
||||
if (bad) {
|
||||
const shown = approx ? `${stated.atLeast}+` : String(stated);
|
||||
findings.push({
|
||||
level: 'ERROR',
|
||||
msg: `catalog says ${shown} ${axis} but the plugin's badge says ${badge} (catalog stat line is stale)`,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const status = findings.some(f => f.level === 'ERROR') ? 'ERROR'
|
||||
: findings.some(f => f.level === 'WARN') ? 'WARN'
|
||||
: 'OK';
|
||||
|
|
@ -110,16 +201,22 @@ export function inspectPlugin(catalogDir, plugin) {
|
|||
} catch { /* leave null → flagged */ }
|
||||
|
||||
let readmeBadge = null;
|
||||
let statBadges = null;
|
||||
try {
|
||||
readmeBadge = extractBadgeVersion(readFileSync(join(repoDir, 'README.md'), 'utf8'));
|
||||
} catch { /* no README → badge check skipped */ }
|
||||
const pluginReadme = readFileSync(join(repoDir, 'README.md'), 'utf8');
|
||||
readmeBadge = extractBadgeVersion(pluginReadme);
|
||||
statBadges = extractStatBadges(pluginReadme);
|
||||
} catch { /* no README → badge + stat checks skipped */ }
|
||||
|
||||
let catalogLabel = null;
|
||||
let catalogStats = null;
|
||||
try {
|
||||
catalogLabel = extractCatalogLabel(readFileSync(join(catalogDir, 'README.md'), 'utf8'), name);
|
||||
} catch { /* no catalog README → label check skipped */ }
|
||||
const catalogReadme = readFileSync(join(catalogDir, 'README.md'), 'utf8');
|
||||
catalogLabel = extractCatalogLabel(catalogReadme, name);
|
||||
catalogStats = extractCatalogStats(catalogReadme, name);
|
||||
} catch { /* no catalog README → label + stat checks skipped */ }
|
||||
|
||||
return classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags: gitTags(repoDir), catalogLabel });
|
||||
return classifyPlugin({ name, catalogRef, pluginVersion, readmeBadge, tags: gitTags(repoDir), catalogLabel, statBadges, catalogStats });
|
||||
}
|
||||
|
||||
export function runGate(catalogDir, { strict = false } = {}) {
|
||||
|
|
|
|||
|
|
@ -8,6 +8,8 @@ import {
|
|||
extractBadgeVersion,
|
||||
extractCatalogLabel,
|
||||
classifyPlugin,
|
||||
extractStatBadges,
|
||||
extractCatalogStats,
|
||||
} from './check-versions.mjs';
|
||||
|
||||
test('normalizeVersion strips a leading v', () => {
|
||||
|
|
@ -128,3 +130,142 @@ test('ERROR dominates WARN when both apply', () => {
|
|||
});
|
||||
assert.equal(r.status, 'ERROR');
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Stat-badge mirroring (added 2026-08-02, after measuring all 11 plugin READMEs
|
||||
// + the catalog's 11 stat lines). Rule chosen by the operator: PER-AXIS. An axis
|
||||
// is gated only when the plugin carries a matching badge; a badge-less axis is
|
||||
// skipped, not flagged. Measured basis: 21 axis-pairs are badge-covered, and 14
|
||||
// axes across 8 plugins are badge-less (voyage 4, ms-ai-architect 3,
|
||||
// repo-mailbox 2, ai-psychosis 2, linkedin-studio 1, claude-design 1,
|
||||
// graceful-handoff 1) — so a hardcoded per-PLUGIN exception list would have had
|
||||
// to name 8 of 11 repos.
|
||||
|
||||
test('extractStatBadges reads numeric shields badges, ignoring non-numeric ones', () => {
|
||||
const readme = [
|
||||
'',
|
||||
'',
|
||||
'',
|
||||
'',
|
||||
'',
|
||||
].join('\n');
|
||||
const s = extractStatBadges(readme);
|
||||
assert.equal(s.get('command'), 20);
|
||||
assert.equal(s.get('test'), 2013);
|
||||
// version/platform/license carry non-numeric values → never stat axes
|
||||
assert.equal(s.has('version'), false);
|
||||
assert.equal(s.has('platform'), false);
|
||||
assert.equal(s.has('license'), false);
|
||||
});
|
||||
|
||||
test('extractStatBadges handles a zero value and the -- literal-hyphen escape', () => {
|
||||
const readme = [
|
||||
'',
|
||||
'',
|
||||
].join('\n');
|
||||
const s = extractStatBadges(readme);
|
||||
assert.equal(s.get('hook'), 0, '0 is a real count, not a missing badge');
|
||||
assert.equal(s.has('state-helper'), false, 'non-numeric value → not a stat axis');
|
||||
});
|
||||
|
||||
test('extractCatalogStats parses the per-plugin stat line into axis counts', () => {
|
||||
const cat = [
|
||||
'### [Config-Audit](https://git.fromaitochitta.com/open/config-audit) `v5.13.0`',
|
||||
'',
|
||||
'Some prose with 99 red herrings in it.',
|
||||
'',
|
||||
'7 agents · 16 scanners · 21 commands · 1410 tests · [Full documentation →](https://x)',
|
||||
'',
|
||||
'---',
|
||||
].join('\n');
|
||||
const s = extractCatalogStats(cat, 'config-audit');
|
||||
assert.equal(s.get('agent'), 7);
|
||||
assert.equal(s.get('scanner'), 16);
|
||||
assert.equal(s.get('command'), 21);
|
||||
assert.equal(s.get('test'), 1410);
|
||||
assert.equal(s.has('red herring'), false, 'prose above the stat line must not leak in');
|
||||
});
|
||||
|
||||
test('extractCatalogStats reads a parenthetical count as its own axis', () => {
|
||||
const cat = [
|
||||
'### [MS AI Architect](https://git.fromaitochitta.com/open/ms-ai-architect) `v1.17.0`',
|
||||
'',
|
||||
'12 agents · 29 commands · 5 skills (389 docs) · 2 hooks · [Full documentation →](https://x)',
|
||||
].join('\n');
|
||||
const s = extractCatalogStats(cat, 'ms-ai-architect');
|
||||
assert.equal(s.get('skill'), 5);
|
||||
assert.equal(s.get('reference'), 389, 'doc/docs normalizes onto the reference axis');
|
||||
assert.equal(s.get('hook'), 2);
|
||||
});
|
||||
|
||||
test('extractCatalogStats returns an empty map when the plugin has no entry', () => {
|
||||
assert.equal(extractCatalogStats('### [Other](https://x/open/other) `v1.0.0`', 'absent').size, 0);
|
||||
});
|
||||
|
||||
test('stat mismatch on a badge-covered axis → ERROR (the config-audit 1441/1410 case)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'config-audit', catalogRef: 'v5.13.0', pluginVersion: '5.13.0',
|
||||
readmeBadge: '5.13.0', tags: ['v5.13.0'], catalogLabel: '5.13.0',
|
||||
statBadges: new Map([['test', 1441], ['agent', 7]]),
|
||||
catalogStats: new Map([['test', 1410], ['agent', 7]]),
|
||||
});
|
||||
assert.equal(r.status, 'ERROR');
|
||||
assert.ok(r.findings.some(f => f.level === 'ERROR' && /test/.test(f.msg) && /1441/.test(f.msg) && /1410/.test(f.msg)));
|
||||
});
|
||||
|
||||
test('every stat axis agreeing → stays OK', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'okr', catalogRef: 'v1.8.2', pluginVersion: '1.8.2',
|
||||
readmeBadge: '1.8.2', tags: ['v1.8.2'], catalogLabel: '1.8.2',
|
||||
statBadges: new Map([['agent', 7], ['command', 14], ['hook', 3], ['reference', 17]]),
|
||||
catalogStats: new Map([['agent', 7], ['command', 14], ['hook', 3]]),
|
||||
});
|
||||
assert.equal(r.status, 'OK', 'a badge the catalog simply does not restate is not a finding');
|
||||
});
|
||||
|
||||
test('badge-less catalog axis is SKIPPED, not flagged (the whole voyage case)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'voyage', catalogRef: 'v5.9.1', pluginVersion: '5.9.1',
|
||||
readmeBadge: '5.9.1', tags: ['v5.9.1'], catalogLabel: '5.9.1',
|
||||
statBadges: new Map(),
|
||||
catalogStats: new Map([['agent', 24], ['command', 6], ['hook', 7], ['test', 500]]),
|
||||
});
|
||||
assert.equal(r.status, 'OK');
|
||||
assert.ok(!r.findings.some(f => f.level === 'ERROR'), 'no badge → no claim to mirror → no error');
|
||||
});
|
||||
|
||||
test('a zero badge still gates (0 != 3 is a real mismatch, not a missing badge)', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'p', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], catalogLabel: '1.0.0',
|
||||
statBadges: new Map([['hook', 0]]),
|
||||
catalogStats: new Map([['hook', 3]]),
|
||||
});
|
||||
assert.equal(r.status, 'ERROR');
|
||||
});
|
||||
|
||||
test('an approximate catalog count (N+) gates as a LOWER BOUND', () => {
|
||||
const under = classifyPlugin({
|
||||
name: 'p', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], catalogLabel: '1.0.0',
|
||||
statBadges: new Map([['test', 400]]),
|
||||
catalogStats: new Map([['test', { atLeast: 500 }]]),
|
||||
});
|
||||
assert.equal(under.status, 'ERROR', '"500+ tests" while the badge says 400 overstates');
|
||||
|
||||
const over = classifyPlugin({
|
||||
name: 'p', catalogRef: 'v1.0.0', pluginVersion: '1.0.0', readmeBadge: '1.0.0',
|
||||
tags: ['v1.0.0'], catalogLabel: '1.0.0',
|
||||
statBadges: new Map([['test', 2013]]),
|
||||
catalogStats: new Map([['test', { atLeast: 500 }]]),
|
||||
});
|
||||
assert.equal(over.status, 'OK', '"500+" is satisfied by any badge >= 500');
|
||||
});
|
||||
|
||||
test('stat maps omitted (legacy callers) → mirroring skipped entirely', () => {
|
||||
const r = classifyPlugin({
|
||||
name: 'p', catalogRef: 'v1.0.0', pluginVersion: '1.0.0',
|
||||
readmeBadge: '1.0.0', tags: ['v1.0.0'], catalogLabel: '1.0.0',
|
||||
});
|
||||
assert.equal(r.status, 'OK');
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue