feat(gate): install truth, honest badges anywhere, and two stripCode bugs
Install truth is the brief's first control and the gate only checked syntax. Now: the marketplace URL must be the real one (offline, from the register), and the plugin must actually be pinned in the catalog (one call, SKIP if unreachable). A well-formed `claude plugin install x@mkt` fails silently when x was never pinned. This makes the gate block ITSELF until publication finishes - the run against this repo now has exactly one ERROR, and it is true: repo-standard is not in the catalog yet. That is the post-publish acceptance test, enforced mechanically instead of remembered. Badge honesty no longer keys on img.shields.io. A self-hosted SVG asserts the same unverified thing, and the README claimed the general rule while the code checked one host. Two stripCode bugs, both silent false passes: - 4-space indent treated as code unconditionally made links inside nested list items invisible. Fixed by requiring a blank line to OPEN a block. - That fix alone ended the block after line 1, so multi-line indented templates leaked back into scanning. Caught by the gate on this repo's own SKILL.md, which shows a README template containing a CHANGELOG link. A block now opens on a blank line and continues while the indent holds. Also corrected two claims in this README: it said "one network call" when there are two, and it still argued against a CONTRIBUTING using reasoning the solo-maintainer section had already replaced. 77 tests. llm-security regression: still zero link and boilerplate noise. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WYJ3FHLtVgzFXMZ6UF598h
This commit is contained in:
parent
720850a9ad
commit
6b1db0096e
5 changed files with 187 additions and 16 deletions
12
CHANGELOG.md
12
CHANGELOG.md
|
|
@ -38,11 +38,16 @@ First release. Covers the checks that a single repository can answer on its own.
|
|||
- **Badge honesty** — a static badge asserting a test, build or coverage run is
|
||||
a claim dressed as evidence. Version, licence and platform badges assert no
|
||||
run and are fine.
|
||||
- **Install truth, not just install syntax** — the marketplace URL in the install
|
||||
block must be the real one (offline, against the register), and the plugin must
|
||||
actually be pinned in the catalog (one call; `SKIP` if unreachable). A perfectly
|
||||
formed `claude plugin install x@mkt` fails silently when `x` was never pinned,
|
||||
and that is the single most disqualifying thing a repo can publish.
|
||||
- **Boilerplate, licence-claim and relative-link checks.**
|
||||
- **The skill** (`skills/repo-standard/`) — the judgement the script cannot
|
||||
encode: what a description should say, why the summary card must never be used
|
||||
to verify one, what not to retrofit, and where the per-repo boundary is.
|
||||
- 67 tests over the pure classifiers, using measured false positives as
|
||||
- 77 tests over the pure classifiers, using measured false positives as
|
||||
reference fixtures.
|
||||
|
||||
### Notes
|
||||
|
|
@ -69,6 +74,11 @@ First release. Covers the checks that a single repository can answer on its own.
|
|||
link to a naive scanner — the first version produced about thirty findings
|
||||
against a real repository and every one was noise.
|
||||
- Relative links resolve against the file they appear in, not the repo root.
|
||||
- Indented text counts as a code block only when a blank line opens it, and then
|
||||
continues while the indent holds. The first rule alone made links inside nested
|
||||
list items invisible; the second alone let everything after a block's first line
|
||||
leak back in. Both were silent false passes, which is worse than noise — the
|
||||
second was caught by the gate on this plugin's own SKILL.md.
|
||||
- A README H1 that differs from the repo name is a `WARN`, not an `ERROR`. The
|
||||
thread that has to hold is description == catalog == opening line, and the H1
|
||||
is none of those three; a human title is a naming choice the operator owns.
|
||||
|
|
|
|||
|
|
@ -63,7 +63,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
## Commands
|
||||
|
||||
```bash
|
||||
npm test # 67 tests
|
||||
npm test # 77 tests
|
||||
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
|
||||
node scripts/repo-standard-check.mjs --offline # no network call
|
||||
node scripts/repo-standard-check.mjs --json # machine output
|
||||
|
|
|
|||
19
README.md
19
README.md
|
|
@ -41,9 +41,10 @@ a replacement for the two commands above:
|
|||
|
||||
## Requirements
|
||||
|
||||
Node 18 or newer. No dependencies. The one network call — the org listing, used
|
||||
to read a repo's published description — reads anonymously, so it needs no
|
||||
token; `--offline` skips it.
|
||||
Node 18 or newer. No dependencies. Two network calls: the org listing (for a
|
||||
repo's published description) and the catalog manifest (to confirm the install
|
||||
command resolves). Both read anonymously, so no token is needed, and `--offline`
|
||||
skips both — the checks that depended on them then report `SKIP`, not `OK`.
|
||||
|
||||
## What it does
|
||||
|
||||
|
|
@ -58,13 +59,14 @@ The repository's **class** decides what each check means:
|
|||
| Check | What fails it |
|
||||
| --- | --- |
|
||||
| First screen | no H1 on line 1, or the line under it is not the published description. An H1 that merely differs from the repo name is a `WARN` — that is a naming choice, not a defect |
|
||||
| Install block | the form for this class is missing, incomplete, or shown over `ssh://` |
|
||||
| Install block | the form for this class is missing, incomplete, shown over `ssh://`, or points at the wrong marketplace |
|
||||
| Install truth | the plugin is not pinned in the catalog, so the documented command cannot succeed for anyone |
|
||||
| Required headings | `## Install`, `## Non-goals`, `## Changelog` — per class. Present at the wrong depth is its own finding |
|
||||
| Required files | a file this class (or trait) needs is absent |
|
||||
| Repo references | an `open/<name>` in URL position resolves to nothing |
|
||||
| Relative links | a link points at a file that is not tracked |
|
||||
| Licence claim | the README cites a licence the repo has no file for |
|
||||
| Badges | a static badge asserts a test, build or coverage run that nothing verifies |
|
||||
| Badges | a static image badge — any host — asserts a test, build or coverage run that nothing verifies |
|
||||
| Boilerplate | template text nobody filled in |
|
||||
| Version consistency | manifest, README badge, newest CHANGELOG entry and the git tag disagree |
|
||||
| Description | empty, or past the length bound |
|
||||
|
|
@ -111,9 +113,8 @@ judgement. It lives in `register/repos.json`.
|
|||
| standalone | pip/uv | README, LICENSE |
|
||||
| org-profile | none | README |
|
||||
|
||||
A flat standard across every class would demand a CONTRIBUTING from a CSS
|
||||
library that accepts no contributions, and a roadmap from a five-line profile.
|
||||
That is how a gate teaches people to switch it off.
|
||||
A flat standard across every class would demand a roadmap from a five-line
|
||||
profile. That is how a gate teaches people to switch it off.
|
||||
|
||||
### Three outcomes on references, not two
|
||||
|
||||
|
|
@ -148,7 +149,7 @@ without that, a raw scan turns three dead names into about twenty.
|
|||
npm test
|
||||
```
|
||||
|
||||
67 tests over the pure classifiers. The reference fixtures are measured false
|
||||
77 tests over the pure classifiers. The reference fixtures are measured false
|
||||
positives, each with its expected verdict — the six that produced the
|
||||
three-outcome reference rule, plus the noise sources found by running the gate
|
||||
against a real repository: regexes inside code spans that are markdown links to
|
||||
|
|
|
|||
|
|
@ -215,6 +215,23 @@ export function checkInstallBlock({ readme, name, klass }, register) {
|
|||
});
|
||||
}
|
||||
|
||||
// A well-formed command pointing at the wrong marketplace is still a command
|
||||
// that does not work. Checked against the register, so it needs no network.
|
||||
if (hasAdd && mkt.url) {
|
||||
const urls = addLines
|
||||
.map((l) => /marketplace\s+add\s+(\S+)/.exec(l)?.[1])
|
||||
.filter(Boolean)
|
||||
.map((u) => u.replace(/[`'"]+$/, ''));
|
||||
if (urls.length && !urls.some((u) => normalizeRepoRef(u) === normalizeRepoRef(mkt.url))) {
|
||||
findings.push({
|
||||
level: 'ERROR',
|
||||
code: 'INSTALL-URL-MISMATCH',
|
||||
bucket: 'broken',
|
||||
msg: `\`marketplace add\` points at ${urls[0]}, but this marketplace is ${mkt.url}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (form === 'plugin' || form === 'catalog') {
|
||||
if (!hasAdd) {
|
||||
findings.push({
|
||||
|
|
@ -279,6 +296,26 @@ export function checkInstallBlock({ readme, name, klass }, register) {
|
|||
|
||||
// Per class, never flat. A flat standard demands a CONTRIBUTING from a CSS
|
||||
// library that takes no contributions and a ROADMAP from a five-line profile.
|
||||
// Syntax is not truth. The most disqualifying failure a repo can have is an
|
||||
// install command that does not work for a stranger, and a perfectly formed
|
||||
// `claude plugin install x@mkt` fails silently if `x` was never pinned in the
|
||||
// catalog. This is the one check that answers the brief's first question.
|
||||
export function checkInstallTruth({ name, klass, catalogNames }) {
|
||||
if (klass !== 'plugin') return [{ level: 'OK', code: 'INSTALL-TRUTH', msg: 'not a marketplace plugin — nothing to resolve' }];
|
||||
if (!catalogNames) {
|
||||
return [{ level: 'SKIP', code: 'INSTALL-TRUTH', msg: 'catalog not reachable — cannot verify the install command actually resolves' }];
|
||||
}
|
||||
if (!catalogNames.includes(name)) {
|
||||
return [{
|
||||
level: 'ERROR',
|
||||
code: 'INSTALL-NOT-IN-CATALOG',
|
||||
bucket: 'broken',
|
||||
msg: `\`${name}\` is not pinned in the marketplace catalog — the documented install command cannot succeed for anyone`,
|
||||
}];
|
||||
}
|
||||
return [{ level: 'OK', code: 'INSTALL-TRUTH', msg: 'the install command resolves against the catalog' }];
|
||||
}
|
||||
|
||||
// Requirements come from two axes. The CLASS is structural — it can be read off
|
||||
// the catalog and the remotes. A TRAIT is about what the code does, which no
|
||||
// remote can tell you: `security` attaches the obligations a tool acquires by
|
||||
|
|
@ -395,7 +432,9 @@ const CLAIM_BADGE = /(tests?|build|ci|coverage|passing|status)/i;
|
|||
export function checkBadges({ readme }) {
|
||||
const findings = [];
|
||||
for (const line of String(readme ?? '').split('\n')) {
|
||||
for (const m of line.matchAll(/(\[)?!\[([^\]]*)\]\((https:\/\/img\.shields\.io\/badge\/[^)]+)\)(\])?/g)) {
|
||||
// Any image, any host. Restricting this to img.shields.io would have missed
|
||||
// a self-hosted SVG asserting exactly the same unverified thing.
|
||||
for (const m of line.matchAll(/(\[)?!\[([^\]]*)\]\(([^)\s]+)\)(\])?/g)) {
|
||||
const linked = m[1] === '[' && m[4] === ']';
|
||||
const label = `${m[2]} ${m[3]}`;
|
||||
if (!linked && CLAIM_BADGE.test(label)) {
|
||||
|
|
@ -471,6 +510,8 @@ export function checkLicenseClaim({ readme, present }) {
|
|||
// one of them was noise.
|
||||
export function stripCode(text) {
|
||||
let fenced = false;
|
||||
let prevBlank = true;
|
||||
let inIndented = false;
|
||||
return String(text ?? '')
|
||||
.split('\n')
|
||||
.map((line) => {
|
||||
|
|
@ -479,7 +520,22 @@ export function stripCode(text) {
|
|||
return '';
|
||||
}
|
||||
if (fenced) return '';
|
||||
if (/^(\s{4,}|\t)\S/.test(line)) return ''; // indented code block
|
||||
|
||||
const blank = line.trim() === '';
|
||||
const indented = /^(\s{4,}|\t)\S/.test(line);
|
||||
// An indented line OPENS a code block only after a blank line — otherwise
|
||||
// a nested list item would count, which made links inside nested bullets
|
||||
// invisible. But once open, the block CONTINUES while lines stay indented;
|
||||
// requiring a blank line before every line let everything after line 1
|
||||
// leak back into scanning.
|
||||
if (indented && (prevBlank || inIndented)) {
|
||||
inIndented = true;
|
||||
prevBlank = false;
|
||||
return '';
|
||||
}
|
||||
if (!blank && !indented) inIndented = false;
|
||||
prevBlank = blank;
|
||||
if (inIndented && blank) return '';
|
||||
return line.replace(/`[^`]*`/g, '');
|
||||
})
|
||||
.join('\n');
|
||||
|
|
@ -567,7 +623,7 @@ export function bucketsOf(findings) {
|
|||
}
|
||||
|
||||
export function classifyRepo(
|
||||
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags },
|
||||
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, catalogNames },
|
||||
register,
|
||||
) {
|
||||
const klass = register.repos?.[name];
|
||||
|
|
@ -591,6 +647,7 @@ export function classifyRepo(
|
|||
const findings = [
|
||||
...checkFirstScreen({ readme, name, description }),
|
||||
...checkInstallBlock({ readme, name, klass }, register),
|
||||
...checkInstallTruth({ name, klass, catalogNames }),
|
||||
...checkHeadings({ readme, klass, traits }, register),
|
||||
...checkRequiredFiles({ present, klass, traits }, register),
|
||||
...checkLinks({ files }, register),
|
||||
|
|
@ -614,6 +671,23 @@ export function loadRegister(path = REGISTER_PATH) {
|
|||
// ONE call. The org listing already carries description and topics; fetching
|
||||
// per repo trips the rate limiter (HTTP 429). Reads anonymously — verified —
|
||||
// so this works for any reader, not only for someone holding a token.
|
||||
// The catalog's plugin list, read straight from the forge. One more call, and
|
||||
// it is what turns "the install line is well-formed" into "the install line
|
||||
// works". Null on any failure, which reads as SKIP rather than a pass.
|
||||
async function fetchCatalogNames(register) {
|
||||
const mkt = register.marketplace ?? {};
|
||||
if (!mkt.name) return null;
|
||||
const url = `${register.forge}/api/v1/repos/${register.org}/${mkt.name}/raw/.claude-plugin/marketplace.json`;
|
||||
try {
|
||||
const res = await fetch(url, { headers: { accept: 'application/json' } });
|
||||
if (!res.ok) return null;
|
||||
const json = JSON.parse(await res.text());
|
||||
return (json.plugins ?? []).map((p) => p.name).filter(Boolean);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function fetchOrgListing(register) {
|
||||
const url = `${register.forge}/api/v1/orgs/${register.org}/repos?limit=50`;
|
||||
const res = await fetch(url, { headers: { accept: 'application/json' } });
|
||||
|
|
@ -683,7 +757,7 @@ function gitTags(dir) {
|
|||
}
|
||||
}
|
||||
|
||||
export function inspectRepo(dir, name, register, description) {
|
||||
export function inspectRepo(dir, name, register, description, catalogNames = null) {
|
||||
const tracked = gitFiles(dir);
|
||||
const present = (tracked ?? []).filter((f) => existsSync(join(dir, f)));
|
||||
|
||||
|
|
@ -710,6 +784,7 @@ export function inspectRepo(dir, name, register, description) {
|
|||
readmeBadge: extractBadgeVersion(readme),
|
||||
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
|
||||
tags: gitTags(dir),
|
||||
catalogNames,
|
||||
}, register);
|
||||
}
|
||||
|
||||
|
|
@ -775,7 +850,9 @@ async function main(argv) {
|
|||
const name = arg('--name', repoNameFrom(dir));
|
||||
|
||||
let description = null;
|
||||
let catalogNames = null;
|
||||
if (!argv.includes('--offline')) {
|
||||
catalogNames = await fetchCatalogNames(register);
|
||||
try {
|
||||
const listing = await fetchOrgListing(register);
|
||||
const row = listing.find((r) => r.name === name);
|
||||
|
|
@ -786,7 +863,7 @@ async function main(argv) {
|
|||
}
|
||||
}
|
||||
|
||||
const result = inspectRepo(dir, name, register, description);
|
||||
const result = inspectRepo(dir, name, register, description, catalogNames);
|
||||
|
||||
if (argv.includes('--json')) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ import {
|
|||
checkLicenseClaim,
|
||||
checkInternalLinks,
|
||||
resolveRelative,
|
||||
checkInstallTruth,
|
||||
classifyRepo,
|
||||
levelOf,
|
||||
} from './repo-standard-check.mjs';
|
||||
|
|
@ -408,6 +409,7 @@ test('a fully compliant plugin repo classifies OK', () => {
|
|||
readmeBadge: '0.7.0',
|
||||
changelogTop: '0.7.0',
|
||||
tags: ['v0.7.0'],
|
||||
catalogNames: ['repo-mailbox'],
|
||||
},
|
||||
REGISTER,
|
||||
);
|
||||
|
|
@ -694,3 +696,84 @@ test('a nested link to a genuinely absent sibling is still an ERROR', () => {
|
|||
});
|
||||
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING'), true);
|
||||
});
|
||||
|
||||
// ------------------------------- stripCode must not swallow nested list items
|
||||
|
||||
test('a link inside a nested list item is still scanned', () => {
|
||||
// An indented line is only a code block when a blank line precedes it.
|
||||
// Treating every 4-space indent as code made nested bullets invisible to both
|
||||
// the link and boilerplate checks — a silent false pass, which is worse than
|
||||
// the noise it was meant to remove.
|
||||
const text = ['- top level', ' - nested with [a link](gone.md)'].join('\n');
|
||||
const f = checkInternalLinks({ files: { 'README.md': text }, present: ['README.md'] });
|
||||
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING'), true);
|
||||
});
|
||||
|
||||
test('a genuine indented code block is still skipped', () => {
|
||||
const text = ['Example:', '', ' curl [x](not-real.md)'].join('\n');
|
||||
const f = checkInternalLinks({ files: { 'README.md': text }, present: ['README.md'] });
|
||||
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
|
||||
});
|
||||
|
||||
// ------------------------------------------ badge honesty beyond shields.io
|
||||
|
||||
test('a self-hosted badge asserting a run is caught too', () => {
|
||||
const f = checkBadges({ readme: '' });
|
||||
assert.equal(f.some((x) => x.code === 'BADGE-STATIC-CLAIM'), true);
|
||||
});
|
||||
|
||||
// ------------------------------------------------------------ install truth
|
||||
|
||||
test('the marketplace URL in the install block must be the real one', () => {
|
||||
const readme = [
|
||||
'## Install',
|
||||
'claude plugin marketplace add https://git.fromaitochitta.com/open/WRONG.git',
|
||||
`claude plugin install repo-standard@${MKT.name}`,
|
||||
].join('\n');
|
||||
const f = checkInstallBlock({ readme, name: 'repo-standard', klass: 'plugin' }, REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'INSTALL-URL-MISMATCH' && x.bucket === 'broken'), true);
|
||||
});
|
||||
|
||||
test('a plugin absent from the catalog has an install line that cannot work', () => {
|
||||
// Syntax is not truth. The brief's first control asks whether the command
|
||||
// works for a stranger, not whether it is well-formed.
|
||||
const f = checkInstallTruth({ name: 'ghost', klass: 'plugin', catalogNames: ['repo-mailbox', 'llm-security'] });
|
||||
assert.equal(f.some((x) => x.code === 'INSTALL-NOT-IN-CATALOG' && x.bucket === 'broken'), true);
|
||||
});
|
||||
|
||||
test('a plugin present in the catalog passes install truth', () => {
|
||||
const f = checkInstallTruth({ name: 'repo-mailbox', klass: 'plugin', catalogNames: ['repo-mailbox'] });
|
||||
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
|
||||
});
|
||||
|
||||
test('an unreachable catalog is SKIP, never a pass', () => {
|
||||
const f = checkInstallTruth({ name: 'repo-mailbox', klass: 'plugin', catalogNames: null });
|
||||
assert.equal(f[0].level, 'SKIP');
|
||||
});
|
||||
|
||||
test('non-plugin classes are not measured against the catalog', () => {
|
||||
const f = checkInstallTruth({ name: 'portfolio-optimiser', klass: 'standalone', catalogNames: null });
|
||||
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
|
||||
});
|
||||
|
||||
test('an indented code block stays code past its first line', () => {
|
||||
// The blank-line rule fixed nested lists but broke multi-line indented blocks:
|
||||
// only line 1 followed a blank line, so lines 2+ leaked back into scanning.
|
||||
// Caught by the gate on this plugin's own SKILL.md, which shows a README
|
||||
// template containing a link that does not exist relative to that file.
|
||||
const text = [
|
||||
'Template:',
|
||||
'',
|
||||
' # <name>',
|
||||
' ## Changelog',
|
||||
' See [CHANGELOG.md](CHANGELOG.md).',
|
||||
].join('\n');
|
||||
const f = checkInternalLinks({ files: { 'skills/x/SKILL.md': text }, present: ['skills/x/SKILL.md'] });
|
||||
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
|
||||
});
|
||||
|
||||
test('an indented block ends at the first unindented line', () => {
|
||||
const text = ['Template:', '', ' code here', '', 'Back to prose with [a link](gone.md).'].join('\n');
|
||||
const f = checkInternalLinks({ files: { 'README.md': text }, present: ['README.md'] });
|
||||
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING'), true);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue