open/repo-standard
1
0
Fork 0
Code Issues Releases 2 Activity

feat(gate): buckets, traits, and the checks the brief calls load-bearing

Browse source 
Measured this build against a documentation brief for public repos. The
five original checks covered roughly one of its ten sections, so this
adds what a single repo can answer on its own.

New: required README headings per class (Non-goals is the cheapest
trust-builder there is), in-repo version consistency across manifest /
badge / CHANGELOG / tag, badge honesty, boilerplate, licence-claim,
and relative links. Findings now carry a BUCKET beside the level -
broken / missing / weakening - and output is grouped by it, because
that is the order the work gets done in.

Traits are a second axis beside class: class is structural and readable
off the catalog, a trait says what the code does. `security` attaches
SECURITY.md and a Known limitations section. The two names carrying it
are proposed, not measured - that list is the operator's.

Solo-maintained settles a category: CONTRIBUTING, CODE_OF_CONDUCT and
MAINTAINERS are required by no class. Consumer-facing documents are
untouched by that; SECURITY.md exists for the stranger who finds a hole.

Three bugs found by running against llm-security, not by reading:
- ~30 link findings, all noise. Regexes inside code spans are
  `[...](...)` to a naive scanner. Strip code first.
- `file:` and other schemes were treated as repo-relative paths.
- Relative links were resolved against the repo root instead of the
  file they sit in, calling two files missing that sat next to the
  README linking them.
Same fix applied to the boilerplate check: a document ABOUT placeholder
detection was tripping the placeholder detector.

Also removed this repo's own static tests badge. There is no CI - the
forge has zero Actions runners registered - so it could never become
real, and it is the exact anti-pattern the gate now flags.

67 tests. Against llm-security every remaining finding is real and
matches the census's independent hand-measurement.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WYJ3FHLtVgzFXMZ6UF598h
This commit is contained in:
Kjell Tore Guttormsen 2026-07-27 16:06:33 +02:00
commit 720850a9ad
7 changed files with 915 additions and 39 deletions
Download patch file Download diff file Expand all files Collapse all files

33
CHANGELOG.md
View file

@ -22,10 +22,27 @@ First release. Covers the checks that a single repository can answer on its own.
repository" (`WARN`). Sharing an outcome would let real dead links hide among
correct text. Only names in URL position are treated as references; the `.git`
suffix is normalised first.
- **Two-axis findings** — every `ERROR`/`WARN` carries a **bucket** alongside its
level: `broken` (a stranger is blocked or misled), `missing` (an expected
artefact is absent), `weakening` (present, but it reads as amateur). Output is
grouped by bucket, because that is the order findings get acted on.
- **Traits, a second axis beside class** — class is structural and readable off
the catalog; a trait is about what the code does. `security` attaches a
`SECURITY.md` requirement and a `## Known limitations` section.
- **Required README headings per class** — `## Install`, `## Non-goals`,
`## Changelog`. A heading present at the wrong depth is its own finding, not
"missing".
- **In-repo version consistency** — manifest, README version badge, newest
CHANGELOG entry and the git tag must agree. An untagged repo `SKIP`s the tag
comparison rather than failing it.
- **Badge honesty** — a static badge asserting a test, build or coverage run is
a claim dressed as evidence. Version, licence and platform badges assert no
run and are fine.
- **Boilerplate, licence-claim and relative-link checks.**
- **The skill** (`skills/repo-standard/`) — the judgement the script cannot
encode: what a description should say, why the summary card must never be used
to verify one, what not to retrofit, and where the per-repo boundary is.
- 34 tests over the pure classifiers, using the measured false positives as
- 67 tests over the pure classifiers, using measured false positives as
reference fixtures.
### Notes
@ -38,6 +55,20 @@ First release. Covers the checks that a single repository can answer on its own.
form as the defect.
- The org listing is read in **one** call, anonymously. Per-repo fetching trips
the forge's rate limiter.
- `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` and `MAINTAINERS.md` are required by
no class: the project is solo-maintained and publishes that stance. This is
not a rule against having them — files already present are a cleanup decision,
not a gate finding. `SECURITY.md` is unaffected; it exists for the outsider who
finds a hole.
- There is no CI badge because there is no CI: the forge has no Actions runner
registered. The substitute is one command from a clean clone, stated honestly.
A badge asserting the test count would be the very anti-pattern the gate flags,
and an earlier draft of this README carried exactly that.
- The link and boilerplate checks ignore fenced blocks and inline code spans.
Documentation about regexes and placeholders is full of text that is a markdown
link to a naive scanner — the first version produced about thirty findings
against a real repository and every one was noise.
- Relative links resolve against the file they appear in, not the repo root.
- A README H1 that differs from the repo name is a `WARN`, not an `ERROR`. The
thread that has to hold is description == catalog == opening line, and the H1
is none of those three; a human title is a naming choice the operator owns.

21
CLAUDE.md
View file

@ -29,6 +29,25 @@ would recreate, in data, exactly the drift this plugin exists to remove.
- **It records, it does not fix.** Findings first, remediation afterwards.
Patching while measuring is how the inconsistency it detects was produced.
- **`SKIP` is never a pass.** A check that could not run says so and names why.
- **When a check fires many times in one repo, suspect the CHECK.** The first
link pass produced ~30 findings against `llm-security` and all were noise —
regexes in code spans, `file:` URLs, relative paths resolved against the wrong
directory. Strip code before scanning text; resolve links against the file
they sit in. A gate that is wrong this often gets switched off.
- **Two axes on every finding.** Level (`ERROR`/`WARN`/`SKIP`/`OK`) and bucket
(`broken`/`missing`/`weakening`). They are independent — a `weakening` finding
can be an `ERROR`.
- **Class is structural, traits are judgement.** Class is read off the catalog
and the remotes. A trait (`security`) says what the code does, which no remote
knows, and the operator owns that list.
- **Who the reader is decides what is required.** Contributor-facing documents
are required by no class — solo-maintained, and the published stance says so.
Consumer-facing ones (`SECURITY.md`, `LICENSE`, non-goals, limitations) are
untouched by that. This is not a rule against having the others.
- **No CI badge, because there is no CI.** The forge has zero Actions runners
registered (measured). The substitute is one command from a clean clone, said
plainly. A static badge asserting a run is the anti-pattern this gate flags —
and an early draft of this README carried one.
- **Three outcomes on references.** "No match" and "match on a known non-repo"
must stay distinct findings. Collapsing them hides real loss inside correct
text — the exact defect class this gate exists to catch.
@ -44,7 +63,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
## Commands
```bash
npm test # 32 tests
npm test # 67 tests
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
node scripts/repo-standard-check.mjs --offline # no network call
node scripts/repo-standard-check.mjs --json # machine output

59
README.md
View file

@ -13,9 +13,14 @@ checks that surface in one repository and reports what it finds.
![Version](https://img.shields.io/badge/version-0.1.0-blue)
![Platform](https://img.shields.io/badge/platform-Claude_Code_Plugin-purple)
![Skills](https://img.shields.io/badge/skills-1-orange)
![Tests](https://img.shields.io/badge/tests-34-green)
![License](https://img.shields.io/badge/license-MIT-lightgrey)
**Status:** maintained, solo. The public API is the gate's finding codes and the
register schema; both can still change before 1.0. There is no CI — the forge
has no Actions runner — so the test claim is one you run yourself, in one
command, from a clean clone: `npm test`. A badge asserting it would be a claim
dressed as evidence.
## Install
Use the `https://` form. The forge UI's clone button hands out an `ssh://` URL,
@ -48,19 +53,50 @@ Run it inside a repository:
node scripts/repo-standard-check.mjs --dir "$PWD"
```
It checks five things, and the repository's **class** decides what each one means:
The repository's **class** decides what each check means:
| Check | What fails it |
| --- | --- |
| First screen | no H1 on line 1, or the line under it is not the published description. An H1 that merely differs from the repo name is a `WARN` — that is a naming choice, not a defect |
| Install block | the form for this class is missing, incomplete, or shown over `ssh://` |
| Required files | a file this class needs is absent |
| References | an `open/<name>` in URL position resolves to nothing |
| Required headings | `## Install`, `## Non-goals`, `## Changelog` — per class. Present at the wrong depth is its own finding |
| Required files | a file this class (or trait) needs is absent |
| Repo references | an `open/<name>` in URL position resolves to nothing |
| Relative links | a link points at a file that is not tracked |
| Licence claim | the README cites a licence the repo has no file for |
| Badges | a static badge asserts a test, build or coverage run that nothing verifies |
| Boilerplate | template text nobody filled in |
| Version consistency | manifest, README badge, newest CHANGELOG entry and the git tag disagree |
| Description | empty, or past the length bound |
Findings are `ERROR`, `WARN`, `SKIP` or `OK`; the process exits 1 on any
`ERROR`. A `SKIP` means the check could not run — an unreachable forge, a repo
that is not in the register. It is not a pass, and the output says which.
Findings carry two independent things: a **level** (`ERROR`, `WARN`, `SKIP`,
`OK`) and a **bucket**, which is what you actually triage on:
- **broken now** — a stranger is blocked or misled
- **missing** — an expected artefact is absent
- **weakening** — present and working, but it reads as amateur
The process exits 1 on any `ERROR`. A `SKIP` means the check could not run — an
unreachable forge, an untagged repo, a link that leaves the repository. It is
not a pass, and the output prints those separately under a heading that says so.
### Traits — a second axis
Class is structural. A **trait** is about what the code does, which no remote can
tell you. `security` attaches the obligations a tool takes on by handling
untrusted input: a `SECURITY.md` with a real disclosure channel, and a
`## Known limitations` section. Traits live in the register and are the
maintainer's judgement, not a reading.
### What is deliberately not required
`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` and `MAINTAINERS.md` are required by no
class. This project is solo-maintained and says so publicly; contributor-facing
documentation for something that accepts no contributors is theatre, and a code
of conduct with an unattended placeholder address is worse than none — it is a
visible unfinished template. Consumer-facing documents are untouched by that.
Being solo is not a reason to skip a `SECURITY.md`; someone finding a hole still
needs somewhere to send it.
### The class decides what is required
@ -112,9 +148,12 @@ without that, a raw scan turns three dead names into about twenty.
npm test
```
34 tests over the pure classifiers. The reference fixtures are the measured
false positives that produced the three-outcome rule, each with its expected
verdict.
67 tests over the pure classifiers. The reference fixtures are measured false
positives, each with its expected verdict — the six that produced the
three-outcome reference rule, plus the noise sources found by running the gate
against a real repository: regexes inside code spans that are markdown links to
a naive scanner, `file:` URLs, and relative links resolved against the wrong
directory.
## Changelog

52
register/repos.json
View file

@ -52,36 +52,76 @@
},
"$comment_classes": [
"Per class: which files are required, and which install form the README must",
"carry. A flat standard across all classes would demand a CONTRIBUTING from a",
"CSS library that takes no contributions, and a ROADMAP from a 5-line profile.",
"ROADMAP is deliberately absent everywhere: it is 0/18 today and is step 8,",
"drafted from STATE by a human. A gate that fails every repo teaches people to",
"switch the gate off."
"Per class: required files, required README headings, and the install form.",
"A flat standard across all classes would demand a ROADMAP from a 5-line",
"profile. ROADMAP is deliberately absent everywhere: it is 0/18 today and is",
"drafted from STATE by a human. A gate that fails every repo teaches people",
"to switch the gate off.",
"",
"CONTRIBUTING.md, CODE_OF_CONDUCT.md and MAINTAINERS.md are deliberately NOT",
"required anywhere. The maintainer works alone and the catalog's published",
"stance already says so — 'solo-maintained, fork-and-own; issues welcome as",
"signals, pull requests not accepted'. Contributor-facing documentation for a",
"project that accepts no contributors is theatre, and a CODE_OF_CONDUCT with",
"an unattended placeholder address is worse than none: it is a visible",
"unfinished template. This is NOT a rule against having them — files already",
"present are a separate cleanup decision, not a gate finding.",
"",
"Consumer-facing documentation is unaffected by working alone, and that is",
"the whole distinction: SECURITY.md, LICENSE, CHANGELOG, non-goals and honest",
"limitations exist for the reader, not for a contributor."
],
"classes": {
"plugin": {
"required_files": ["README.md", "LICENSE", "CHANGELOG.md", ".claude-plugin/plugin.json"],
"required_headings": ["## Install", "## Non-goals", "## Changelog"],
"install": "plugin"
},
"catalog": {
"required_files": ["README.md", "LICENSE", "GOVERNANCE.md", "CONVENTIONS.md", ".claude-plugin/marketplace.json"],
"required_headings": ["## Install", "## Non-goals"],
"install": "catalog"
},
"shared-asset": {
"required_files": ["README.md", "LICENSE"],
"required_headings": ["## Non-goals"],
"install": "vendor"
},
"org-profile": {
"required_files": ["README.md"],
"required_headings": [],
"install": "none"
},
"standalone": {
"required_files": ["README.md", "LICENSE"],
"required_headings": ["## Install", "## Non-goals"],
"install": "package"
}
},
"$comment_traits": [
"A SECOND axis, orthogonal to class. Class is structural (read off the",
"catalog and the remotes); a trait is about what the code DOES, which no",
"remote can tell you. `security` attaches the obligations a tool acquires by",
"handling untrusted input: a real disclosure channel, and limitations stated",
"with their mechanism.",
"",
"PROPOSED BY THE GATE'S AUTHOR, NOT MEASURED — the operator owns this list.",
"Marking a repo `security` decides that it owes a SECURITY.md, so adding or",
"removing a name here is a judgement, not a reading. These two were picked",
"because both process untrusted input as their stated purpose."
],
"traits": {
"llm-security": ["security"],
"llm-ingestion-pipeline-security": ["security"]
},
"trait_requirements": {
"security": {
"required_files": ["SECURITY.md"],
"required_headings": ["## Known limitations"]
}
},
"description_max_codepoints": 180,
"$comment_length": [
"180 codepoints, not bytes and not UTF-16 units. The same string measures 248",

404
scripts/repo-standard-check.mjs
View file

@ -32,6 +32,18 @@ const REGISTER_PATH = join(HERE, '..', 'register', 'repos.json');
const LEVELS = ['OK', 'SKIP', 'WARN', 'ERROR'];
// Findings carry a level AND a bucket, and the two are independent axes.
// The level says how sure and how loud; the bucket says what KIND of problem it
// is, which is what a reader triages on:
//
// broken works wrongly right now — a stranger is blocked or misled
// missing an expected artefact is simply absent
// weakening present and functional, but it reads as amateur
//
// A weakening finding can still be an ERROR: a README opening line that
// contradicts the published description blocks nobody, and is still wrong.
export const BUCKETS = ['broken', 'missing', 'weakening'];
// ------------------------------------------------------------ pure helpers
// Codepoints. Not bytes (an em-dash costs 3) and not UTF-16 units (`👉` costs 2).
@ -85,12 +97,14 @@ export function checkLinks({ files }, register) {
findings.push({
level: 'WARN',
code: 'LINK-NON-REPO',
bucket: 'weakening',
msg: `${path}:${ref.line} — \`open/${ref.name}\` resolves to a known non-repo: ${register.non_repos[ref.name]}`,
});
} else {
findings.push({
level: 'ERROR',
code: 'LINK-DEAD',
bucket: 'broken',
msg: `${path}:${ref.line} — \`open/${ref.name}\` matches no repo in the register (dead reference)`,
});
}
@ -105,9 +119,9 @@ export function checkDescription(description, register) {
}
const max = register.description_max_codepoints ?? 180;
const n = countCodepoints(description);
if (n === 0) return [{ level: 'ERROR', code: 'DESC-EMPTY', msg: 'forge description is empty' }];
if (n === 0) return [{ level: 'ERROR', code: 'DESC-EMPTY', bucket: 'missing', msg: 'forge description is empty' }];
if (n > max) {
return [{ level: 'ERROR', code: 'DESC-TOO-LONG', msg: `forge description is ${n} codepoints, bound is ${max}` }];
return [{ level: 'ERROR', code: 'DESC-TOO-LONG', bucket: 'weakening', msg: `forge description is ${n} codepoints, bound is ${max}` }];
}
return [{ level: 'OK', code: 'DESC', msg: `description ${n}/${max} codepoints` }];
}
@ -132,6 +146,7 @@ export function checkFirstScreen({ readme, name, description }) {
findings.push({
level: 'ERROR',
code: 'README-H1',
bucket: 'missing',
msg: `README must open with an H1 (expected \`# ${name}\`, found: ${heading === null ? '<empty file>' : `\`${heading}\``})`,
});
return findings;
@ -140,6 +155,7 @@ export function checkFirstScreen({ readme, name, description }) {
findings.push({
level: 'WARN',
code: 'README-H1',
bucket: 'weakening',
msg: `H1 is \`${heading}\`, not \`# ${name}\` — deliberate title, or drift? Operator's call.`,
});
} else {
@ -157,6 +173,7 @@ export function checkFirstScreen({ readme, name, description }) {
findings.push({
level: 'ERROR',
code: 'README-DESC',
bucket: 'weakening',
msg: `README opening line does not match the forge description\n README: ${opening}\n forge: ${description}`,
});
} else {
@ -193,6 +210,7 @@ export function checkInstallBlock({ readme, name, klass }, register) {
findings.push({
level: 'ERROR',
code: 'INSTALL-SSH',
bucket: 'broken',
msg: '`marketplace add` is shown with an ssh:// URL — it rejects those ("Invalid git URL"). Use the https form.',
});
}
@ -202,6 +220,7 @@ export function checkInstallBlock({ readme, name, klass }, register) {
findings.push({
level: 'ERROR',
code: 'INSTALL-NO-MARKETPLACE',
bucket: 'broken',
msg: `no \`plugin marketplace add\` line — the reader is never told to add \`${mkt.name}\` (${mkt.url})`,
});
} else {
@ -220,6 +239,7 @@ export function checkInstallBlock({ readme, name, klass }, register) {
findings.push({
level: 'ERROR',
code: 'INSTALL-NO-CLI',
bucket: 'broken',
msg: `no CLI install command for this repo — expected \`claude plugin install ${name}@${mkt.name}\` (or the \`/plugin install\` form). An \`enabledPlugins\` block is a welcome addition, but it is not a CLI command.`,
});
} else {
@ -231,6 +251,7 @@ export function checkInstallBlock({ readme, name, klass }, register) {
findings.push({
level: 'ERROR',
code: 'INSTALL-WRONG-FORM',
bucket: 'broken',
msg: 'shared asset shows a plugin install line — it is vendored into consumers, not installed. Document how to vendor it.',
});
}
@ -240,12 +261,14 @@ export function checkInstallBlock({ readme, name, klass }, register) {
findings.push({
level: 'ERROR',
code: 'INSTALL-WRONG-FORM',
bucket: 'broken',
msg: 'standalone project shows a plugin install line — use the pip/uv form.',
});
} else if (!/\b(pip\s+install|uv\s+(?:pip\s+)?(?:add|sync|install|run)|uvx)\b/.test(text)) {
findings.push({
level: 'WARN',
code: 'INSTALL-NO-PACKAGE-FORM',
bucket: 'missing',
msg: 'no pip/uv install form found — expected for a standalone project',
});
}
@ -256,13 +279,37 @@ export function checkInstallBlock({ readme, name, klass }, register) {
// Per class, never flat. A flat standard demands a CONTRIBUTING from a CSS
// library that takes no contributions and a ROADMAP from a five-line profile.
export function checkRequiredFiles({ present, klass }, register) {
const required = register.classes?.[klass]?.required_files ?? [];
// Requirements come from two axes. The CLASS is structural — it can be read off
// the catalog and the remotes. A TRAIT is about what the code does, which no
// remote can tell you: `security` attaches the obligations a tool acquires by
// handling untrusted input.
//
// Note what is NOT here: CONTRIBUTING, CODE_OF_CONDUCT, MAINTAINERS. The
// maintainer works alone and the published stance says so. Contributor-facing
// documentation for a project that accepts no contributors is theatre, and a
// code of conduct with an unattended placeholder address is worse than none.
// Consumer-facing documents are untouched by that — SECURITY.md exists for the
// outsider who finds a hole, and being solo does not remove them.
function requirementsFor(klass, traits, register) {
const cls = register.classes?.[klass] ?? {};
const files = [...(cls.required_files ?? [])];
const headings = [...(cls.required_headings ?? [])];
for (const t of traits ?? []) {
const tr = register.trait_requirements?.[t];
if (!tr) continue;
for (const f of tr.required_files ?? []) if (!files.includes(f)) files.push(f);
for (const h of tr.required_headings ?? []) if (!headings.includes(h)) headings.push(h);
}
return { files, headings };
}
export function checkRequiredFiles({ present, klass, traits }, register) {
const { files: required } = requirementsFor(klass, traits, register);
const have = new Set(present ?? []);
const findings = [];
for (const f of required) {
if (!have.has(f)) {
findings.push({ level: 'ERROR', code: 'FILE-MISSING', msg: `missing required file for class \`${klass}\`: ${f}` });
findings.push({ level: 'ERROR', code: 'FILE-MISSING', bucket: 'missing', msg: `missing required file for class \`${klass}\`: ${f}` });
}
}
if (findings.length === 0 && required.length > 0) {
@ -271,6 +318,238 @@ export function checkRequiredFiles({ present, klass }, register) {
return findings;
}
// Fixed headings, because experienced readers skip rather than read. `## Install`
// on a predictable heading is what agents pattern-match on, and `## Non-goals`
// is the cheapest trust-builder there is: it proves someone thought about the
// boundary, and it stops misuse before it starts.
export function checkHeadings({ readme, klass, traits }, register) {
const { headings: required } = requirementsFor(klass, traits, register);
const text = String(readme ?? '');
const present = new Set(
text.split('\n').map((l) => l.trim()).filter((l) => l.startsWith('#')),
);
const findings = [];
for (const h of required) {
if ([...present].some((p) => p.toLowerCase() === h.toLowerCase())) continue;
// Same title, wrong depth: say that, rather than "missing". The contract
// wants a predictable top-level heading because that is what an agent
// pattern-matches on — but the section does exist, and the fix is a
// different edit than writing one from scratch.
const title = h.replace(/^#+\s*/, '');
const atOtherLevel = [...present].find(
(p) => p.replace(/^#+\s*/, '').toLowerCase() === title.toLowerCase(),
);
if (atOtherLevel) {
findings.push({
level: 'ERROR',
code: 'HEADING-LEVEL',
bucket: 'weakening',
msg: `README has \`${atOtherLevel}\` but the contract wants \`${h}\` — a predictable top-level heading is what readers and agents scan for`,
});
} else {
findings.push({ level: 'ERROR', code: 'HEADING-MISSING', bucket: 'missing', msg: `README has no \`${h}\` section` });
}
}
if (findings.length === 0 && required.length > 0) {
findings.push({ level: 'OK', code: 'HEADINGS', msg: `all ${required.length} required headings present` });
}
return findings;
}
// One version, four places it can be written down. This is the check that
// removes a whole defect class — "README says v0.3.1, the tag does not exist" —
// and the one that would have caught this repo's own 32→34 test-count drift.
export function checkVersionConsistency({ pluginVersion, readmeBadge, changelogTop, tags }) {
const findings = [];
const v = pluginVersion ? String(pluginVersion).replace(/^v/, '') : null;
if (!v) {
return [{ level: 'SKIP', code: 'VERSION-UNAVAILABLE', msg: 'no package version found — nothing to compare against' }];
}
if (readmeBadge !== null && readmeBadge !== undefined && readmeBadge !== v) {
findings.push({ level: 'ERROR', code: 'VERSION-BADGE', bucket: 'weakening', msg: `README version badge is ${readmeBadge}, manifest says ${v}` });
}
if (changelogTop !== null && changelogTop !== undefined && changelogTop !== v) {
findings.push({ level: 'ERROR', code: 'VERSION-CHANGELOG', bucket: 'weakening', msg: `newest CHANGELOG entry is ${changelogTop}, manifest says ${v}` });
}
// Nothing released yet is a state, not a defect — and it must say so rather
// than pass quietly, because "SKIP is never a pass" is the whole discipline.
if (!tags || tags.length === 0) {
findings.push({ level: 'SKIP', code: 'VERSION-TAG', msg: `repo has no tags — cannot verify that v${v} was ever released` });
} else if (!tags.includes(`v${v}`)) {
findings.push({ level: 'ERROR', code: 'VERSION-TAG', bucket: 'broken', msg: `no tag \`v${v}\` — the documented version was never released (tags: ${tags.slice(-3).join(', ')})` });
}
if (findings.every((f) => f.level === 'OK' || f.level === 'SKIP')) {
findings.push({ level: 'OK', code: 'VERSION', msg: `version ${v} agrees across manifest, README and CHANGELOG` });
}
return findings;
}
// A static image asserting "tests: 642 passing" is a claim dressed as evidence.
// Version, licence and platform badges assert no run, so they are fine static.
const CLAIM_BADGE = /(tests?|build|ci|coverage|passing|status)/i;
export function checkBadges({ readme }) {
const findings = [];
for (const line of String(readme ?? '').split('\n')) {
for (const m of line.matchAll(/(\[)?!\[([^\]]*)\]\((https:\/\/img\.shields\.io\/badge\/[^)]+)\)(\])?/g)) {
const linked = m[1] === '[' && m[4] === ']';
const label = `${m[2]} ${m[3]}`;
if (!linked && CLAIM_BADGE.test(label)) {
findings.push({
level: 'WARN',
code: 'BADGE-STATIC-CLAIM',
bucket: 'weakening',
msg: `static badge asserts a run that nothing verifies: \`${m[2]}\`. A badge like this is a claim dressed as evidence — link it to a real run, or drop it.`,
});
}
}
}
if (findings.length === 0) findings.push({ level: 'OK', code: 'BADGES', msg: 'no static badge asserts an unverified run' });
return findings;
}
// Template text that was never filled in. A visible unfinished template costs
// more trust than the missing document would have.
const BOILERPLATE = [
/your-project-name/i,
/\byour-org\b/i,
/\[INSERT[^\]]*\]/i,
/<your[- ][a-z]+>/i,
/TODO:\s*(fill|replace|update)/i,
/example@example\.(com|org)/i,
/FIXME/,
];
export function checkBoilerplate({ files }) {
const findings = [];
for (const [path, text] of Object.entries(files ?? {})) {
// Same discipline as the link check: code spans and fenced blocks are where
// a document ABOUT placeholders keeps its examples.
stripCode(text).split('\n').forEach((line, i) => {
for (const re of BOILERPLATE) {
if (re.test(line)) {
findings.push({
level: 'WARN',
code: 'BOILERPLATE',
bucket: 'weakening',
msg: `${path}:${i + 1} — unfilled template text: \`${line.trim().slice(0, 70)}\``,
});
return;
}
}
});
}
if (findings.length === 0) findings.push({ level: 'OK', code: 'BOILERPLATE', msg: 'no unfilled template text found' });
return findings;
}
// "LICENSE mentioned in the README, no file in the repo" is its own anti-signal:
// the claim is load-bearing for anyone deciding whether they may use this.
export function checkLicenseClaim({ readme, present }) {
const text = String(readme ?? '');
const claims = /\bLICEN[SC]E\b/i.test(text) || /\b(MIT|Apache|BSD|GPL)\b.{0,20}licen[sc]e/i.test(text);
const have = (present ?? []).some((f) => /^LICEN[SC]E(\.\w+)?$/i.test(f));
if (claims && !have) {
return [{
level: 'ERROR',
code: 'LICENSE-CLAIMED-ABSENT',
bucket: 'broken',
msg: 'README cites a licence but the repo has no LICENSE file — the claim a reader relies on to use this is unbacked',
}];
}
return [{ level: 'OK', code: 'LICENSE-CLAIM', msg: have ? 'LICENSE present' : 'no licence claim to back' }];
}
// Blank out fenced blocks and inline code spans, keeping line numbers intact.
// Documentation about regexes is full of strings that ARE markdown links to a
// naive scanner: `["']([A-Za-z0-9\-._]{16,64})["']` is `[...](...)` exactly.
// Running the first version against a real repo produced ~30 findings and every
// one of them was noise.
export function stripCode(text) {
let fenced = false;
return String(text ?? '')
.split('\n')
.map((line) => {
if (/^\s*(```|~~~)/.test(line)) {
fenced = !fenced;
return '';
}
if (fenced) return '';
if (/^(\s{4,}|\t)\S/.test(line)) return ''; // indented code block
return line.replace(/`[^`]*`/g, '');
})
.join('\n');
}
// A relative link resolves against the file it sits in, not against the repo
// root. Getting this wrong called two files missing that were right there next
// to the README linking them — and it would have done so in every nested doc.
// Returns null when the path escapes the repo, which is unresolvable from
// inside one repo rather than broken.
export function resolveRelative(fromFile, target) {
if (target.startsWith('/')) return null;
const baseParts = String(fromFile).split('/').slice(0, -1);
const out = [...baseParts];
for (const part of target.split('/')) {
if (part === '' || part === '.') continue;
if (part === '..') {
if (out.length === 0) return null;
out.pop();
} else {
out.push(part);
}
}
return out.join('/');
}
// Relative file links only. Anchor resolution depends on per-renderer heading
// slug rules and is a rabbit hole; external URLs need the network. Both are
// deliberately out — a check that is sometimes wrong teaches people to ignore it.
export function checkInternalLinks({ files, present }) {
const have = new Set(present ?? []);
const findings = [];
for (const [path, text] of Object.entries(files ?? {})) {
stripCode(text).split('\n').forEach((line, i) => {
for (const m of line.matchAll(/\[[^\]]*\]\(([^)\s]+)\)/g)) {
const target = m[1];
// Any scheme at all, not just http — `file:`, `vscode:`, `ftp:` are all
// somebody else's to resolve.
if (/^[a-z][a-z0-9+.-]*:/i.test(target) || /^[#<]/.test(target)) continue;
const clean = target.split('#')[0];
if (!clean) continue;
const resolved = resolveRelative(path, clean);
// A path that leaves the repo cannot be judged from inside it — a
// plugin README pointing up at its marketplace is the ordinary case.
if (resolved === null) {
findings.push({
level: 'SKIP',
code: 'LINK-OUTSIDE-REPO',
msg: `${path}:${i + 1} — \`${clean}\` points outside this repo; the gate sees one repo and cannot resolve it`,
});
continue;
}
if (!have.has(resolved)) {
findings.push({
level: 'ERROR',
code: 'LINK-INTERNAL-MISSING',
bucket: 'broken',
msg: `${path}:${i + 1} — link points at \`${clean}\` (${resolved}), which is not a tracked file`,
});
}
}
});
}
if (!findings.some((f) => f.level === 'ERROR')) {
findings.push({ level: 'OK', code: 'LINKS-INTERNAL', msg: 'every resolvable relative link resolves' });
}
return findings;
}
export function levelOf(findings) {
let worst = 'OK';
for (const f of findings ?? []) {
@ -279,13 +558,26 @@ export function levelOf(findings) {
return worst;
}
export function classifyRepo({ name, files, present, description }, register) {
export function bucketsOf(findings) {
const out = { broken: 0, missing: 0, weakening: 0 };
for (const f of findings ?? []) {
if (f.bucket && out[f.bucket] !== undefined) out[f.bucket] += 1;
}
return out;
}
export function classifyRepo(
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags },
register,
) {
const klass = register.repos?.[name];
if (!klass) {
return {
name,
klass: null,
traits: [],
status: 'SKIP',
buckets: { broken: 0, missing: 0, weakening: 0 },
findings: [{
level: 'SKIP',
code: 'REPO-UNREGISTERED',
@ -294,16 +586,23 @@ export function classifyRepo({ name, files, present, description }, register) {
};
}
const traits = register.traits?.[name] ?? [];
const readme = (files ?? {})['README.md'] ?? '';
const findings = [
...checkFirstScreen({ readme, name, description }),
...checkInstallBlock({ readme, name, klass }, register),
...checkRequiredFiles({ present, klass }, register),
...checkHeadings({ readme, klass, traits }, register),
...checkRequiredFiles({ present, klass, traits }, register),
...checkLinks({ files }, register),
...checkInternalLinks({ files, present }),
...checkLicenseClaim({ readme, present }),
...checkBadges({ readme }),
...checkBoilerplate({ files }),
...checkVersionConsistency({ pluginVersion, readmeBadge, changelogTop, tags }),
...checkDescription(description, register),
];
return { name, klass, status: levelOf(findings), findings };
return { name, klass, traits, status: levelOf(findings), buckets: bucketsOf(findings), findings };
}
// ---------------------------------------------------------------- I/O shell
@ -341,6 +640,49 @@ function repoNameFrom(dir) {
}
}
// The version the package itself claims, from whichever manifest this class uses.
function readPackageVersion(dir) {
for (const p of ['.claude-plugin/plugin.json', 'package.json', 'pyproject.toml']) {
const full = join(dir, p);
if (!existsSync(full)) continue;
try {
const raw = readFileSync(full, 'utf8');
if (p.endsWith('.json')) {
const v = JSON.parse(raw).version;
if (v) return String(v);
} else {
const m = /^\s*version\s*=\s*["']([^"']+)["']/m.exec(raw);
if (m) return m[1];
}
} catch { /* unparseable — try the next one */ }
}
return null;
}
export function extractBadgeVersion(readmeText) {
const m = /badge\/version-(\d+\.\d+\.\d+)/.exec(readmeText || '');
return m ? m[1] : null;
}
// Newest released version in the CHANGELOG. `## [Unreleased]` is skipped by
// design — it is not a claim that anything shipped.
export function extractChangelogTop(changelogText) {
for (const line of String(changelogText || '').split('\n')) {
const m = /^##\s*\[?v?(\d+\.\d+\.\d+)\]?/.exec(line.trim());
if (m) return m[1];
}
return null;
}
function gitTags(dir) {
try {
return execFileSync('git', ['-C', dir, 'tag', '--list', 'v*'], { encoding: 'utf8' })
.split('\n').map((s) => s.trim()).filter(Boolean);
} catch {
return [];
}
}
export function inspectRepo(dir, name, register, description) {
const tracked = gitFiles(dir);
const present = (tracked ?? []).filter((f) => existsSync(join(dir, f)));
@ -354,19 +696,53 @@ export function inspectRepo(dir, name, register, description) {
if (!files['README.md'] && existsSync(join(dir, 'README.md'))) {
files['README.md'] = readFileSync(join(dir, 'README.md'), 'utf8');
}
return classifyRepo({ name, files, present, description }, register);
const readme = files['README.md'] ?? '';
let changelog = null;
try { changelog = readFileSync(join(dir, 'CHANGELOG.md'), 'utf8'); } catch { /* absent */ }
return classifyRepo({
name,
files,
present,
description,
pluginVersion: readPackageVersion(dir),
readmeBadge: extractBadgeVersion(readme),
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
tags: gitTags(dir),
}, register);
}
// Grouped by bucket, because that is the order the findings actually get acted
// on: what blocks a stranger today, then what is absent, then what merely reads
// badly. Severity within a bucket is secondary to that.
const BUCKET_TITLE = {
broken: 'BROKEN NOW — a stranger is blocked or misled',
missing: 'MISSING — an expected artefact is absent',
weakening: 'WEAKENING — present, but it reads as amateur',
};
function render(result) {
const mark = { OK: '✓', WARN: '!', ERROR: '✗', SKIP: '·' };
const klass = result.klass ? ` [${result.klass}]` : '';
console.log(`\n${mark[result.status]} ${result.name}${klass} — ${result.status}`);
for (const f of result.findings) {
if (f.level === 'OK') continue;
console.log(` ${mark[f.level]} ${f.level} ${f.code}: ${f.msg}`);
const traits = result.traits?.length ? ` {${result.traits.join(', ')}}` : '';
console.log(`\n${mark[result.status]} ${result.name}${klass}${traits} — ${result.status}`);
for (const bucket of BUCKETS) {
const inBucket = result.findings.filter((f) => f.bucket === bucket);
if (!inBucket.length) continue;
console.log(`\n ${BUCKET_TITLE[bucket]}`);
for (const f of inBucket) console.log(` ${mark[f.level]} ${f.level} ${f.code}: ${f.msg}`);
}
const skipped = result.findings.filter((f) => f.level === 'SKIP');
if (skipped.length) {
console.log('\n NOT CHECKED — these are not passes');
for (const f of skipped) console.log(` ${mark.SKIP} ${f.code}: ${f.msg}`);
}
const okCount = result.findings.filter((f) => f.level === 'OK').length;
if (okCount) console.log(` ${mark.OK} ${okCount} check(s) passed`);
console.log(`\n ${mark.OK} ${okCount} check(s) passed`);
}
async function refresh(register) {

318
scripts/repo-standard-check.test.mjs
View file

@ -17,6 +17,13 @@ import {
checkFirstScreen,
checkInstallBlock,
checkRequiredFiles,
checkVersionConsistency,
checkHeadings,
checkBadges,
checkBoilerplate,
checkLicenseClaim,
checkInternalLinks,
resolveRelative,
classifyRepo,
levelOf,
} from './repo-standard-check.mjs';
@ -45,15 +52,27 @@ const REGISTER = {
classes: {
plugin: {
required_files: ['README.md', 'LICENSE', 'CHANGELOG.md', '.claude-plugin/plugin.json'],
required_headings: ['## Install', '## Non-goals', '## Changelog'],
install: 'plugin',
},
catalog: {
required_files: ['README.md', 'LICENSE', 'GOVERNANCE.md', 'CONVENTIONS.md', '.claude-plugin/marketplace.json'],
required_headings: ['## Install', '## Non-goals'],
install: 'catalog',
},
'shared-asset': { required_files: ['README.md', 'LICENSE'], install: 'vendor' },
'org-profile': { required_files: ['README.md'], install: 'none' },
standalone: { required_files: ['README.md', 'LICENSE'], install: 'package' },
'shared-asset': { required_files: ['README.md', 'LICENSE'], required_headings: ['## Non-goals'], install: 'vendor' },
'org-profile': { required_files: ['README.md'], required_headings: [], install: 'none' },
standalone: { required_files: ['README.md', 'LICENSE'], required_headings: ['## Install', '## Non-goals'], install: 'package' },
},
traits: {
'llm-security': ['security'],
'llm-ingestion-pipeline-security': ['security'],
},
trait_requirements: {
security: {
required_files: ['SECURITY.md'],
required_headings: ['## Known limitations'],
},
},
description_max_codepoints: 180,
};
@ -367,9 +386,17 @@ test('a fully compliant plugin repo classifies OK', () => {
'',
'Body text.',
'',
'![Version](https://img.shields.io/badge/version-0.7.0-blue)',
'',
'## Install',
`claude plugin marketplace add ${MKT.url}`,
`claude plugin install repo-mailbox@${MKT.name}`,
'',
'## Non-goals',
'It is not a state store.',
'',
'## Changelog',
'See [CHANGELOG.md](CHANGELOG.md).',
].join('\n');
const r = classifyRepo(
{
@ -377,8 +404,293 @@ test('a fully compliant plugin repo classifies OK', () => {
files: { 'README.md': readme },
present: ['README.md', 'LICENSE', 'CHANGELOG.md', '.claude-plugin/plugin.json'],
description: 'A local mailbox for coordination.',
pluginVersion: '0.7.0',
readmeBadge: '0.7.0',
changelogTop: '0.7.0',
tags: ['v0.7.0'],
},
REGISTER,
);
assert.equal(r.status, 'OK');
assert.deepEqual(r.buckets, { broken: 0, missing: 0, weakening: 0 });
});
test('the security trait travels from the register into the verdict', () => {
const r = classifyRepo(
{
name: 'llm-security',
files: { 'README.md': '# llm-security\ndesc\n## Install\n## Non-goals\n## Changelog\n' },
present: ['README.md', 'LICENSE', 'CHANGELOG.md', '.claude-plugin/plugin.json'],
description: 'desc',
},
REGISTER,
);
assert.deepEqual(r.traits, ['security']);
assert.equal(r.findings.some((f) => f.code === 'FILE-MISSING' && f.msg.includes('SECURITY.md')), true);
assert.equal(r.findings.some((f) => f.code === 'HEADING-MISSING' && f.msg.includes('Known limitations')), true);
});
// =========================================================================
// The brief's reporting contract: findings carry a BUCKET alongside a level.
// "broken now" (strangers are blocked or misled), "missing" (an expected
// artefact is absent), "weakening" (present, but reads as amateur). The two
// axes are independent: a weakening finding can still be an ERROR.
// =========================================================================
test('every ERROR and WARN finding carries a bucket', () => {
const readme = ['## Install', `"enabledPlugins": { "okr@${MKT.name}": true }`].join('\n');
const all = [
...checkInstallBlock({ readme, name: 'okr', klass: 'plugin' }, REGISTER),
...checkLinks({ files: { 'README.md': 'https://git.fromaitochitta.com/open/nonesuch' } }, REGISTER),
...checkRequiredFiles({ present: ['README.md'], klass: 'plugin' }, REGISTER),
...checkDescription('', REGISTER),
];
const graded = all.filter((f) => f.level === 'ERROR' || f.level === 'WARN');
assert.ok(graded.length > 0);
for (const f of graded) {
assert.ok(['broken', 'missing', 'weakening'].includes(f.bucket), `${f.code} has bucket ${f.bucket}`);
}
});
test('an unusable install path is `broken`, an absent file is `missing`', () => {
const readme = ['## Install', 'nothing useful here'].join('\n');
const inst = checkInstallBlock({ readme, name: 'okr', klass: 'plugin' }, REGISTER);
assert.equal(inst.find((f) => f.code === 'INSTALL-NO-CLI').bucket, 'broken');
const files = checkRequiredFiles({ present: ['README.md'], klass: 'plugin' }, REGISTER);
assert.equal(files.find((f) => f.code === 'FILE-MISSING').bucket, 'missing');
});
// ------------------------------------------------------- version consistency
test('version consistency: manifest, README badge and CHANGELOG must agree', () => {
const ok = checkVersionConsistency({ pluginVersion: '0.1.0', readmeBadge: '0.1.0', changelogTop: '0.1.0', tags: ['v0.1.0'] });
assert.equal(ok.filter((f) => f.level === 'ERROR').length, 0);
const drift = checkVersionConsistency({ pluginVersion: '0.1.0', readmeBadge: '0.0.9', changelogTop: '0.1.0', tags: ['v0.1.0'] });
assert.equal(drift.some((f) => f.level === 'ERROR' && f.code === 'VERSION-BADGE'), true);
const stale = checkVersionConsistency({ pluginVersion: '0.2.0', readmeBadge: '0.2.0', changelogTop: '0.1.0', tags: ['v0.1.0', 'v0.2.0'] });
assert.equal(stale.some((f) => f.level === 'ERROR' && f.code === 'VERSION-CHANGELOG'), true);
});
test('an untagged repo SKIPs the tag comparison rather than failing it', () => {
// Nothing has been released yet. That is a state, not a defect — and "SKIP is
// never a pass" means it must say so rather than quietly succeed.
const f = checkVersionConsistency({ pluginVersion: '0.1.0', readmeBadge: '0.1.0', changelogTop: '0.1.0', tags: [] });
assert.equal(f.some((x) => x.code === 'VERSION-TAG' && x.level === 'SKIP'), true);
assert.equal(f.some((x) => x.level === 'ERROR'), false);
});
test('a released version with no matching tag is an ERROR', () => {
const f = checkVersionConsistency({ pluginVersion: '0.3.1', readmeBadge: '0.3.1', changelogTop: '0.3.1', tags: ['v0.1.0'] });
assert.equal(f.some((x) => x.level === 'ERROR' && x.code === 'VERSION-TAG'), true);
});
// -------------------------------------------------------- required headings
test('required headings are per class — Non-goals is required, not optional', () => {
const full = '# x\n## Install\n## Non-goals\n## Changelog\n';
assert.equal(checkHeadings({ readme: full, klass: 'plugin' }, REGISTER).filter((f) => f.level === 'ERROR').length, 0);
const noNonGoals = '# x\n## Install\n## Changelog\n';
const f = checkHeadings({ readme: noNonGoals, klass: 'plugin' }, REGISTER);
assert.equal(f.some((x) => x.code === 'HEADING-MISSING' && x.msg.includes('Non-goals')), true);
assert.equal(f.find((x) => x.code === 'HEADING-MISSING').bucket, 'missing');
});
test('org-profile requires no headings at all', () => {
const f = checkHeadings({ readme: '# .profile\nprofile\n', klass: 'org-profile' }, REGISTER);
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
});
// ------------------------------------------------------------ badge honesty
test('a static badge asserting test or build status is a finding', () => {
// "tests: 642 passing" as a static image is a claim dressed as evidence.
const f = checkBadges({ readme: '![Tests](https://img.shields.io/badge/tests-34-green)' });
assert.equal(f.some((x) => x.code === 'BADGE-STATIC-CLAIM'), true);
assert.equal(f.find((x) => x.code === 'BADGE-STATIC-CLAIM').bucket, 'weakening');
});
test('static version, licence and platform badges are fine — they assert no run', () => {
const readme = [
'![Version](https://img.shields.io/badge/version-0.1.0-blue)',
'![License](https://img.shields.io/badge/license-MIT-lightgrey)',
'![Platform](https://img.shields.io/badge/platform-Claude_Code_Plugin-purple)',
].join('\n');
assert.equal(checkBadges({ readme }).filter((f) => f.level !== 'OK').length, 0);
});
test('a build badge that links to a real run is fine', () => {
const readme = '[![CI](https://forge.example/repo/badges/workflows/ci.yml/badge.svg)](https://forge.example/repo/actions)';
assert.equal(checkBadges({ readme }).filter((f) => f.level !== 'OK').length, 0);
});
// -------------------------------------------------------------- boilerplate
test('unfinished template text is a finding', () => {
const f = checkBoilerplate({ files: { 'README.md': 'Install your-project-name today' } });
assert.equal(f.some((x) => x.code === 'BOILERPLATE'), true);
const g = checkBoilerplate({ files: { 'CODE_OF_CONDUCT.md': 'Report to [INSERT EMAIL ADDRESS]' } });
assert.equal(g.some((x) => x.code === 'BOILERPLATE'), true);
});
test('ordinary prose is not boilerplate', () => {
const f = checkBoilerplate({ files: { 'README.md': 'This project solves a real problem.' } });
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
// --------------------------------------------------- licence claim vs. file
test('a README that cites a LICENSE the repo does not have is broken', () => {
const f = checkLicenseClaim({ readme: 'Released under the [MIT licence](LICENSE).', present: ['README.md'] });
assert.equal(f.some((x) => x.code === 'LICENSE-CLAIMED-ABSENT' && x.bucket === 'broken'), true);
});
test('a cited LICENSE that exists passes', () => {
const f = checkLicenseClaim({ readme: 'See [LICENSE](LICENSE).', present: ['README.md', 'LICENSE'] });
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
});
// ------------------------------------------------------------ internal links
test('a relative link to a file that does not exist is a finding', () => {
const f = checkInternalLinks(
{ files: { 'README.md': 'See [the design](docs/design.md).' }, present: ['README.md'] },
);
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING'), true);
});
test('relative links that resolve, anchors and external URLs are left alone', () => {
const f = checkInternalLinks({
files: { 'README.md': '[a](docs/design.md) [b](#section) [c](https://example.com) [d](mailto:x@y.z)' },
present: ['README.md', 'docs/design.md'],
});
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
// ----------------------------------------------------------------- traits
test('the security trait adds a SECURITY.md requirement on top of the class', () => {
const f = checkRequiredFiles({ present: ['README.md', 'LICENSE'], klass: 'standalone', traits: ['security'] }, REGISTER);
assert.equal(f.some((x) => x.code === 'FILE-MISSING' && x.msg.includes('SECURITY.md')), true);
});
test('a repo without the security trait owes no SECURITY.md', () => {
const f = checkRequiredFiles({ present: ['README.md', 'LICENSE'], klass: 'standalone' }, REGISTER);
assert.equal(f.some((x) => x.msg.includes('SECURITY.md')), false);
});
test('the security trait requires limitations to be stated', () => {
const f = checkHeadings({ readme: '# x\n## Install\n## Non-goals\n', klass: 'standalone', traits: ['security'] }, REGISTER);
assert.equal(f.some((x) => x.code === 'HEADING-MISSING' && x.msg.includes('Known limitations')), true);
});
test('CONTRIBUTING and CODE_OF_CONDUCT are required by no class — the maintainer works alone', () => {
for (const klass of Object.keys(REGISTER.classes)) {
const req = REGISTER.classes[klass].required_files;
assert.equal(req.includes('CONTRIBUTING.md'), false);
assert.equal(req.includes('CODE_OF_CONDUCT.md'), false);
assert.equal(req.includes('MAINTAINERS.md'), false);
}
});
// ---------------------------------------------- link check: the noise sources
// All three found by running against llm-security, which produced ~30 false
// positives on the first pass. A check that is wrong this often teaches people
// to ignore it, which is worse than not having it.
test('a regex inside an inline code span is not a markdown link', () => {
// `["']([A-Za-z0-9\-._]{16,64})["']` is `[...](...)` to a naive scanner.
const line = '- **Regex:** `(?i)\\bapi[_\\-]?key\\s*[:=]\\s*["\']([A-Za-z0-9\\-._]{16,64})["\']`';
const f = checkInternalLinks({ files: { 'k.md': line }, present: ['k.md'] });
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
test('fenced code blocks are not scanned for links', () => {
const text = ['```bash', 'curl [x](not-a-real-file.md)', '```'].join('\n');
const f = checkInternalLinks({ files: { 'r.md': text }, present: ['r.md'] });
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
test('any URI scheme is left alone, not just http', () => {
const text = '[a](file:///abs/path.html) [b](ftp://x/y) [c](vscode://z)';
const f = checkInternalLinks({ files: { 'r.md': text }, present: ['r.md'] });
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
test('a path that escapes the repo is unresolvable, not broken', () => {
// Legitimately common: a plugin README pointing up at its marketplace.
// The gate sees one repo, so it cannot judge — and must not pretend to.
const f = checkInternalLinks({ files: { 'README.md': '[d](../../README.md)' }, present: ['README.md'] });
assert.equal(f.some((x) => x.code === 'LINK-OUTSIDE-REPO' && x.level === 'SKIP'), true);
assert.equal(f.some((x) => x.level === 'ERROR'), false);
});
test('a genuinely missing sibling file is still an ERROR', () => {
const f = checkInternalLinks({ files: { 'README.md': '[e](docs/gone.md)' }, present: ['README.md'] });
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING' && x.level === 'ERROR'), true);
});
test('a required heading present at the wrong level says so', () => {
// llm-security has `### Install` nested under `## Quick Start`. The contract
// wants it at level 2 — the finding should name that, not just "missing".
const f = checkHeadings({ readme: '# x\n## Quick Start\n### Install\n## Non-goals\n## Changelog\n', klass: 'plugin' }, REGISTER);
const hit = f.find((x) => x.code === 'HEADING-LEVEL');
assert.ok(hit, 'expected a HEADING-LEVEL finding');
assert.match(hit.msg, /### Install/);
});
test('a document ABOUT placeholders does not trip the placeholder detector', () => {
// llm-security documents the very strings it scans for. Inside code spans and
// fenced blocks they are subject matter, not unfinished template text.
const text = [
'Skip if the matched value contains: `your-project-name`, `FIXME`, `changeme`.',
'',
'```bash',
'cd <your-fork>',
'```',
].join('\n');
const f = checkBoilerplate({ files: { 'k.md': text } });
assert.equal(f.filter((x) => x.level !== 'OK').length, 0);
});
test('placeholder text in ordinary prose is still caught', () => {
const f = checkBoilerplate({ files: { 'README.md': 'Install your-project-name to begin.' } });
assert.equal(f.some((x) => x.code === 'BOILERPLATE'), true);
});
// --------------------------------- relative links resolve against their file
test('resolveRelative joins against the containing file, not the repo root', () => {
assert.equal(resolveRelative('examples/demo/README.md', 'expected.md'), 'examples/demo/expected.md');
assert.equal(resolveRelative('README.md', 'docs/design.md'), 'docs/design.md');
assert.equal(resolveRelative('docs/a/b.md', '../c.md'), 'docs/c.md');
assert.equal(resolveRelative('docs/a.md', './b.md'), 'docs/b.md');
});
test('resolveRelative returns null when the path escapes the repo', () => {
assert.equal(resolveRelative('README.md', '../../README.md'), null);
assert.equal(resolveRelative('docs/a.md', '../../../x.md'), null);
assert.equal(resolveRelative('README.md', '/etc/passwd'), null);
});
test('a link from a nested README to its sibling resolves', () => {
// Found against llm-security: both files existed, and the gate called them
// missing because it compared a file-relative target to repo-root paths.
const f = checkInternalLinks({
files: { 'examples/demo/README.md': 'See [findings](expected-findings.md).' },
present: ['examples/demo/README.md', 'examples/demo/expected-findings.md'],
});
assert.equal(f.filter((x) => x.level === 'ERROR').length, 0);
});
test('a nested link to a genuinely absent sibling is still an ERROR', () => {
const f = checkInternalLinks({
files: { 'examples/demo/README.md': 'See [gone](gone.md).' },
present: ['examples/demo/README.md'],
});
assert.equal(f.some((x) => x.code === 'LINK-INTERNAL-MISSING'), true);
});

67
skills/repo-standard/SKILL.md
View file

@ -31,7 +31,29 @@ network call, `--json` for machine output, `--refresh` to compare the bundled
register against the live org listing.
**Never report a `SKIP` as a pass.** A SKIP means the gate could not see enough
to judge — an unreachable forge, a repo missing from the register. Say which.
to judge — an unreachable forge, an untagged repo, a link leaving the
repository. Say which.
**Report in buckets, not in a flat list.** Every finding also carries `broken`
(a stranger is blocked or misled), `missing` (an expected artefact is absent) or
`weakening` (present, but reads as amateur). That is the order the work actually
gets done in, and it is more useful to a reader than severity alone. The two
axes are independent: a README opening line that contradicts the published
description blocks nobody and is still an ERROR.
## Before you trust a run, look at the noise
The first version of the link check produced about thirty findings against a
real repository and every single one was noise: regexes inside code spans that
are `[...](...)` to a naive scanner, `file:` URLs, and relative links compared
against the repo root instead of the directory they sit in. The check now
strips code and resolves properly.
The lesson generalises, so apply it to your own reading: **when a check fires
many times in one repo, suspect the check before the repo.** A gate that is
wrong this often teaches people to ignore it, which is worse than not having it
at all. If you find a new noise source, fix the classifier and add the case as a
fixture — do not tell the user to ignore some of the output.
## What it measures, and what it cannot
@ -60,9 +82,46 @@ judgement call. `register/repos.json` holds it.
| standalone | pip/uv |
| org-profile | no install section |
A flat standard across all classes would demand a CONTRIBUTING from a CSS
library that takes no contributions and a ROADMAP from a five-line profile.
That is how a gate teaches people to switch it off.
A flat standard across all classes would demand a ROADMAP from a five-line
profile. That is how a gate teaches people to switch it off.
**Traits are a second axis.** Class is structural — readable off the catalog and
the remotes. A trait is about what the code *does*, which no remote knows.
`security` attaches what a tool owes once it handles untrusted input: a
`SECURITY.md` with a channel someone actually reads, and limitations stated with
their mechanism. Which repos carry a trait is the maintainer's judgement; do not
add or remove one on your own initiative, propose it.
## Who the reader is decides what is required
This project is solo-maintained, and that settles a whole category. Documents
written for *contributors* — `CONTRIBUTING.md`, `CODE_OF_CONDUCT.md`,
`MAINTAINERS.md`, PR templates, CLA/DCO — are required by no class here, because
there are no contributors and the published stance says so plainly. A code of
conduct with an unattended placeholder address is worse than none: it is a
visible unfinished template.
Documents written for *consumers and outsiders* are completely unaffected by
that, and working alone is never a reason to skip them: `SECURITY.md` exists for
the stranger who finds a hole, `LICENSE` for the one deciding whether they may
use it, non-goals and limitations for the one deciding whether it fits.
Do not turn this into a rule against having those files. A `CODE_OF_CONDUCT.md`
already sitting in a repo is a cleanup decision for the operator, not a gate
finding.
## Badges, and the one that is a lie
Only badges that measure something. A static image reading `tests: 642 passing`
is a claim dressed as evidence, and the gate flags it — an early draft of this
plugin's own README carried exactly that, which is how it got caught. Version,
licence and platform badges assert no run and are fine as static images.
There is no CI in this organisation: the forge has no Actions runner registered.
So do not add a CI badge, and do not pretend. The honest substitute is a single
command that runs the whole suite from a clean clone, stated in the README along
with the fact that nothing runs it automatically. Verifiability is the goal; CI
is one mechanism for it, and it is not available here.
## Two things about the install block worth knowing before you edit one