feat(engine): VERIFY-COMMAND — the one command a stranger has instead of CI
The forge has no Actions runners, so this org publishes no CI badge; the stated substitute is one command runnable from a clean clone. A repo with something runnable and no such command in its README is a WARN — and the finding names what the repo already has, so the remedy is one line. The subject is MEASURED, never read off a class. Five of 21 clones have nothing runnable at all and answer VERIFY-NONE at OK; they span plugin, shared-asset AND standalone, so every class-level phrasing of this rule would fail a correct repository somewhere. Measured: 10 document a command, 6 do not, 5 have no subject. Two things bound the rule. It adds no API call, so it has no SKIP at all — copying the null-input guard from every check since PIN-DEAD would print a false "not run". And it runs nothing, so its OK says documented, never passing. Not built, with distinct reasons recorded as invariants: RELEASE-ASSETS is rejected permanently for having NO SUBJECT (0 of 21 READMEs mention an asset download; the 18/18 fire rate is a proxy and must not be quoted as the reason). TAG-SIGNED is BLOCKED ON AN OPERATOR DECISION, not rejected — filing it with the rejections would read as settled when it is one yes/no from acquiring its whole subject. Also fixes this repo's own surface, which had drifted behind its engine: four checks had shipped without a row in the README check table, and Requirements still said "two network calls" after the third was added. 230 tests (from 213). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LwZeAZ8cHmGZofM9dryuT9
This commit is contained in:
parent
0a4373c29b
commit
7862ef5a17
7 changed files with 449 additions and 14 deletions
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "repo-standard",
|
"name": "repo-standard",
|
||||||
"version": "0.9.0",
|
"version": "0.10.0",
|
||||||
"description": "Per-repo gate for the open/ presentation standard: README first screen, install block, files required by the repo's class, and dead repo references.",
|
"description": "Per-repo gate for the open/ presentation standard: README first screen, install block, files required by the repo's class, and dead repo references.",
|
||||||
"author": {
|
"author": {
|
||||||
"name": "Kjell Tore Guttormsen"
|
"name": "Kjell Tore Guttormsen"
|
||||||
|
|
|
||||||
54
CHANGELOG.md
54
CHANGELOG.md
|
|
@ -6,8 +6,43 @@ versioning is [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
## [0.10.0] — 2026-08-12
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
||||||
|
- **`VERIFY-COMMAND` — can a stranger check that this repository works?** The
|
||||||
|
forge has no Actions runners registered, so this org publishes no CI badge;
|
||||||
|
the stated substitute is one command a stranger can run from a clean clone. A
|
||||||
|
repository with something runnable and no such command in its README is a
|
||||||
|
`WARN` in the `missing` bucket, and the finding names what the repo already
|
||||||
|
has, so the remedy is one line.
|
||||||
|
|
||||||
|
| outcome | level | code |
|
||||||
|
|---|---|---|
|
||||||
|
| the README shows a verification command | `OK` | `VERIFY-COMMAND` |
|
||||||
|
| nothing runnable exists in the repository | `OK` | `VERIFY-NONE` |
|
||||||
|
| a suite exists and the README shows no command | `WARN` | `VERIFY-MISSING` |
|
||||||
|
|
||||||
|
The subject is **measured**, never read off a class: a `scripts.test` entry, or
|
||||||
|
a tracked file that is unambiguously an executable test — not merely a file
|
||||||
|
living under `tests/`, which counted golden transcripts and fixtures and would
|
||||||
|
have made "you have tests a stranger cannot run" false on the loudest findings
|
||||||
|
it produced. Measured across all 21 registered clones: 10 document a command,
|
||||||
|
6 do not, and 5 have nothing runnable at all. Those five span three different
|
||||||
|
classes, so every class-level phrasing of this rule fails a correct repository
|
||||||
|
somewhere.
|
||||||
|
|
||||||
|
The 6-of-21 rate is acceptable for a reason that travels with the rule: the
|
||||||
|
org publishes exactly one stance on verification, so an undocumented suite is
|
||||||
|
not a second legitimate convention the way bump-first vs bump-at-release was
|
||||||
|
for the rejected `VERSION-DRIFT`.
|
||||||
|
|
||||||
|
It adds **no API call** — it reads the README and `package.json` — and so has
|
||||||
|
no `SKIP` at all, unlike every check added since `PIN-DEAD`. It can only ever
|
||||||
|
fell a MISSING command: nothing here runs anything, so its `OK` says
|
||||||
|
*documented*, never *passing*.
|
||||||
|
|
||||||
|
|
||||||
- **`RELEASE-CURRENT` — does the release page show the version the code is on?**
|
- **`RELEASE-CURRENT` — does the release page show the version the code is on?**
|
||||||
A stranger who wants a version they can name lands on the forge's release
|
A stranger who wants a version they can name lands on the forge's release
|
||||||
page, and it is the one surface refs cannot answer: a release is not a ref, so
|
page, and it is the one surface refs cannot answer: a release is not a ref, so
|
||||||
|
|
@ -75,6 +110,25 @@ versioning is [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
different limiters or the older figure is wrong, and the note cannot tell them
|
different limiters or the older figure is wrong, and the note cannot tell them
|
||||||
apart. One API-axis figure this repo can stand behind was recorded instead: 22
|
apart. One API-axis figure this repo can stand behind was recorded instead: 22
|
||||||
sequential `/api/v1/` calls at 2s pacing, anonymous, zero 429.
|
sequential `/api/v1/` calls at 2s pacing, anonymous, zero 429.
|
||||||
|
- **The README's own surface, which had drifted behind the engine.** Four checks
|
||||||
|
had shipped with no row in the check table (`PIN-DEAD`, `TAG-ANNOTATED`,
|
||||||
|
`RELEASE-CURRENT`, `REMOTE-SYNC`), and the Requirements section still said
|
||||||
|
"two network calls" after the third had been added. A stale count on the
|
||||||
|
README of the repo whose job is presentation is the defect this gate exists to
|
||||||
|
find, one file over.
|
||||||
|
|
||||||
|
### Not built
|
||||||
|
|
||||||
|
- **`RELEASE-ASSETS` — rejected permanently, for having no subject.** No release
|
||||||
|
in the org carries an asset, but the deciding measurement is the other one: 0
|
||||||
|
of 21 READMEs mention an asset download at all. Every install path here runs
|
||||||
|
through `marketplace add`, `plugin install`, `pip`/`uv` or a clone, so no
|
||||||
|
reader is harmed and no remedy would ever be performed. The fire rate alone
|
||||||
|
(18 of 18) is a proxy and must not be quoted as the reason.
|
||||||
|
- **`TAG-SIGNED` — blocked on an operator decision, which is not the same as
|
||||||
|
rejected.** No repo signs its tags today. If the operator decides this org
|
||||||
|
signs them, the check acquires its entire subject the same day; a rejection is
|
||||||
|
permanent, and this is one yes/no away.
|
||||||
|
|
||||||
## [0.9.0] — 2026-08-09
|
## [0.9.0] — 2026-08-09
|
||||||
|
|
||||||
|
|
|
||||||
43
CLAUDE.md
43
CLAUDE.md
|
|
@ -271,13 +271,54 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
||||||
repo with a correct manifest is invisible to `VERSION-TAG`. Recorded
|
repo with a correct manifest is invisible to `VERSION-TAG`. Recorded
|
||||||
rather than deferred: a decision that is wrong is worse than no record, and
|
rather than deferred: a decision that is wrong is worse than no record, and
|
||||||
the next session should not re-derive this measurement.
|
the next session should not re-derive this measurement.
|
||||||
|
- **`RELEASE-ASSETS` is rejected permanently, and the reason is NO SUBJECT — not
|
||||||
|
the fire rate.** All 18 releases in the org carry no assets, but that number is
|
||||||
|
the weaker half of the measurement. The other half: **0 of 21 READMEs mention
|
||||||
|
an asset download at all** — no `releases/download` link, no tarball, nothing.
|
||||||
|
Every install path in this org goes through `marketplace add`, `plugin
|
||||||
|
install`, `pip`/`uv`, or a clone. So there is no reader who is harmed and no
|
||||||
|
remedy anyone would perform. That is a stronger rejection than VERSION-DRIFT's,
|
||||||
|
which was "fires on repos following the other legitimate convention"; this one
|
||||||
|
has no convention on either side because it has no subject. Do not re-derive
|
||||||
|
it from "18/18" alone — the fire rate is a proxy, and reasoning from the proxy
|
||||||
|
is what would resurrect this check the first time one repo publishes an asset.
|
||||||
|
- **`TAG-SIGNED` is BLOCKED ON AN OPERATOR DECISION, not rejected — and filing it
|
||||||
|
with the rejections is the error to avoid.** 0 of 18 repos sign their tags
|
||||||
|
today, which looks like the same measurement that killed VERSION-DRIFT and is
|
||||||
|
not: if the operator decides this org signs tags, the check acquires its whole
|
||||||
|
subject the same day. A rejection is permanent; this is one yes/no away.
|
||||||
|
Recorded here so the next session reads a pending question instead of
|
||||||
|
re-measuring a settled one. The scope is also fixed in advance: `git cat-file
|
||||||
|
tag` reads the signature block from the clone, so the check can answer *is it
|
||||||
|
signed* and must never claim *the forge vouches for it* — the forge's
|
||||||
|
"Verified" badge needs a key registered there.
|
||||||
|
- **A gate whose subject varies per repo MEASURES the subject; it does not read
|
||||||
|
it off the class.** `VERIFY-COMMAND` fires only where something runnable
|
||||||
|
exists (`scripts.test`, or a tracked executable test file), and answers
|
||||||
|
`VERIFY-NONE` at `OK` where nothing does. The five subject-less repos —
|
||||||
|
human-friendly-style, llm-security-commons, playground-design-system,
|
||||||
|
portfolio-optimiser-commons, app-creator — span `plugin`, `shared-asset` AND
|
||||||
|
`standalone`, so every class-level phrasing of this rule fails a correct
|
||||||
|
repository somewhere. The rate is fine (6 of 21 `WARN`) for a reason that must
|
||||||
|
travel with the rule: the org publishes exactly ONE stance on verification
|
||||||
|
("no CI badge, so one command from a clean clone"), so an undocumented suite
|
||||||
|
is not a second convention the way bump-first vs bump-at-release was for
|
||||||
|
VERSION-DRIFT. And the check runs nothing, so its `OK` says *documented*,
|
||||||
|
never *passing* — a green line implying a passing suite would be a claim on
|
||||||
|
the surface that nobody verified.
|
||||||
|
- **The check table in the README is part of the surface it measures.** Four
|
||||||
|
checks shipped without a row (`PIN-DEAD`, `TAG-ANNOTATED`, `RELEASE-CURRENT`,
|
||||||
|
`REMOTE-SYNC`), and the Requirements section still said "two network calls"
|
||||||
|
after the third was added — the same stale-count defect the invariant above
|
||||||
|
warns about, one file over, on the README of the repo whose entire job is
|
||||||
|
presentation. A new check is not shipped until its row exists.
|
||||||
- **No hook until the rule is precise.** A blocking gate that fails a correct
|
- **No hook until the rule is precise.** A blocking gate that fails a correct
|
||||||
repository is the mechanism that gets gates switched off.
|
repository is the mechanism that gets gates switched off.
|
||||||
|
|
||||||
## Commands
|
## Commands
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
npm test # 213 tests
|
npm test # 230 tests
|
||||||
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
|
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
|
||||||
node scripts/repo-standard-check.mjs --offline # no network call
|
node scripts/repo-standard-check.mjs --offline # no network call
|
||||||
node scripts/repo-standard-check.mjs --json # machine output
|
node scripts/repo-standard-check.mjs --json # machine output
|
||||||
|
|
|
||||||
49
README.md
49
README.md
|
|
@ -10,7 +10,7 @@ checks that surface in one repository and reports what it finds.
|
||||||
|
|
||||||
*AI-generated: all code produced by Claude Code through dialog-driven development.*
|
*AI-generated: all code produced by Claude Code through dialog-driven development.*
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -41,12 +41,17 @@ a replacement for the two commands above:
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
Node 18 or newer. No dependencies. Two network calls: the org listing (for a
|
Node 18 or newer. No dependencies. Three API calls: the org listing (for a
|
||||||
repo's published description) and the catalog manifest (to confirm the install
|
repo's published description), the catalog manifest (to confirm the install
|
||||||
command resolves). Both read anonymously, so no token is needed, and `--offline`
|
command resolves), and the repo's releases listing (to see the version page a
|
||||||
skips both — the checks that depended on them then report `SKIP`, not `OK`.
|
stranger reads). Refs come over the git protocol instead — `git ls-remote
|
||||||
Those land as `notRun` and clear on a networked re-run, so an offline pass is
|
--tags`, which does not spend the API budget — and a release is not a ref, which
|
||||||
never a fully judged one.
|
is why it is the one subject that needs a call of its own.
|
||||||
|
|
||||||
|
All of it reads anonymously, so no token is needed, and `--offline` skips every
|
||||||
|
one — the checks that depended on them then report `SKIP`, not `OK`. Those land
|
||||||
|
as `notRun` and clear on a networked re-run, so an offline pass is never a fully
|
||||||
|
judged one.
|
||||||
|
|
||||||
## What it does
|
## What it does
|
||||||
|
|
||||||
|
|
@ -63,6 +68,7 @@ The repository's **class** decides what each check means:
|
||||||
| First screen | no H1 on line 1, or the line under it is not the published description. An H1 that merely differs from the repo name is a `WARN` — that is a naming choice, not a defect |
|
| First screen | no H1 on line 1, or the line under it is not the published description. An H1 that merely differs from the repo name is a `WARN` — that is a naming choice, not a defect |
|
||||||
| Install block | the form for this class is missing, incomplete, shown over `ssh://`, or points at the wrong marketplace |
|
| Install block | the form for this class is missing, incomplete, shown over `ssh://`, or points at the wrong marketplace |
|
||||||
| Install truth | the plugin is not pinned in the catalog, so the documented command cannot succeed for anyone |
|
| Install truth | the plugin is not pinned in the catalog, so the documented command cannot succeed for anyone |
|
||||||
|
| Install pins | a `@v…` or `--branch v…` in an install command names a tag the forge does not have, so the command a stranger copies 404s |
|
||||||
| Required headings | `## Install`, `## Non-goals`, `## Changelog` — per class. Present at the wrong depth is its own finding |
|
| Required headings | `## Install`, `## Non-goals`, `## Changelog` — per class. Present at the wrong depth is its own finding |
|
||||||
| Required files | a file this class (or trait) needs is absent |
|
| Required files | a file this class (or trait) needs is absent |
|
||||||
| Repo references | an `open/<name>` in URL position resolves to nothing |
|
| Repo references | an `open/<name>` in URL position resolves to nothing |
|
||||||
|
|
@ -73,6 +79,10 @@ The repository's **class** decides what each check means:
|
||||||
| README language | the prose is not in the language this repo's readers were declared to speak |
|
| README language | the prose is not in the language this repo's readers were declared to speak |
|
||||||
| Boilerplate | template text nobody filled in |
|
| Boilerplate | template text nobody filled in |
|
||||||
| Version consistency | manifest, README badge, newest CHANGELOG entry and the git tag disagree |
|
| Version consistency | manifest, README badge, newest CHANGELOG entry and the git tag disagree |
|
||||||
|
| Tag integrity | the newest tag is lightweight — movable without a trace, under a catalog that pins by tag. Older ones are exposed once as a count, because their only "fix" is force-moving a published ref |
|
||||||
|
| Release current | the newest release the forge publishes is older than the newest tag it lists, so the version page shows a version older than the code. `WARN` — publishing a release for an existing tag moves nothing |
|
||||||
|
| Remote sync | a tag exists only in this clone. The manifest can read as correctly tagged while no stranger can resolve the version |
|
||||||
|
| Verify command | the repo has something runnable and the README shows no command to run it. With no CI badge in this org, that command is the only evidence a stranger gets |
|
||||||
| Description | empty, or past the length bound |
|
| Description | empty, or past the length bound |
|
||||||
|
|
||||||
Findings carry two independent things: a **level** (`ERROR`, `WARN`, `SKIP`,
|
Findings carry two independent things: a **level** (`ERROR`, `WARN`, `SKIP`,
|
||||||
|
|
@ -114,9 +124,9 @@ catalog, an unregistered repo, a repo that has not been tagged yet.
|
||||||
The summary line names only what someone has an action on:
|
The summary line names only what someone has an action on:
|
||||||
|
|
||||||
```
|
```
|
||||||
✓ portfolio-optimiser [standalone] — OK (repo-standard v0.9.0 @<sha>)
|
✓ portfolio-optimiser [standalone] — OK (repo-standard v0.10.0 @<sha>)
|
||||||
✗ some-plugin [plugin] — ERROR · 2 not run (repo-standard v0.9.0 @<sha>)
|
✗ some-plugin [plugin] — ERROR · 2 not run (repo-standard v0.10.0 @<sha>)
|
||||||
· stranger — SKIP · 1 not run (repo-standard v0.9.0 @<sha>)
|
· stranger — SKIP · 1 not run (repo-standard v0.10.0 @<sha>)
|
||||||
```
|
```
|
||||||
|
|
||||||
The deliberate ones are not silenced: they keep their own `NOT JUDGED` heading
|
The deliberate ones are not silenced: they keep their own `NOT JUDGED` heading
|
||||||
|
|
@ -186,6 +196,25 @@ A class turns the demand off with `readme_desc_match: false` in the register.
|
||||||
The exemption is still reported as a finding that names the reason: an exception
|
The exemption is still reported as a finding that names the reason: an exception
|
||||||
nobody can see is indistinguishable from a check that silently stopped running.
|
nobody can see is indistinguishable from a check that silently stopped running.
|
||||||
|
|
||||||
|
### One command, because there is no CI
|
||||||
|
|
||||||
|
The forge has no Actions runners registered. A green badge here would assert a
|
||||||
|
run that never happens, which is the dishonest-badge finding one row up. The
|
||||||
|
substitute this project publishes is one command a stranger can run from a clean
|
||||||
|
clone, said plainly — so a repository with something runnable and no such command
|
||||||
|
in its README is a `WARN`.
|
||||||
|
|
||||||
|
Two things bound it. It fires only where a subject was **measured**: a
|
||||||
|
`scripts.test` entry, or a tracked file that is unambiguously an executable
|
||||||
|
test. Five of the twenty-one clones have neither — they hold prose, output styles
|
||||||
|
and domain packs — and they answer `VERIFY-NONE` at `OK`, because a check that
|
||||||
|
ran and found nothing to judge has produced a verdict. Those five span three
|
||||||
|
different classes, which is why the subject is measured and not read off a class.
|
||||||
|
|
||||||
|
And it can only ever fell a **missing** command. Nothing here runs anything, so a
|
||||||
|
green line means documented, never passing. Measured across the org: six
|
||||||
|
repositories have a suite their README never mentions.
|
||||||
|
|
||||||
### What is deliberately not required
|
### What is deliberately not required
|
||||||
|
|
||||||
`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` and `MAINTAINERS.md` are required by no
|
`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` and `MAINTAINERS.md` are required by no
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
{
|
{
|
||||||
"name": "repo-standard",
|
"name": "repo-standard",
|
||||||
"version": "0.9.0",
|
"version": "0.10.0",
|
||||||
"private": true,
|
"private": true,
|
||||||
"type": "module",
|
"type": "module",
|
||||||
"engines": {
|
"engines": {
|
||||||
|
|
|
||||||
|
|
@ -861,6 +861,86 @@ export function checkRemoteSync({ tags, forgeTagsSelf }) {
|
||||||
}];
|
}];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The families a clean clone actually runs, read off the corpus rather than
|
||||||
|
// imagined: npm/pnpm/yarn scripts, `node --test`, a named test file, pytest,
|
||||||
|
// make, a shell test script, and the `--selftest` flag repo-mailbox ships.
|
||||||
|
// `npm install` must NOT match — the install block is fenced in every repo in
|
||||||
|
// the org, and matching it would hand a green line to every repo this check
|
||||||
|
// exists to find.
|
||||||
|
const VERIFY_COMMAND = new RegExp([
|
||||||
|
'(^|\\s)(npm|pnpm|yarn)\\s+(run\\s+\\S*test\\S*|test)\\b',
|
||||||
|
'(^|\\s)node\\s+--test\\b',
|
||||||
|
'\\.test\\.(mjs|cjs|js|ts)\\b',
|
||||||
|
'(^|\\s)(python3?\\s+-m\\s+)?pytest\\b',
|
||||||
|
'(^|\\s)make\\s+(test|check)\\b',
|
||||||
|
'--selftest\\b',
|
||||||
|
'(^|[\\s./])\\S*(test|selftest)\\S*\\.sh\\b',
|
||||||
|
].join('|'));
|
||||||
|
|
||||||
|
// The complement of `stripCode`, and deliberately derived FROM it: the link
|
||||||
|
// checks need code removed, this one needs exactly what was removed. A second
|
||||||
|
// hand-rolled fence parser is how two copies of one rule drift apart.
|
||||||
|
export function codeLines(text) {
|
||||||
|
const src = String(text ?? '').split('\n');
|
||||||
|
const stripped = stripCode(text).split('\n');
|
||||||
|
return src.filter((line, i) => stripped[i] === ''
|
||||||
|
&& line.trim() !== ''
|
||||||
|
&& !/^\s*(```|~~~)/.test(line));
|
||||||
|
}
|
||||||
|
|
||||||
|
// There is no CI badge in this org because there is no CI — the published
|
||||||
|
// substitute, stated in this repo's own README, is one command a stranger can
|
||||||
|
// run from a clean clone. That SINGLE published stance is what licenses a check
|
||||||
|
// that fires on a third of the org: VERSION-DRIFT was rejected because twelve
|
||||||
|
// of the fifteen repos it felled were simply following the other legitimate
|
||||||
|
// convention, and here there is no other convention. A repo with a runnable
|
||||||
|
// suite and no documented command is not on a different plan; it is
|
||||||
|
// undocumented.
|
||||||
|
//
|
||||||
|
// The subject is MEASURED, never read off a class. Across all 21 registered
|
||||||
|
// clones (2026-08-12), 16 have something runnable and 5 do not —
|
||||||
|
// human-friendly-style, llm-security-commons, playground-design-system,
|
||||||
|
// portfolio-optimiser-commons and app-creator hold prose, output styles and
|
||||||
|
// domain packs. Those five span the `plugin`, `shared-asset` and `standalone`
|
||||||
|
// classes, so any class-level requirement would have failed a correct
|
||||||
|
// repository somewhere. Nothing to verify is an OK, the RELEASE-NONE shape.
|
||||||
|
//
|
||||||
|
// What this check can NEVER do is report that a documented command works — it
|
||||||
|
// runs nothing. It fells a missing command and nothing else, and the message
|
||||||
|
// says so, because a green line implying a passing suite is a claim on the
|
||||||
|
// surface that nobody verified. It also reads only the README and package.json,
|
||||||
|
// so unlike every check since PIN-DEAD it has no null network input and
|
||||||
|
// therefore no SKIP at all.
|
||||||
|
export function checkVerifyCommand({ readme, testScript, testFileCount }) {
|
||||||
|
const count = Number(testFileCount ?? 0);
|
||||||
|
if (!testScript && count === 0) {
|
||||||
|
return [{
|
||||||
|
level: 'OK',
|
||||||
|
code: 'VERIFY-NONE',
|
||||||
|
msg: 'no test script and no tracked test file — nothing here a stranger could run, so no verification command is owed',
|
||||||
|
}];
|
||||||
|
}
|
||||||
|
|
||||||
|
const found = codeLines(readme).find((l) => VERIFY_COMMAND.test(l));
|
||||||
|
if (found) {
|
||||||
|
return [{
|
||||||
|
level: 'OK',
|
||||||
|
code: 'VERIFY-COMMAND',
|
||||||
|
msg: `README shows \`${found.trim()}\` — a stranger has one command to run. This gate does not run it, so this says documented, never passing.`,
|
||||||
|
}];
|
||||||
|
}
|
||||||
|
|
||||||
|
const have = testScript
|
||||||
|
? `\`${testScript}\` is defined in package.json`
|
||||||
|
: `${count} tracked test file(s) exist`;
|
||||||
|
return [{
|
||||||
|
level: 'WARN',
|
||||||
|
code: 'VERIFY-MISSING',
|
||||||
|
bucket: 'missing',
|
||||||
|
msg: `${have}, but no README code block shows a command to run them — with no CI badge to fall back on, a stranger has no way to check this repo works. Show the command in a fenced block.`,
|
||||||
|
}];
|
||||||
|
}
|
||||||
|
|
||||||
// Counting badges needs a NARROWER rule than detecting a dishonest one. The
|
// Counting badges needs a NARROWER rule than detecting a dishonest one. The
|
||||||
// claim check reads any image, any host, on purpose. Here the opposite error
|
// claim check reads any image, any host, on purpose. Here the opposite error
|
||||||
// matters: counting a screenshot or an architecture diagram as clutter would
|
// matters: counting a screenshot or an architecture diagram as clutter would
|
||||||
|
|
@ -1298,7 +1378,7 @@ export function bucketsOf(findings) {
|
||||||
}
|
}
|
||||||
|
|
||||||
export function classifyRepo(
|
export function classifyRepo(
|
||||||
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames, forgeTagsByRepo, forgeTagsSelf, releases },
|
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames, forgeTagsByRepo, forgeTagsSelf, releases, testScript, testFileCount },
|
||||||
register,
|
register,
|
||||||
) {
|
) {
|
||||||
const klass = register.repos?.[name];
|
const klass = register.repos?.[name];
|
||||||
|
|
@ -1339,6 +1419,7 @@ export function classifyRepo(
|
||||||
...checkTagIntegrity({ tagObjects, name }, register),
|
...checkTagIntegrity({ tagObjects, name }, register),
|
||||||
...checkReleaseCurrent({ forgeTagsSelf, releases }),
|
...checkReleaseCurrent({ forgeTagsSelf, releases }),
|
||||||
...checkRemoteSync({ tags, forgeTagsSelf }),
|
...checkRemoteSync({ tags, forgeTagsSelf }),
|
||||||
|
...checkVerifyCommand({ readme, testScript, testFileCount }),
|
||||||
...checkDescription(description, register),
|
...checkDescription(description, register),
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|
@ -1505,6 +1586,41 @@ function readPackageVersion(dir) {
|
||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// The declared way to run this repo's tests, if there is one. Only
|
||||||
|
// package.json carries it — a plugin manifest has no scripts, and pyproject's
|
||||||
|
// runner is not a command a stranger can copy.
|
||||||
|
function readTestScript(dir) {
|
||||||
|
const full = join(dir, 'package.json');
|
||||||
|
if (!existsSync(full)) return null;
|
||||||
|
try {
|
||||||
|
const scripts = JSON.parse(readFileSync(full, 'utf8')).scripts ?? {};
|
||||||
|
return scripts.test ? String(scripts.test) : null;
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Files that are unambiguously EXECUTABLE tests, not merely files living under
|
||||||
|
// `tests/`. The looser rule counted golden files, fixtures and transcripts —
|
||||||
|
// portfolio-optimiser's `tests/golden/demo-transcript.stdout` among them — and
|
||||||
|
// a finding that says "you have tests a stranger cannot run" is false the
|
||||||
|
// moment its subject is a fixture. Measured: 16 of 21 clones have a real one.
|
||||||
|
// One alternative per family rather than one regex, because the shell-script
|
||||||
|
// family needs a boundary the others do not: a bare `test` substring makes
|
||||||
|
// `latest-release.sh` a test file.
|
||||||
|
const EXECUTABLE_TEST_FILE = [
|
||||||
|
/\.test\.(mjs|cjs|js|ts)$/,
|
||||||
|
/(^|\/)test_[^/]+\.py$/,
|
||||||
|
/_test\.py$/,
|
||||||
|
/(^|\/)([^/]*[-_.])?tests?([-_.][^/]*)?\.sh$/,
|
||||||
|
/(^|\/)tests?\/[^/]*\.sh$/,
|
||||||
|
/selftest/,
|
||||||
|
];
|
||||||
|
|
||||||
|
export function countTestFiles(tracked) {
|
||||||
|
return (tracked ?? []).filter((f) => EXECUTABLE_TEST_FILE.some((re) => re.test(f))).length;
|
||||||
|
}
|
||||||
|
|
||||||
export function extractBadgeVersion(readmeText) {
|
export function extractBadgeVersion(readmeText) {
|
||||||
const m = /badge\/version-(\d+\.\d+\.\d+)/.exec(readmeText || '');
|
const m = /badge\/version-(\d+\.\d+\.\d+)/.exec(readmeText || '');
|
||||||
return m ? m[1] : null;
|
return m ? m[1] : null;
|
||||||
|
|
@ -1621,6 +1737,8 @@ export function inspectRepo(dir, name, register, description, catalogNames = nul
|
||||||
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
|
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
|
||||||
tags: gitTags(dir),
|
tags: gitTags(dir),
|
||||||
tagObjects: gitTagObjects(dir),
|
tagObjects: gitTagObjects(dir),
|
||||||
|
testScript: readTestScript(dir),
|
||||||
|
testFileCount: countTestFiles(tracked),
|
||||||
catalogNames,
|
catalogNames,
|
||||||
forgeTagsSelf,
|
forgeTagsSelf,
|
||||||
releases,
|
releases,
|
||||||
|
|
|
||||||
|
|
@ -43,6 +43,9 @@ import {
|
||||||
skipsOf,
|
skipsOf,
|
||||||
checkReleaseCurrent,
|
checkReleaseCurrent,
|
||||||
checkRemoteSync,
|
checkRemoteSync,
|
||||||
|
checkVerifyCommand,
|
||||||
|
countTestFiles,
|
||||||
|
codeLines,
|
||||||
} from './repo-standard-check.mjs';
|
} from './repo-standard-check.mjs';
|
||||||
|
|
||||||
const REGISTER = {
|
const REGISTER = {
|
||||||
|
|
@ -2159,3 +2162,193 @@ test('unfiltered forge refs cannot manufacture an unpushed-tag ERROR either', ()
|
||||||
const f = checkRemoteSync({ tags: ['v7.7.2'], forgeTagsSelf: ['v7.7.2', 'pre-polyrepo-archive'] });
|
const f = checkRemoteSync({ tags: ['v7.7.2'], forgeTagsSelf: ['v7.7.2', 'pre-polyrepo-archive'] });
|
||||||
assert.equal(f[0].level, 'OK');
|
assert.equal(f[0].level, 'OK');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// ----------------------------------------------------------- VERIFY-COMMAND
|
||||||
|
//
|
||||||
|
// The org publishes no CI badge because it has no CI — this repo's own stated
|
||||||
|
// substitute is "one command from a clean clone, said plainly". That single
|
||||||
|
// published stance is what licenses a check firing on a third of the org:
|
||||||
|
// unlike VERSION-DRIFT, there is no second legitimate convention a repo could
|
||||||
|
// be following. A repo with a runnable suite and no documented command is not
|
||||||
|
// on the other convention; it is undocumented.
|
||||||
|
//
|
||||||
|
// Measured 2026-08-12 across all 21 registered clones. Subject present (a
|
||||||
|
// `scripts.test` entry or a tracked executable test file) in 16 of 21; the
|
||||||
|
// five without one — human-friendly-style, llm-security-commons,
|
||||||
|
// playground-design-system, portfolio-optimiser-commons, app-creator — hold
|
||||||
|
// prose, output styles and domain packs, nothing a stranger could run. Those
|
||||||
|
// five are exactly the repos a class-level requirement would have failed for
|
||||||
|
// being correct, which is why the subject is MEASURED and not read off a class.
|
||||||
|
//
|
||||||
|
// The check can only ever fell a MISSING command. It never runs one, so it can
|
||||||
|
// never report that a documented command works — the message must not imply it.
|
||||||
|
|
||||||
|
test('a repo with nothing runnable is an OK, not a skip — the check found no subject', () => {
|
||||||
|
// The VERSION-NONE / RELEASE-NONE shape: the check ran, saw everything, and
|
||||||
|
// there was nothing here to be wrong.
|
||||||
|
const f = checkVerifyCommand({ readme: '# x\n', testScript: null, testFileCount: 0 });
|
||||||
|
assert.equal(f[0].level, 'OK');
|
||||||
|
assert.equal(f[0].code, 'VERIFY-NONE');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a suite with no command anywhere in the README is a WARN in the missing bucket', () => {
|
||||||
|
const f = checkVerifyCommand({
|
||||||
|
readme: '# okr\n\nA plugin.\n',
|
||||||
|
testScript: 'node --test tests/',
|
||||||
|
testFileCount: 24,
|
||||||
|
});
|
||||||
|
assert.equal(f[0].level, 'WARN');
|
||||||
|
assert.equal(f[0].code, 'VERIFY-MISSING');
|
||||||
|
assert.equal(f[0].bucket, 'missing');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the WARN names the command the repo already has, so the remedy is one line', () => {
|
||||||
|
const f = checkVerifyCommand({ readme: '# okr\n', testScript: 'node --test tests/', testFileCount: 24 });
|
||||||
|
assert.match(f[0].msg, /node --test tests\//);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a suite with no `scripts.test` still gets a WARN, and it names the file count instead', () => {
|
||||||
|
const f = checkVerifyCommand({ readme: '# config-audit\n', testScript: null, testFileCount: 106 });
|
||||||
|
assert.equal(f[0].code, 'VERIFY-MISSING');
|
||||||
|
assert.match(f[0].msg, /106/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a fenced `npm test` is an OK naming the command it found', () => {
|
||||||
|
const readme = '# x\n\n## Tests\n\n```bash\nnpm test\n```\n';
|
||||||
|
const f = checkVerifyCommand({ readme, testScript: 'node --test', testFileCount: 3 });
|
||||||
|
assert.equal(f[0].level, 'OK');
|
||||||
|
assert.equal(f[0].code, 'VERIFY-COMMAND');
|
||||||
|
assert.match(f[0].msg, /npm test/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the OK never claims the command works — this gate does not run it', () => {
|
||||||
|
// The one thing this check must not oversell. It can fell a missing command
|
||||||
|
// and nothing else; a green line here means "documented", never "passing".
|
||||||
|
const readme = '# x\n\n```bash\nnpm test\n```\n';
|
||||||
|
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
|
||||||
|
assert.doesNotMatch(f[0].msg, /passe?s|works|green|verified/i);
|
||||||
|
assert.match(f[0].msg, /not run|does not run/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a command in prose does not count — a stranger copies out of a code block', () => {
|
||||||
|
const readme = '# x\n\nRun npm test to check it.\n';
|
||||||
|
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
|
||||||
|
assert.equal(f[0].code, 'VERIFY-MISSING');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('an indented code block counts as one — Markdown has two fences', () => {
|
||||||
|
const readme = '# x\n\nTests:\n\n pytest -q\n';
|
||||||
|
const f = checkVerifyCommand({ readme, testScript: null, testFileCount: 25 });
|
||||||
|
assert.equal(f[0].code, 'VERIFY-COMMAND');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('`npm install` is not a verification command', () => {
|
||||||
|
// The install block is fenced in every repo in the org. Matching it would
|
||||||
|
// hand a green line to every repo the check exists to find.
|
||||||
|
const readme = '# x\n\n## Install\n\n```bash\nnpm install\n```\n';
|
||||||
|
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
|
||||||
|
assert.equal(f[0].code, 'VERIFY-MISSING');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('the families a clean clone actually runs are all recognised', () => {
|
||||||
|
// Measured from the corpus rather than imagined: npm/pnpm/yarn scripts,
|
||||||
|
// `node --test`, a named test file, pytest, make, a shell test script, and
|
||||||
|
// the `--selftest` flag repo-mailbox ships.
|
||||||
|
const cases = [
|
||||||
|
'npm test',
|
||||||
|
'npm run test:unit',
|
||||||
|
'pnpm test',
|
||||||
|
'yarn test',
|
||||||
|
'node --test',
|
||||||
|
'node scripts/repo-standard-check.test.mjs',
|
||||||
|
'pytest',
|
||||||
|
'python3 -m pytest tests/',
|
||||||
|
'make check',
|
||||||
|
'./scripts/coord-selftest.sh',
|
||||||
|
'bash tests/run-tests.sh',
|
||||||
|
'coord-selftest --selftest',
|
||||||
|
];
|
||||||
|
for (const cmd of cases) {
|
||||||
|
const f = checkVerifyCommand({
|
||||||
|
readme: `# x\n\n\`\`\`bash\n${cmd}\n\`\`\`\n`,
|
||||||
|
testScript: null,
|
||||||
|
testFileCount: 1,
|
||||||
|
});
|
||||||
|
assert.equal(f[0].code, 'VERIFY-COMMAND', `not recognised: ${cmd}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a `$` prompt prefix does not hide the command', () => {
|
||||||
|
const f = checkVerifyCommand({ readme: '# x\n\n```\n$ npm test\n```\n', testScript: null, testFileCount: 1 });
|
||||||
|
assert.equal(f[0].code, 'VERIFY-COMMAND');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('VERIFY-COMMAND reads no network input, so it has no SKIP at all', () => {
|
||||||
|
// Every check written since PIN-DEAD opens by guarding a null network input.
|
||||||
|
// This one reads the README and package.json only — copying that reflex would
|
||||||
|
// print a false "not run" for a check that ran perfectly well offline.
|
||||||
|
for (const args of [
|
||||||
|
{ readme: '', testScript: null, testFileCount: 0 },
|
||||||
|
{ readme: '', testScript: 'npm test', testFileCount: 0 },
|
||||||
|
{ readme: '```\nnpm test\n```', testScript: null, testFileCount: 9 },
|
||||||
|
]) {
|
||||||
|
assert.equal(checkVerifyCommand(args).every((f) => f.level !== 'SKIP'), true);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('codeLines returns what stripCode blanked, and nothing else', () => {
|
||||||
|
// One fence state machine, not two. The link checks strip code out; this one
|
||||||
|
// needs exactly the complement, and a second hand-rolled parser is how the
|
||||||
|
// two copies drift.
|
||||||
|
const md = '# t\n\nprose npm test\n\n```bash\nnpm test\n```\n\n pytest -q\n';
|
||||||
|
const lines = codeLines(md);
|
||||||
|
assert.deepEqual(lines.map((l) => l.trim()), ['npm test', 'pytest -q']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('classifyRepo carries the verification inputs through to a finding', () => {
|
||||||
|
// The wiring, not the rule: a check that is never called from the pipeline
|
||||||
|
// passes its own unit tests forever while measuring nothing.
|
||||||
|
const r = classifyRepo(
|
||||||
|
{ name: 'repo-mailbox', files: {}, present: [], description: null, testScript: 'npm test', testFileCount: 4 },
|
||||||
|
REGISTER,
|
||||||
|
);
|
||||||
|
assert.equal(r.findings.some((f) => f.code === 'VERIFY-MISSING'), true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('countTestFiles counts executable tests, not everything under tests/', () => {
|
||||||
|
// The looser rule counted golden files and transcripts. A WARN saying "you
|
||||||
|
// have tests a stranger cannot run" is false the moment its subject is a
|
||||||
|
// fixture — measured on portfolio-optimiser's tests/golden/*.stdout.
|
||||||
|
const tracked = [
|
||||||
|
'scripts/x.test.mjs',
|
||||||
|
'tests/test_engine.py',
|
||||||
|
'tests/engine_test.py',
|
||||||
|
'scripts/coord-selftest.sh',
|
||||||
|
'tests/run-tests.sh',
|
||||||
|
'tests/golden/demo-transcript.stdout',
|
||||||
|
'tests/conftest.py',
|
||||||
|
'tests/fixtures/sample.md',
|
||||||
|
'README.md',
|
||||||
|
];
|
||||||
|
assert.equal(countTestFiles(tracked), 5);
|
||||||
|
assert.equal(countTestFiles([]), 0);
|
||||||
|
assert.equal(countTestFiles(null), 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a `test` substring alone does not make a shell script a test', () => {
|
||||||
|
// `latest-release.sh` contains `test`. The shell family needs a boundary the
|
||||||
|
// other families get for free from their suffixes.
|
||||||
|
assert.equal(countTestFiles(['scripts/latest-release.sh', 'tests/conftest.py']), 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('a directory separator is a name boundary too, or a whole repo reads as having no tests', () => {
|
||||||
|
// claude-design ships five shell tests under `tests/` and read as VERIFY-NONE
|
||||||
|
// — a false "nothing here to verify", which is a claim about the repo, not a
|
||||||
|
// missing finding. `^` only anchors the whole string; the basename of
|
||||||
|
// `tests/test-sc1-dogfood-log.sh` starts after a slash.
|
||||||
|
assert.equal(countTestFiles([
|
||||||
|
'tests/test-sc1-dogfood-log.sh',
|
||||||
|
'tests/test-skill-triggers.sh',
|
||||||
|
'tests/validate-plugin.sh',
|
||||||
|
]), 3);
|
||||||
|
});
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue