feat(engine): VERIFY-COMMAND — the one command a stranger has instead of CI

The forge has no Actions runners, so this org publishes no CI badge; the
stated substitute is one command runnable from a clean clone. A repo with
something runnable and no such command in its README is a WARN — and the
finding names what the repo already has, so the remedy is one line.

The subject is MEASURED, never read off a class. Five of 21 clones have
nothing runnable at all and answer VERIFY-NONE at OK; they span plugin,
shared-asset AND standalone, so every class-level phrasing of this rule
would fail a correct repository somewhere. Measured: 10 document a
command, 6 do not, 5 have no subject.

Two things bound the rule. It adds no API call, so it has no SKIP at all
— copying the null-input guard from every check since PIN-DEAD would
print a false "not run". And it runs nothing, so its OK says documented,
never passing.

Not built, with distinct reasons recorded as invariants: RELEASE-ASSETS
is rejected permanently for having NO SUBJECT (0 of 21 READMEs mention an
asset download; the 18/18 fire rate is a proxy and must not be quoted as
the reason). TAG-SIGNED is BLOCKED ON AN OPERATOR DECISION, not rejected
— filing it with the rejections would read as settled when it is one
yes/no from acquiring its whole subject.

Also fixes this repo's own surface, which had drifted behind its engine:
four checks had shipped without a row in the README check table, and
Requirements still said "two network calls" after the third was added.

230 tests (from 213).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwZeAZ8cHmGZofM9dryuT9
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 23:21:34 +02:00
commit e00ed3340c
7 changed files with 449 additions and 14 deletions

View file

@ -861,6 +861,86 @@ export function checkRemoteSync({ tags, forgeTagsSelf }) {
}];
}
// The families a clean clone actually runs, read off the corpus rather than
// imagined: npm/pnpm/yarn scripts, `node --test`, a named test file, pytest,
// make, a shell test script, and the `--selftest` flag repo-mailbox ships.
// `npm install` must NOT match — the install block is fenced in every repo in
// the org, and matching it would hand a green line to every repo this check
// exists to find.
const VERIFY_COMMAND = new RegExp([
'(^|\\s)(npm|pnpm|yarn)\\s+(run\\s+\\S*test\\S*|test)\\b',
'(^|\\s)node\\s+--test\\b',
'\\.test\\.(mjs|cjs|js|ts)\\b',
'(^|\\s)(python3?\\s+-m\\s+)?pytest\\b',
'(^|\\s)make\\s+(test|check)\\b',
'--selftest\\b',
'(^|[\\s./])\\S*(test|selftest)\\S*\\.sh\\b',
].join('|'));
// The complement of `stripCode`, and deliberately derived FROM it: the link
// checks need code removed, this one needs exactly what was removed. A second
// hand-rolled fence parser is how two copies of one rule drift apart.
export function codeLines(text) {
const src = String(text ?? '').split('\n');
const stripped = stripCode(text).split('\n');
return src.filter((line, i) => stripped[i] === ''
&& line.trim() !== ''
&& !/^\s*(```|~~~)/.test(line));
}
// There is no CI badge in this org because there is no CI — the published
// substitute, stated in this repo's own README, is one command a stranger can
// run from a clean clone. That SINGLE published stance is what licenses a check
// that fires on a third of the org: VERSION-DRIFT was rejected because twelve
// of the fifteen repos it felled were simply following the other legitimate
// convention, and here there is no other convention. A repo with a runnable
// suite and no documented command is not on a different plan; it is
// undocumented.
//
// The subject is MEASURED, never read off a class. Across all 21 registered
// clones (2026-08-12), 16 have something runnable and 5 do not —
// human-friendly-style, llm-security-commons, playground-design-system,
// portfolio-optimiser-commons and app-creator hold prose, output styles and
// domain packs. Those five span the `plugin`, `shared-asset` and `standalone`
// classes, so any class-level requirement would have failed a correct
// repository somewhere. Nothing to verify is an OK, the RELEASE-NONE shape.
//
// What this check can NEVER do is report that a documented command works — it
// runs nothing. It fells a missing command and nothing else, and the message
// says so, because a green line implying a passing suite is a claim on the
// surface that nobody verified. It also reads only the README and package.json,
// so unlike every check since PIN-DEAD it has no null network input and
// therefore no SKIP at all.
export function checkVerifyCommand({ readme, testScript, testFileCount }) {
const count = Number(testFileCount ?? 0);
if (!testScript && count === 0) {
return [{
level: 'OK',
code: 'VERIFY-NONE',
msg: 'no test script and no tracked test file — nothing here a stranger could run, so no verification command is owed',
}];
}
const found = codeLines(readme).find((l) => VERIFY_COMMAND.test(l));
if (found) {
return [{
level: 'OK',
code: 'VERIFY-COMMAND',
msg: `README shows \`${found.trim()}\` — a stranger has one command to run. This gate does not run it, so this says documented, never passing.`,
}];
}
const have = testScript
? `\`${testScript}\` is defined in package.json`
: `${count} tracked test file(s) exist`;
return [{
level: 'WARN',
code: 'VERIFY-MISSING',
bucket: 'missing',
msg: `${have}, but no README code block shows a command to run them — with no CI badge to fall back on, a stranger has no way to check this repo works. Show the command in a fenced block.`,
}];
}
// Counting badges needs a NARROWER rule than detecting a dishonest one. The
// claim check reads any image, any host, on purpose. Here the opposite error
// matters: counting a screenshot or an architecture diagram as clutter would
@ -1298,7 +1378,7 @@ export function bucketsOf(findings) {
}
export function classifyRepo(
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames, forgeTagsByRepo, forgeTagsSelf, releases },
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames, forgeTagsByRepo, forgeTagsSelf, releases, testScript, testFileCount },
register,
) {
const klass = register.repos?.[name];
@ -1339,6 +1419,7 @@ export function classifyRepo(
...checkTagIntegrity({ tagObjects, name }, register),
...checkReleaseCurrent({ forgeTagsSelf, releases }),
...checkRemoteSync({ tags, forgeTagsSelf }),
...checkVerifyCommand({ readme, testScript, testFileCount }),
...checkDescription(description, register),
];
@ -1505,6 +1586,41 @@ function readPackageVersion(dir) {
return null;
}
// The declared way to run this repo's tests, if there is one. Only
// package.json carries it — a plugin manifest has no scripts, and pyproject's
// runner is not a command a stranger can copy.
function readTestScript(dir) {
const full = join(dir, 'package.json');
if (!existsSync(full)) return null;
try {
const scripts = JSON.parse(readFileSync(full, 'utf8')).scripts ?? {};
return scripts.test ? String(scripts.test) : null;
} catch {
return null;
}
}
// Files that are unambiguously EXECUTABLE tests, not merely files living under
// `tests/`. The looser rule counted golden files, fixtures and transcripts —
// portfolio-optimiser's `tests/golden/demo-transcript.stdout` among them — and
// a finding that says "you have tests a stranger cannot run" is false the
// moment its subject is a fixture. Measured: 16 of 21 clones have a real one.
// One alternative per family rather than one regex, because the shell-script
// family needs a boundary the others do not: a bare `test` substring makes
// `latest-release.sh` a test file.
const EXECUTABLE_TEST_FILE = [
/\.test\.(mjs|cjs|js|ts)$/,
/(^|\/)test_[^/]+\.py$/,
/_test\.py$/,
/(^|\/)([^/]*[-_.])?tests?([-_.][^/]*)?\.sh$/,
/(^|\/)tests?\/[^/]*\.sh$/,
/selftest/,
];
export function countTestFiles(tracked) {
return (tracked ?? []).filter((f) => EXECUTABLE_TEST_FILE.some((re) => re.test(f))).length;
}
export function extractBadgeVersion(readmeText) {
const m = /badge\/version-(\d+\.\d+\.\d+)/.exec(readmeText || '');
return m ? m[1] : null;
@ -1621,6 +1737,8 @@ export function inspectRepo(dir, name, register, description, catalogNames = nul
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
tags: gitTags(dir),
tagObjects: gitTagObjects(dir),
testScript: readTestScript(dir),
testFileCount: countTestFiles(tracked),
catalogNames,
forgeTagsSelf,
releases,