feat(engine): VERIFY-COMMAND — the one command a stranger has instead of CI

The forge has no Actions runners, so this org publishes no CI badge; the
stated substitute is one command runnable from a clean clone. A repo with
something runnable and no such command in its README is a WARN — and the
finding names what the repo already has, so the remedy is one line.

The subject is MEASURED, never read off a class. Five of 21 clones have
nothing runnable at all and answer VERIFY-NONE at OK; they span plugin,
shared-asset AND standalone, so every class-level phrasing of this rule
would fail a correct repository somewhere. Measured: 10 document a
command, 6 do not, 5 have no subject.

Two things bound the rule. It adds no API call, so it has no SKIP at all
— copying the null-input guard from every check since PIN-DEAD would
print a false "not run". And it runs nothing, so its OK says documented,
never passing.

Not built, with distinct reasons recorded as invariants: RELEASE-ASSETS
is rejected permanently for having NO SUBJECT (0 of 21 READMEs mention an
asset download; the 18/18 fire rate is a proxy and must not be quoted as
the reason). TAG-SIGNED is BLOCKED ON AN OPERATOR DECISION, not rejected
— filing it with the rejections would read as settled when it is one
yes/no from acquiring its whole subject.

Also fixes this repo's own surface, which had drifted behind its engine:
four checks had shipped without a row in the README check table, and
Requirements still said "two network calls" after the third was added.

230 tests (from 213).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LwZeAZ8cHmGZofM9dryuT9
This commit is contained in:
Kjell Tore Guttormsen 2026-08-12 23:21:34 +02:00
commit e00ed3340c
7 changed files with 449 additions and 14 deletions

View file

@ -43,6 +43,9 @@ import {
skipsOf,
checkReleaseCurrent,
checkRemoteSync,
checkVerifyCommand,
countTestFiles,
codeLines,
} from './repo-standard-check.mjs';
const REGISTER = {
@ -2159,3 +2162,193 @@ test('unfiltered forge refs cannot manufacture an unpushed-tag ERROR either', ()
const f = checkRemoteSync({ tags: ['v7.7.2'], forgeTagsSelf: ['v7.7.2', 'pre-polyrepo-archive'] });
assert.equal(f[0].level, 'OK');
});
// ----------------------------------------------------------- VERIFY-COMMAND
//
// The org publishes no CI badge because it has no CI — this repo's own stated
// substitute is "one command from a clean clone, said plainly". That single
// published stance is what licenses a check firing on a third of the org:
// unlike VERSION-DRIFT, there is no second legitimate convention a repo could
// be following. A repo with a runnable suite and no documented command is not
// on the other convention; it is undocumented.
//
// Measured 2026-08-12 across all 21 registered clones. Subject present (a
// `scripts.test` entry or a tracked executable test file) in 16 of 21; the
// five without one — human-friendly-style, llm-security-commons,
// playground-design-system, portfolio-optimiser-commons, app-creator — hold
// prose, output styles and domain packs, nothing a stranger could run. Those
// five are exactly the repos a class-level requirement would have failed for
// being correct, which is why the subject is MEASURED and not read off a class.
//
// The check can only ever fell a MISSING command. It never runs one, so it can
// never report that a documented command works — the message must not imply it.
test('a repo with nothing runnable is an OK, not a skip — the check found no subject', () => {
// The VERSION-NONE / RELEASE-NONE shape: the check ran, saw everything, and
// there was nothing here to be wrong.
const f = checkVerifyCommand({ readme: '# x\n', testScript: null, testFileCount: 0 });
assert.equal(f[0].level, 'OK');
assert.equal(f[0].code, 'VERIFY-NONE');
});
test('a suite with no command anywhere in the README is a WARN in the missing bucket', () => {
const f = checkVerifyCommand({
readme: '# okr\n\nA plugin.\n',
testScript: 'node --test tests/',
testFileCount: 24,
});
assert.equal(f[0].level, 'WARN');
assert.equal(f[0].code, 'VERIFY-MISSING');
assert.equal(f[0].bucket, 'missing');
});
test('the WARN names the command the repo already has, so the remedy is one line', () => {
const f = checkVerifyCommand({ readme: '# okr\n', testScript: 'node --test tests/', testFileCount: 24 });
assert.match(f[0].msg, /node --test tests\//);
});
test('a suite with no `scripts.test` still gets a WARN, and it names the file count instead', () => {
const f = checkVerifyCommand({ readme: '# config-audit\n', testScript: null, testFileCount: 106 });
assert.equal(f[0].code, 'VERIFY-MISSING');
assert.match(f[0].msg, /106/);
});
test('a fenced `npm test` is an OK naming the command it found', () => {
const readme = '# x\n\n## Tests\n\n```bash\nnpm test\n```\n';
const f = checkVerifyCommand({ readme, testScript: 'node --test', testFileCount: 3 });
assert.equal(f[0].level, 'OK');
assert.equal(f[0].code, 'VERIFY-COMMAND');
assert.match(f[0].msg, /npm test/);
});
test('the OK never claims the command works — this gate does not run it', () => {
// The one thing this check must not oversell. It can fell a missing command
// and nothing else; a green line here means "documented", never "passing".
const readme = '# x\n\n```bash\nnpm test\n```\n';
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
assert.doesNotMatch(f[0].msg, /passe?s|works|green|verified/i);
assert.match(f[0].msg, /not run|does not run/i);
});
test('a command in prose does not count — a stranger copies out of a code block', () => {
const readme = '# x\n\nRun npm test to check it.\n';
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
assert.equal(f[0].code, 'VERIFY-MISSING');
});
test('an indented code block counts as one — Markdown has two fences', () => {
const readme = '# x\n\nTests:\n\n pytest -q\n';
const f = checkVerifyCommand({ readme, testScript: null, testFileCount: 25 });
assert.equal(f[0].code, 'VERIFY-COMMAND');
});
test('`npm install` is not a verification command', () => {
// The install block is fenced in every repo in the org. Matching it would
// hand a green line to every repo the check exists to find.
const readme = '# x\n\n## Install\n\n```bash\nnpm install\n```\n';
const f = checkVerifyCommand({ readme, testScript: 'npm test', testFileCount: 1 });
assert.equal(f[0].code, 'VERIFY-MISSING');
});
test('the families a clean clone actually runs are all recognised', () => {
// Measured from the corpus rather than imagined: npm/pnpm/yarn scripts,
// `node --test`, a named test file, pytest, make, a shell test script, and
// the `--selftest` flag repo-mailbox ships.
const cases = [
'npm test',
'npm run test:unit',
'pnpm test',
'yarn test',
'node --test',
'node scripts/repo-standard-check.test.mjs',
'pytest',
'python3 -m pytest tests/',
'make check',
'./scripts/coord-selftest.sh',
'bash tests/run-tests.sh',
'coord-selftest --selftest',
];
for (const cmd of cases) {
const f = checkVerifyCommand({
readme: `# x\n\n\`\`\`bash\n${cmd}\n\`\`\`\n`,
testScript: null,
testFileCount: 1,
});
assert.equal(f[0].code, 'VERIFY-COMMAND', `not recognised: ${cmd}`);
}
});
test('a `$` prompt prefix does not hide the command', () => {
const f = checkVerifyCommand({ readme: '# x\n\n```\n$ npm test\n```\n', testScript: null, testFileCount: 1 });
assert.equal(f[0].code, 'VERIFY-COMMAND');
});
test('VERIFY-COMMAND reads no network input, so it has no SKIP at all', () => {
// Every check written since PIN-DEAD opens by guarding a null network input.
// This one reads the README and package.json only — copying that reflex would
// print a false "not run" for a check that ran perfectly well offline.
for (const args of [
{ readme: '', testScript: null, testFileCount: 0 },
{ readme: '', testScript: 'npm test', testFileCount: 0 },
{ readme: '```\nnpm test\n```', testScript: null, testFileCount: 9 },
]) {
assert.equal(checkVerifyCommand(args).every((f) => f.level !== 'SKIP'), true);
}
});
test('codeLines returns what stripCode blanked, and nothing else', () => {
// One fence state machine, not two. The link checks strip code out; this one
// needs exactly the complement, and a second hand-rolled parser is how the
// two copies drift.
const md = '# t\n\nprose npm test\n\n```bash\nnpm test\n```\n\n pytest -q\n';
const lines = codeLines(md);
assert.deepEqual(lines.map((l) => l.trim()), ['npm test', 'pytest -q']);
});
test('classifyRepo carries the verification inputs through to a finding', () => {
// The wiring, not the rule: a check that is never called from the pipeline
// passes its own unit tests forever while measuring nothing.
const r = classifyRepo(
{ name: 'repo-mailbox', files: {}, present: [], description: null, testScript: 'npm test', testFileCount: 4 },
REGISTER,
);
assert.equal(r.findings.some((f) => f.code === 'VERIFY-MISSING'), true);
});
test('countTestFiles counts executable tests, not everything under tests/', () => {
// The looser rule counted golden files and transcripts. A WARN saying "you
// have tests a stranger cannot run" is false the moment its subject is a
// fixture — measured on portfolio-optimiser's tests/golden/*.stdout.
const tracked = [
'scripts/x.test.mjs',
'tests/test_engine.py',
'tests/engine_test.py',
'scripts/coord-selftest.sh',
'tests/run-tests.sh',
'tests/golden/demo-transcript.stdout',
'tests/conftest.py',
'tests/fixtures/sample.md',
'README.md',
];
assert.equal(countTestFiles(tracked), 5);
assert.equal(countTestFiles([]), 0);
assert.equal(countTestFiles(null), 0);
});
test('a `test` substring alone does not make a shell script a test', () => {
// `latest-release.sh` contains `test`. The shell family needs a boundary the
// other families get for free from their suffixes.
assert.equal(countTestFiles(['scripts/latest-release.sh', 'tests/conftest.py']), 0);
});
test('a directory separator is a name boundary too, or a whole repo reads as having no tests', () => {
// claude-design ships five shell tests under `tests/` and read as VERIFY-NONE
// — a false "nothing here to verify", which is a claim about the repo, not a
// missing finding. `^` only anchors the whole string; the basename of
// `tests/test-sc1-dogfood-log.sh` starts after a slash.
assert.equal(countTestFiles([
'tests/test-sc1-dogfood-log.sh',
'tests/test-skill-triggers.sh',
'tests/validate-plugin.sh',
]), 3);
});