feat(engine): PIN-DEAD — the one command a stranger runs
Reported by org-ops (census 08, R1) and re-measured here against the FORGE
rather than taken on their word: 3 install pins in the org, 1 dead.
`llm-ingestion-pipeline-security` pins ITSELF to `@v0.7.0`; that tag does not
exist, newest is v0.6.1. Anyone copying the single install command out of that
README gets a hard pip failure.
ERROR, not WARN: a dead documentation link costs a stranger a 404, a dead pin
costs them the install.
Not a duplicate of two checks it sits near. `LINK-DEAD` asks whether the repo
exists; `VERSION-TAG` reads the MANIFEST and asks whether that version was ever
tagged. All three land on guard today only because the same wrong number got
written in three places — a README pinning a bad ref in a repo with a correct
manifest is invisible to both.
Resolved against the forge, never the clone: a local tag can exist without
having been pushed, which portfolio-optimiser demonstrates directly. That uses
this session's decided acquisition model — `git ls-remote --tags` on the
register-derived https URL, anonymous, no API budget, and only for the repos a
README actually pins (nothing at all for the 19 that pin none).
A pin at a branch or a sha is a `byDesign` skip. `ls-remote --tags` cannot
answer it, and a loose pin is a different finding from a dead one.
The corpus sweep found a defect a unit test had not: offline, guard emitted the
SKIP *and* an OK reading "1 install pin(s) resolve against the forge" — a pass
asserted for a pin nothing had read. SKIP is never a pass. The OK now counts
only what was actually verified ("N of M"), and two tests pin it.
Measured before and after across all 21 clones: purely additive, no existing
finding moved. 19 repos emit PINS-NONE (OK — the check read the whole README
and found no subject), 2 emit real pins. Online, okf's 2 pins resolve and
guard's 1 does not: exactly one new ERROR org-wide, matching org-ops.
196 tests (was 187).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015AkHEqTSr1k3HbeiHu1ggW
This commit is contained in:
parent
425e7e9bdf
commit
067ab0528d
3 changed files with 246 additions and 7 deletions
23
CLAUDE.md
23
CLAUDE.md
|
|
@ -88,6 +88,16 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
registered (measured). The substitute is one command from a clean clone, said
|
||||
plainly. A static badge asserting a run is the anti-pattern this gate flags —
|
||||
and an early draft of this README carried one.
|
||||
- **A dead pin is not a dead link, and it is not a wrong manifest.**
|
||||
`PIN-DEAD` asks whether the ref a README install command pins actually
|
||||
resolves — `LINK-DEAD` asks whether the repo exists, `VERSION-TAG` asks
|
||||
whether the MANIFEST's version was tagged. All three coincided on guard only
|
||||
because one wrong number was written in three places. It is an `ERROR`
|
||||
because a dead documentation link costs a stranger a 404 while a dead pin
|
||||
costs them the install. Resolved against the FORGE, never the clone: a local
|
||||
tag can exist unpushed, which portfolio-optimiser demonstrates. A pin at a
|
||||
branch or a sha is a `byDesign` skip — `ls-remote --tags` cannot answer it,
|
||||
and looseness is a different finding from deadness.
|
||||
- **Three outcomes on references.** "No match" and "match on a known non-repo"
|
||||
must stay distinct findings. Collapsing them hides real loss inside correct
|
||||
text — the exact defect class this gate exists to catch.
|
||||
|
|
@ -197,11 +207,16 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
gate that fails a correct repository is the mechanism that gets gates
|
||||
switched off, and this one fails almost all of them. The two repos that
|
||||
motivated it are both already answered: guard's manifest claims `0.7.0` with
|
||||
no such tag, which is an existing `VERSION-TAG` `ERROR` (and the same root
|
||||
cause as org-ops' `PIN-DEAD` — the README pins `@v0.7.0` too); okf's case
|
||||
no such tag, which is an existing `VERSION-TAG` `ERROR`; okf's case
|
||||
turns on *behaviour-changing* commits past the tag, which no classifier reads
|
||||
off git. Two legitimate conventions coexist here — bump-at-release and
|
||||
bump-first — and nothing in a clone says which one a repo follows. Recorded
|
||||
bump-first — and nothing in a clone says which one a repo follows.
|
||||
This says nothing about org-ops' `PIN-DEAD`, which is a different check on a
|
||||
different subject: `VERSION-TAG` reads the MANIFEST and asks whether that
|
||||
version was ever tagged, `PIN-DEAD` reads a README INSTALL COMMAND and asks
|
||||
whether the ref it pins resolves. They coincide on guard only because the
|
||||
same wrong number was written in both places; a README pinning a bad ref in a
|
||||
repo with a correct manifest is invisible to `VERSION-TAG`. Recorded
|
||||
rather than deferred: a decision that is wrong is worse than no record, and
|
||||
the next session should not re-derive this measurement.
|
||||
- **No hook until the rule is precise.** A blocking gate that fails a correct
|
||||
|
|
@ -210,7 +225,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
|
|||
## Commands
|
||||
|
||||
```bash
|
||||
npm test # 182 tests
|
||||
npm test # 196 tests
|
||||
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
|
||||
node scripts/repo-standard-check.mjs --offline # no network call
|
||||
node scripts/repo-standard-check.mjs --json # machine output
|
||||
|
|
|
|||
|
|
@ -432,6 +432,99 @@ export function checkInstallTruth({ name, klass, catalogNames }) {
|
|||
return [{ level: 'OK', code: 'INSTALL-TRUTH', msg: 'the install command resolves against the catalog' }];
|
||||
}
|
||||
|
||||
// A pin is the one command a stranger actually runs. Reported by org-ops
|
||||
// (census 08) and re-measured here against the FORGE: 3 pins in the org, 1
|
||||
// dead — `llm-ingestion-pipeline-security` pins ITSELF to `@v0.7.0`, and that
|
||||
// tag does not exist (newest is v0.6.1). ERROR, not WARN, because a dead
|
||||
// documentation link costs a stranger a 404 while a dead pin costs them the
|
||||
// install.
|
||||
//
|
||||
// Deliberately NOT reusing `LINK-DEAD`'s check, only the idea of enumerating:
|
||||
// LINK-DEAD asks "does the repo exist", this asks "does the reference exist".
|
||||
// It is also not `VERSION-TAG`, which reads the MANIFEST — the two coincide on
|
||||
// guard today only because the same wrong number was written in both places.
|
||||
//
|
||||
// Resolved against the forge, never the clone: a local tag can exist without
|
||||
// having been pushed, which is exactly what portfolio-optimiser demonstrates.
|
||||
const escapeRe = (s) => String(s).replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
|
||||
export function extractInstallPins(readme, register) {
|
||||
const forge = String(register?.forge ?? '').replace(/\/+$/, '');
|
||||
const org = register?.org;
|
||||
if (!forge || !org) return [];
|
||||
const re = new RegExp(`(?:git\\+)?${escapeRe(forge)}/${escapeRe(org)}/([A-Za-z0-9._-]+)\\.git@([^\\s"'\`)\\]#]+)`, 'g');
|
||||
const seen = new Set();
|
||||
const pins = [];
|
||||
for (const m of String(readme ?? '').matchAll(re)) {
|
||||
const key = `${m[1]}@${m[2]}`;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
pins.push({ repo: m[1], ref: m[2] });
|
||||
}
|
||||
return pins;
|
||||
}
|
||||
|
||||
// A dotted numeric component is what makes a ref answerable by `ls-remote
|
||||
// --tags`. `main` and a bare sha are neither dead nor alive to this check.
|
||||
const TAG_SHAPED = /^v?\d+\.\d+/;
|
||||
|
||||
export function checkInstallPins({ readme, forgeTagsByRepo }, register) {
|
||||
const pins = extractInstallPins(readme, register);
|
||||
if (pins.length === 0) {
|
||||
// The VERSION-NONE shape: the check ran, read the whole README, and found
|
||||
// no subject. Nothing here can be wrong, which is a verdict.
|
||||
return [{ level: 'OK', code: 'PINS-NONE', msg: 'README pins no install reference — no ref exists here that could be dead' }];
|
||||
}
|
||||
|
||||
const findings = [];
|
||||
let resolved = 0;
|
||||
for (const { repo, ref } of pins) {
|
||||
if (!TAG_SHAPED.test(ref)) {
|
||||
// A branch or a sha is a different weakness — an unpinned install — and
|
||||
// this check can never turn it into a verdict, so nobody has an action.
|
||||
findings.push({
|
||||
level: 'SKIP',
|
||||
skip: 'byDesign',
|
||||
code: 'PIN-NOT-A-TAG',
|
||||
msg: `install pin \`${repo}@${ref}\` is not a version tag — \`ls-remote --tags\` cannot resolve it, and a branch pin is a looseness this check does not judge`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
const tags = forgeTagsByRepo?.[repo];
|
||||
if (!tags) {
|
||||
findings.push({
|
||||
level: 'SKIP',
|
||||
skip: 'notRun',
|
||||
code: 'PIN-UNAVAILABLE',
|
||||
msg: `could not read tags for \`${repo}\` from the forge — the pin \`@${ref}\` was not verified (offline, or the ref listing failed)`,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
if (!tags.includes(ref)) {
|
||||
const newest = [...tags].filter((t) => TAG_SHAPED.test(t)).sort(compareTags).slice(-1)[0];
|
||||
findings.push({
|
||||
level: 'ERROR',
|
||||
code: 'PIN-DEAD',
|
||||
bucket: 'broken',
|
||||
msg: `README install pin \`${repo}@${ref}\` does not exist on the forge — the one command a stranger runs fails outright${newest ? ` (newest tag is \`${newest}\`)` : ''}`,
|
||||
});
|
||||
} else {
|
||||
resolved += 1;
|
||||
}
|
||||
}
|
||||
// Counts what was actually verified, never what was merely present. The
|
||||
// first version said "N install pin(s) resolve" whenever no ERROR fired,
|
||||
// which meant an offline run asserted a pass for a pin nothing had read.
|
||||
if (resolved > 0) {
|
||||
findings.push({
|
||||
level: 'OK',
|
||||
code: 'PINS',
|
||||
msg: `${resolved} of ${pins.length} install pin(s) resolve against the forge`,
|
||||
});
|
||||
}
|
||||
return findings;
|
||||
}
|
||||
|
||||
// Requirements come from two axes. The CLASS is structural — it can be read off
|
||||
// the catalog and the remotes. A TRAIT is about what the code does, which no
|
||||
// remote can tell you: `security` attaches the obligations a tool acquires by
|
||||
|
|
@ -1098,7 +1191,7 @@ export function bucketsOf(findings) {
|
|||
}
|
||||
|
||||
export function classifyRepo(
|
||||
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames },
|
||||
{ name, files, present, description, pluginVersion, readmeBadge, changelogTop, tags, tagObjects, catalogNames, forgeTagsByRepo },
|
||||
register,
|
||||
) {
|
||||
const klass = register.repos?.[name];
|
||||
|
|
@ -1126,6 +1219,7 @@ export function classifyRepo(
|
|||
...checkFirstScreen({ readme, name, description, klass }, register),
|
||||
...checkInstallBlock({ readme, name, klass }, register),
|
||||
...checkInstallTruth({ name, klass, catalogNames }),
|
||||
...checkInstallPins({ readme, forgeTagsByRepo }, register),
|
||||
...checkHeadings({ readme, klass, traits }, register),
|
||||
...checkRequiredFiles({ present, klass, traits }, register),
|
||||
...checkLinks({ files }, register),
|
||||
|
|
@ -1316,6 +1410,30 @@ function gitTags(dir) {
|
|||
// `%(objecttype)` is `tag` for an annotated tag and `commit` for a lightweight
|
||||
// one — the distinction read straight off the local object database, with no
|
||||
// network call, so tag integrity costs nothing against the two-call budget.
|
||||
// Refs from the FORGE, over the git protocol — anonymous, and measured not to
|
||||
// share the API's rate-limit bucket, so it costs nothing against the two-call
|
||||
// budget. The URL is derived from the register, never from `origin`: at least
|
||||
// one repo's origin is `ssh://git@…`, which would need the operator's key and
|
||||
// so would work here and fail for every other reader.
|
||||
function forgeTags(register, repo) {
|
||||
const forge = String(register?.forge ?? '').replace(/\/+$/, '');
|
||||
if (!forge || !register?.org) return null;
|
||||
try {
|
||||
const out = execFileSync('git', ['ls-remote', '--tags', `${forge}/${register.org}/${repo}.git`], {
|
||||
encoding: 'utf8',
|
||||
env: { ...process.env, GIT_TERMINAL_PROMPT: '0' },
|
||||
});
|
||||
return out.split('\n')
|
||||
.map((l) => l.split('\t')[1])
|
||||
.filter((r) => r && !r.endsWith('^{}'))
|
||||
.map((r) => r.replace(/^refs\/tags\//, ''));
|
||||
} catch {
|
||||
// Unreachable forge leaves this null, which reads as SKIP/notRun — never
|
||||
// as a pass.
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function gitTagObjects(dir) {
|
||||
try {
|
||||
return execFileSync('git', ['-C', dir, 'for-each-ref', '--format=%(objecttype) %(refname:short)', 'refs/tags/v*'], { encoding: 'utf8' })
|
||||
|
|
@ -1329,7 +1447,7 @@ function gitTagObjects(dir) {
|
|||
}
|
||||
}
|
||||
|
||||
export function inspectRepo(dir, name, register, description, catalogNames = null) {
|
||||
export function inspectRepo(dir, name, register, description, catalogNames = null, offline = false) {
|
||||
const tracked = gitFiles(dir);
|
||||
const present = (tracked ?? []).filter((f) => existsSync(join(dir, f)));
|
||||
|
||||
|
|
@ -1347,11 +1465,23 @@ export function inspectRepo(dir, name, register, description, catalogNames = nul
|
|||
let changelog = null;
|
||||
try { changelog = readFileSync(join(dir, 'CHANGELOG.md'), 'utf8'); } catch { /* absent */ }
|
||||
|
||||
// Only the repos this README actually pins are fetched — one ref listing
|
||||
// each, and nothing at all for the common case of no pins.
|
||||
let forgeTagsByRepo = null;
|
||||
if (!offline) {
|
||||
forgeTagsByRepo = {};
|
||||
for (const repo of new Set(extractInstallPins(readme, register).map((p) => p.repo))) {
|
||||
const t = forgeTags(register, repo);
|
||||
if (t) forgeTagsByRepo[repo] = t;
|
||||
}
|
||||
}
|
||||
|
||||
return classifyRepo({
|
||||
name,
|
||||
files,
|
||||
present,
|
||||
description,
|
||||
forgeTagsByRepo,
|
||||
pluginVersion: readPackageVersion(dir),
|
||||
readmeBadge: extractBadgeVersion(readme),
|
||||
changelogTop: changelog === null ? null : extractChangelogTop(changelog),
|
||||
|
|
@ -1478,7 +1608,7 @@ async function main(argv) {
|
|||
}
|
||||
}
|
||||
|
||||
const result = inspectRepo(dir, name, register, description, catalogNames);
|
||||
const result = inspectRepo(dir, name, register, description, catalogNames, argv.includes('--offline'));
|
||||
const engineVersion = readEngineVersion();
|
||||
const engineCommit = readEngineCommit();
|
||||
|
||||
|
|
|
|||
|
|
@ -22,6 +22,8 @@ import {
|
|||
checkHeadings,
|
||||
checkBadges,
|
||||
checkTagIntegrity,
|
||||
checkInstallPins,
|
||||
extractInstallPins,
|
||||
checkReadmeLanguage,
|
||||
checkBoilerplate,
|
||||
checkLicenseClaim,
|
||||
|
|
@ -927,6 +929,98 @@ test('newest is the highest version, not the last element of an unsorted list',
|
|||
assert.equal(f.some((x) => x.code === 'TAG-ANNOTATED-HISTORY'), true);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------- dead pins
|
||||
//
|
||||
// Reported by org-ops (census 08) and re-measured here against the FORGE, not
|
||||
// the clones: 3 pins in the org, 1 dead —
|
||||
// `llm-ingestion-pipeline-security` pins ITSELF to `@v0.7.0`, which does not
|
||||
// exist (newest is v0.6.1). ERROR, not WARN: a dead documentation link costs a
|
||||
// stranger a 404, a dead pin costs them the one install command that was
|
||||
// supposed to work.
|
||||
const PIN_REG = { org: 'open', forge: 'https://git.fromaitochitta.com' };
|
||||
|
||||
test('an install pin is extracted with its repo and ref', () => {
|
||||
const readme = '```bash\npip install "guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.7.0"\n```';
|
||||
const pins = extractInstallPins(readme, PIN_REG);
|
||||
assert.equal(pins.length, 1);
|
||||
assert.equal(pins[0].repo, 'llm-ingestion-pipeline-security');
|
||||
assert.equal(pins[0].ref, 'v0.7.0');
|
||||
});
|
||||
|
||||
test('a pin whose ref is not a tag on the forge is an ERROR', () => {
|
||||
const readme = 'pip install "g @ git+https://git.fromaitochitta.com/open/guard-repo.git@v0.7.0"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: { 'guard-repo': ['v0.6.0', 'v0.6.1'] } }, PIN_REG);
|
||||
const hit = f.find((x) => x.code === 'PIN-DEAD');
|
||||
assert.equal(hit.level, 'ERROR');
|
||||
assert.equal(hit.bucket, 'broken');
|
||||
assert.match(hit.msg, /v0\.7\.0/);
|
||||
assert.match(hit.msg, /v0\.6\.1/); // names what DOES exist, so the remedy is obvious
|
||||
});
|
||||
|
||||
test('a pin whose ref resolves is OK', () => {
|
||||
const readme = 'pip install "g @ git+https://git.fromaitochitta.com/open/guard-repo.git@v0.6.1"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: { 'guard-repo': ['v0.6.0', 'v0.6.1'] } }, PIN_REG);
|
||||
assert.equal(f.some((x) => x.level === 'ERROR'), false);
|
||||
assert.equal(f.some((x) => x.code === 'PINS'), true);
|
||||
});
|
||||
|
||||
// `ls-remote --tags` cannot answer a branch or a sha, and a branch pin is a
|
||||
// different weakness (an unpinned install), not a dead one. This check can
|
||||
// never produce a verdict on it, so the skip is byDesign — nobody has an
|
||||
// action that would turn it into one.
|
||||
test('a pin at a branch or a sha is a byDesign skip, never a dead pin', () => {
|
||||
const readme = 'pip install "g @ git+https://git.fromaitochitta.com/open/guard-repo.git@main"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: { 'guard-repo': ['v0.6.1'] } }, PIN_REG);
|
||||
const hit = f.find((x) => x.code === 'PIN-NOT-A-TAG');
|
||||
assert.equal(hit.level, 'SKIP');
|
||||
assert.equal(hit.skip, 'byDesign');
|
||||
});
|
||||
|
||||
test('offline leaves the pin un-judged, and says so as notRun', () => {
|
||||
const readme = 'pip install "g @ git+https://git.fromaitochitta.com/open/guard-repo.git@v0.7.0"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: null }, PIN_REG);
|
||||
const hit = f.find((x) => x.code === 'PIN-UNAVAILABLE');
|
||||
assert.equal(hit.level, 'SKIP');
|
||||
assert.equal(hit.skip, 'notRun');
|
||||
});
|
||||
|
||||
// Caught by the corpus sweep, not by a unit test: offline, guard emitted the
|
||||
// SKIP *and* an OK reading "1 install pin(s) resolve against the forge" — a
|
||||
// pass asserted for a pin nothing had checked. SKIP is never a pass.
|
||||
test('an unverified pin never produces an OK claiming it resolves', () => {
|
||||
const readme = 'pip install "g @ git+https://git.fromaitochitta.com/open/guard-repo.git@v0.7.0"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: null }, PIN_REG);
|
||||
assert.equal(f.some((x) => x.code === 'PINS'), false);
|
||||
assert.equal(f.some((x) => x.level === 'OK'), false);
|
||||
});
|
||||
|
||||
test('the OK counts only the pins actually verified', () => {
|
||||
const readme = [
|
||||
'pip install "a @ git+https://git.fromaitochitta.com/open/alpha.git@v1.0.0"',
|
||||
'pip install "b @ git+https://git.fromaitochitta.com/open/beta.git@v2.0.0"',
|
||||
].join('\n');
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: { alpha: ['v1.0.0'] } }, PIN_REG);
|
||||
const ok = f.find((x) => x.code === 'PINS');
|
||||
assert.match(ok.msg, /1 of 2/);
|
||||
assert.equal(f.some((x) => x.code === 'PIN-UNAVAILABLE'), true);
|
||||
});
|
||||
|
||||
test('a pin at somebody else’s forge is not ours to judge', () => {
|
||||
const readme = 'pip install "x @ git+https://github.com/other/thing.git@v9.9.9"';
|
||||
const f = checkInstallPins({ readme, forgeTagsByRepo: {} }, PIN_REG);
|
||||
assert.equal(f.some((x) => x.code === 'PIN-DEAD'), false);
|
||||
assert.equal(extractInstallPins(readme, PIN_REG).length, 0);
|
||||
});
|
||||
|
||||
// The VERSION-NONE shape: the check ran, read the whole README, and found no
|
||||
// pin. Nothing here can be wrong — that is a verdict, not an absent one.
|
||||
test('a README with no pins has nothing to judge — OK, not SKIP', () => {
|
||||
const f = checkInstallPins({ readme: '# hello\n\nno install pins here', forgeTagsByRepo: {} }, PIN_REG);
|
||||
assert.equal(f.length, 1);
|
||||
assert.equal(f[0].level, 'OK');
|
||||
assert.equal(f[0].code, 'PINS-NONE');
|
||||
});
|
||||
|
||||
// ------------------------------------------------------------ badge honesty
|
||||
|
||||
test('a static badge asserting test or build status is a finding', () => {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue