feat(gate): two presentation checks the evidence actually supports
Adversarial deep research (25 sources, 124 claims extracted, 25 verified: 11 confirmed / 14 refuted) plus direct measurement of all 18 cloned open/ repos. The useful half of the result is what it REFUSED to support, so the research is recorded in docs/presentation-research-2026-08-03.md rather than being spent and forgotten. - BADGE-COUNT (WARN) — past five badges. Trockman et al., ICSE 2018 (n=294,941 npm packages) measured a non-linear relationship with popularity inflecting at five, motivated by surveyed maintainers calling over-badged READMEs cluttered and "trying too hard". WARN and never ERROR: the coefficient sits in an appendix with no CI or p-value. Counting deliberately uses a NARROWER rule than the existing claim check, so a screenshot or an architecture diagram is never counted as clutter. Fires on 8 of 18. - README-LANGUAGE (WARN) — prose not in the language this repo's readers were declared to speak, via a new `locales` axis in the register. Class is structural, a trait is what the code DOES, a locale is who it is FOR — the standard's own "who the reader is decides what is required". English is the default; ms-ai-architect and okr are declared nb, named by the operator as Norway-only in audience. Stopword-frequency comparison over prose with code stripped: a Norwegian flag name in a shell example cannot decide the document. Fires on exactly those two, silent on all sixteen English repos. One design correction found mid-implementation: the first version returned SKIP when a README had too little prose to judge, which broke a passing fixture and would have stopped any terse repo from ever reaching OK. SKIP is for a check that could not RUN; this one ran, saw everything and found no prose to be in the wrong language — the same shape as "no licence claim to back". Insufficient prose is now OK, and evenly bilingual prose is the SKIP, because there the question is live and unanswered. Deliberately NOT built, because the evidence does not reach: any rule about images, diagrams or terminal recordings (every such claim refuted 0-3); a README length bound (no evidence-based target exists); a section count (would fire on 9 of 18 — textbook "suspect the CHECK"); Mermaid source length and #gh-dark-mode-only (zero occurrences, and the instance limit is not readable via the API, so any threshold would be a guess). Verified live against the operator's own forge (15.0.6+gitea-1.22.0): Mermaid DOES render in README.md — two div.mermaid-block iframes carrying real SVG — while #gh-dark-mode-only landed only in Gitea 1.26.0 and is unavailable here. 103 tests green, up from 92. No version bump: the catalog ref still trails at v0.1.1 against 0.1.3, and starting a second release chain over that is the operator's call. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TCAGKZT8h9F46ygzSkhEee
This commit is contained in:
parent
5eba10acdd
commit
dc386d4471
7 changed files with 474 additions and 2 deletions
|
|
@ -434,14 +434,30 @@ export function checkVersionConsistency({ pluginVersion, readmeBadge, changelogT
|
|||
// word loses no real detection.
|
||||
const CLAIM_BADGE = /(tests?|build|ci|coverage|passing)/i;
|
||||
|
||||
// Counting badges needs a NARROWER rule than detecting a dishonest one. The
|
||||
// claim check reads any image, any host, on purpose. Here the opposite error
|
||||
// matters: counting a screenshot or an architecture diagram as clutter would
|
||||
// punish exactly the visual work this standard wants more of.
|
||||
const BADGE_URL = /shields\.io|badgen\.net|\/badges?[/.]/i;
|
||||
|
||||
// Trockman et al., ICSE 2018 (doi 10.1145/3180155.3180209, n=294,941 npm
|
||||
// packages): badge count relates to popularity non-linearly with a predicted
|
||||
// inflection at five, and surveyed maintainers called over-badged READMEs
|
||||
// cluttered and "trying too hard". WARN, never ERROR — the coefficient sits in
|
||||
// an appendix with no CI or p-value, so it carries "more is not better" and
|
||||
// cannot carry a hard limit.
|
||||
const BADGE_INFLECTION = 5;
|
||||
|
||||
export function checkBadges({ readme }) {
|
||||
const findings = [];
|
||||
let badgeCount = 0;
|
||||
for (const line of String(readme ?? '').split('\n')) {
|
||||
// Any image, any host. Restricting this to img.shields.io would have missed
|
||||
// a self-hosted SVG asserting exactly the same unverified thing.
|
||||
for (const m of line.matchAll(/(\[)?!\[([^\]]*)\]\(([^)\s]+)\)(\])?/g)) {
|
||||
const linked = m[1] === '[' && m[4] === ']';
|
||||
const label = `${m[2]} ${m[3]}`;
|
||||
if (BADGE_URL.test(m[3])) badgeCount++;
|
||||
if (!linked && CLAIM_BADGE.test(label)) {
|
||||
findings.push({
|
||||
level: 'WARN',
|
||||
|
|
@ -452,10 +468,92 @@ export function checkBadges({ readme }) {
|
|||
}
|
||||
}
|
||||
}
|
||||
if (badgeCount > BADGE_INFLECTION) {
|
||||
findings.push({
|
||||
level: 'WARN',
|
||||
code: 'BADGE-COUNT',
|
||||
bucket: 'weakening',
|
||||
msg: `${badgeCount} badges — past the measured inflection of ${BADGE_INFLECTION}, where a badge row starts reading as clutter rather than as evidence (Trockman et al., ICSE 2018). Keep the ones a reader acts on.`,
|
||||
});
|
||||
}
|
||||
if (findings.length === 0) findings.push({ level: 'OK', code: 'BADGES', msg: 'no static badge asserts an unverified run' });
|
||||
return findings;
|
||||
}
|
||||
|
||||
// Which language a README is written in is not a property of the code, so no
|
||||
// remote can report it — it is a property of the READER, and the operator owns
|
||||
// it. English is the default; a repo aimed only at a Norwegian readership is
|
||||
// declared `nb` in the register and is then wrong in English, not right.
|
||||
//
|
||||
// Detection is a stopword-frequency comparison rather than a dependency: both
|
||||
// word sets below are chosen to have NO member that is also a common word in
|
||||
// the other language, which is why `at` and `for` (Norwegian and English both)
|
||||
// are deliberately absent from each.
|
||||
const STOPWORDS = {
|
||||
nb: ['og', 'ikke', 'som', 'det', 'den', 'er', 'på', 'til', 'av', 'med', 'om',
|
||||
'har', 'kan', 'skal', 'blir', 'være', 'etter', 'når', 'også', 'hvis',
|
||||
'eller', 'men', 'fra', 'ved', 'mot', 'uten', 'hver', 'alle', 'andre',
|
||||
'seg', 'dette', 'disse', 'mellom', 'gjennom', 'siden', 'fordi', 'derfor'],
|
||||
en: ['the', 'and', 'of', 'to', 'in', 'is', 'that', 'with', 'this', 'are',
|
||||
'be', 'from', 'by', 'as', 'an', 'or', 'not', 'you', 'your', 'we', 'our',
|
||||
'it', 'on', 'which', 'when', 'what', 'how', 'its', 'they', 'their', 'has',
|
||||
'can', 'will', 'should', 'each', 'between', 'because', 'therefore'],
|
||||
};
|
||||
|
||||
// Below this there is not enough running prose for a frequency count to mean
|
||||
// anything, and below the ratio the document is genuinely mixed. Both say so
|
||||
// rather than guessing — a wrong verdict on language is worse than no verdict.
|
||||
const LANG_MIN_HITS = 10;
|
||||
const LANG_MIN_RATIO = 1.5;
|
||||
|
||||
function countStopwords(text, words) {
|
||||
const lower = text.toLowerCase();
|
||||
let n = 0;
|
||||
for (const w of words) {
|
||||
const m = lower.match(new RegExp(`(^|[^\\p{L}])${w}([^\\p{L}]|$)`, 'gu'));
|
||||
if (m) n += m.length;
|
||||
}
|
||||
return n;
|
||||
}
|
||||
|
||||
export function checkReadmeLanguage({ readme, name }, register) {
|
||||
const declared = register?.locales?.[name] ?? 'en';
|
||||
const other = declared === 'nb' ? 'en' : 'nb';
|
||||
// Same discipline as the link and boilerplate checks: a Norwegian flag name
|
||||
// in a shell example must not decide what language the DOCUMENT is in.
|
||||
const prose = stripCode(String(readme ?? ''));
|
||||
const hits = { nb: countStopwords(prose, STOPWORDS.nb), en: countStopwords(prose, STOPWORDS.en) };
|
||||
|
||||
// Not a SKIP. SKIP is for a check that could not RUN — the catalog was
|
||||
// unreachable, the file unreadable. This one ran, saw everything, and found
|
||||
// no prose to be in the wrong language, the same shape as "no licence claim
|
||||
// to back". A thin README is a real problem, and it is checkFirstScreen's;
|
||||
// routing it here would stop any terse repo from ever reaching OK.
|
||||
if (hits[declared] + hits[other] < LANG_MIN_HITS) {
|
||||
return [{
|
||||
level: 'OK',
|
||||
code: 'LANGUAGE',
|
||||
msg: `no running prose to judge (${hits[declared] + hits[other]} marker words) — nothing claims a language`,
|
||||
}];
|
||||
}
|
||||
if (hits[other] >= hits[declared] * LANG_MIN_RATIO) {
|
||||
return [{
|
||||
level: 'WARN',
|
||||
code: 'README-LANGUAGE',
|
||||
bucket: 'weakening',
|
||||
msg: `README reads as \`${other}\` but this repo is declared \`${declared}\` (${hits[other]} vs ${hits[declared]} marker words). Who the reader is decides the language — fix the prose, or fix \`locales\` in the register.`,
|
||||
}];
|
||||
}
|
||||
if (hits[declared] < hits[other] * LANG_MIN_RATIO) {
|
||||
return [{
|
||||
level: 'SKIP',
|
||||
code: 'README-LANGUAGE-UNDECIDABLE',
|
||||
msg: `README mixes languages too evenly to call (${hits.nb} nb vs ${hits.en} en) — declared \`${declared}\`, unverified`,
|
||||
}];
|
||||
}
|
||||
return [{ level: 'OK', code: 'LANGUAGE', msg: `README reads as \`${declared}\`, as declared` }];
|
||||
}
|
||||
|
||||
// Template text that was never filled in. A visible unfinished template costs
|
||||
// more trust than the missing document would have.
|
||||
const FIXME_RE = /FIXME/;
|
||||
|
|
@ -689,6 +787,7 @@ export function classifyRepo(
|
|||
...checkInternalLinks({ files, present }),
|
||||
...checkLicenseClaim({ readme, present }),
|
||||
...checkBadges({ readme }),
|
||||
...checkReadmeLanguage({ readme, name }, register),
|
||||
...checkBoilerplate({ files }),
|
||||
...checkVersionConsistency({ pluginVersion, readmeBadge, changelogTop, tags }),
|
||||
...checkDescription(description, register),
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@ import {
|
|||
checkVersionConsistency,
|
||||
checkHeadings,
|
||||
checkBadges,
|
||||
checkReadmeLanguage,
|
||||
checkBoilerplate,
|
||||
checkLicenseClaim,
|
||||
checkInternalLinks,
|
||||
|
|
@ -563,6 +564,122 @@ test('a build- or CI-status badge is still caught — only bare "status" was too
|
|||
assert.equal(ci.some((x) => x.code === 'BADGE-STATIC-CLAIM'), true);
|
||||
});
|
||||
|
||||
// ----------------------------------------------------------- badge crowding
|
||||
|
||||
// Trockman et al., ICSE 2018 (n=294,941 npm packages) measured a non-linear
|
||||
// relationship between badge count and popularity with a predicted inflection
|
||||
// at five, motivated by survey respondents calling over-badged READMEs
|
||||
// cluttered and "trying too hard". The coefficient sits in an appendix with no
|
||||
// CI or p-value, so this is a WARN and the threshold is the measured inflection
|
||||
// — not a rounder number that would read as invented.
|
||||
const badges = (n, wrap = (s) => s) =>
|
||||
Array.from({ length: n }, (_, i) => wrap(``)).join('\n');
|
||||
|
||||
test('more than five badges is a finding — the measured inflection point', () => {
|
||||
const f = checkBadges({ readme: badges(6) });
|
||||
const hit = f.find((x) => x.code === 'BADGE-COUNT');
|
||||
assert.ok(hit, 'six badges should produce BADGE-COUNT');
|
||||
assert.equal(hit.level, 'WARN');
|
||||
assert.equal(hit.bucket, 'weakening');
|
||||
});
|
||||
|
||||
test('exactly five badges is not a finding — the source cannot carry a harder rule', () => {
|
||||
assert.equal(checkBadges({ readme: badges(5) }).some((x) => x.code === 'BADGE-COUNT'), false);
|
||||
});
|
||||
|
||||
test('a linked badge still counts toward the total — linking answers honesty, not clutter', () => {
|
||||
const readme = badges(6, (s) => `[${s}](https://forge.example/run)`);
|
||||
assert.equal(checkBadges({ readme }).some((x) => x.code === 'BADGE-COUNT'), true);
|
||||
});
|
||||
|
||||
test('content images are not badges, however many there are', () => {
|
||||
const readme = Array.from({ length: 9 }, (_, i) => ``).join('\n');
|
||||
assert.equal(checkBadges({ readme }).some((x) => x.code === 'BADGE-COUNT'), false);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------- README language
|
||||
|
||||
// The operator owns this axis, exactly as they own `traits`. English is the
|
||||
// default; a repo aimed ONLY at a Norwegian readership is declared `nb` and is
|
||||
// then WRONG in English, not right. Measured 2026-08-03: `ms-ai-architect` and
|
||||
// `okr` are the two such repos, and both currently carry English prose.
|
||||
const LOCALE_REGISTER = { ...REGISTER, locales: { okr: 'nb' } };
|
||||
|
||||
const NB_PROSE = [
|
||||
'# okr',
|
||||
'',
|
||||
'Dette er en plugin som ikke gjør noe annet enn å måle mål og resultater.',
|
||||
'Den kan kjøres fra Claude Code, og den skal være tilgjengelig når du',
|
||||
'trenger den. Hvis du vil ha mer, se dokumentasjonen. Alle kommandoer',
|
||||
'blir kjørt fra en aktiv sesjon, og ingenting av dette krever en server.',
|
||||
].join('\n');
|
||||
|
||||
const EN_PROSE = [
|
||||
'# okr',
|
||||
'',
|
||||
'This is a plugin that does nothing other than measure objectives and',
|
||||
'results. It can be run from Claude Code, and it is always available when',
|
||||
'you need it. If you want more, see the documentation. All of the commands',
|
||||
'are run from an active session, and none of this requires a server.',
|
||||
].join('\n');
|
||||
|
||||
test('a repo declared Norwegian that ships English prose is a finding', () => {
|
||||
const f = checkReadmeLanguage({ readme: EN_PROSE, name: 'okr' }, LOCALE_REGISTER);
|
||||
const hit = f.find((x) => x.code === 'README-LANGUAGE');
|
||||
assert.ok(hit, 'declared nb + English prose should produce README-LANGUAGE');
|
||||
assert.equal(hit.level, 'WARN');
|
||||
assert.equal(hit.bucket, 'weakening');
|
||||
});
|
||||
|
||||
test('a repo declared Norwegian that ships Norwegian prose passes', () => {
|
||||
const f = checkReadmeLanguage({ readme: NB_PROSE, name: 'okr' }, LOCALE_REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
|
||||
});
|
||||
|
||||
test('an undeclared repo defaults to English, so Norwegian prose is the finding', () => {
|
||||
const f = checkReadmeLanguage({ readme: NB_PROSE, name: 'repo-standard' }, LOCALE_REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), true);
|
||||
});
|
||||
|
||||
test('an undeclared repo shipping English prose passes', () => {
|
||||
const f = checkReadmeLanguage({ readme: EN_PROSE, name: 'repo-standard' }, LOCALE_REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
|
||||
});
|
||||
|
||||
// Same discipline as the link and boilerplate checks: a Norwegian identifier in
|
||||
// a shell example must not decide what language the DOCUMENT is written in.
|
||||
test('code blocks do not decide the language', () => {
|
||||
const readme = [
|
||||
EN_PROSE,
|
||||
'',
|
||||
'```bash',
|
||||
'kjør --og --ikke --som --det --den --er --på --til --av --med --om',
|
||||
'kjør --har --kan --skal --blir --etter --når --også --hvis --eller',
|
||||
'```',
|
||||
].join('\n');
|
||||
const f = checkReadmeLanguage({ readme, name: 'repo-standard' }, LOCALE_REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
|
||||
});
|
||||
|
||||
// A README with no running prose has no language to be wrong about — the check
|
||||
// ran and found nothing, which is not the same as a check that could not run.
|
||||
// Routing it to SKIP would mean no terse repo could ever classify OK.
|
||||
test('no running prose is an OK, not a SKIP — nothing claims a language', () => {
|
||||
const f = checkReadmeLanguage({ readme: '# thing\n\nA tool.\n', name: 'repo-standard' }, LOCALE_REGISTER);
|
||||
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
|
||||
assert.equal(f[0].level, 'OK');
|
||||
});
|
||||
|
||||
// A document with real prose that the gate genuinely cannot call IS a SKIP —
|
||||
// here the question is live and unanswered, unlike the empty case above.
|
||||
test('prose that mixes languages too evenly is a SKIP, never a pass', () => {
|
||||
const readme = [NB_PROSE, EN_PROSE].join('\n\n');
|
||||
const f = checkReadmeLanguage({ readme, name: 'repo-standard' }, LOCALE_REGISTER);
|
||||
const hit = f.find((x) => x.code === 'README-LANGUAGE-UNDECIDABLE');
|
||||
assert.ok(hit, 'an evenly bilingual README cannot be called');
|
||||
assert.equal(hit.level, 'SKIP');
|
||||
});
|
||||
|
||||
// -------------------------------------------------------------- boilerplate
|
||||
|
||||
test('unfinished template text is a finding', () => {
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue