open/repo-standard
1
0
Fork 0
Code Issues Releases 2 Activity

feat(gate): two presentation checks the evidence actually supports

Browse source 
Adversarial deep research (25 sources, 124 claims extracted, 25 verified:
11 confirmed / 14 refuted) plus direct measurement of all 18 cloned open/
repos. The useful half of the result is what it REFUSED to support, so the
research is recorded in docs/presentation-research-2026-08-03.md rather than
being spent and forgotten.

- BADGE-COUNT (WARN) — past five badges. Trockman et al., ICSE 2018
  (n=294,941 npm packages) measured a non-linear relationship with popularity
  inflecting at five, motivated by surveyed maintainers calling over-badged
  READMEs cluttered and "trying too hard". WARN and never ERROR: the
  coefficient sits in an appendix with no CI or p-value. Counting deliberately
  uses a NARROWER rule than the existing claim check, so a screenshot or an
  architecture diagram is never counted as clutter. Fires on 8 of 18.
- README-LANGUAGE (WARN) — prose not in the language this repo's readers were
  declared to speak, via a new `locales` axis in the register. Class is
  structural, a trait is what the code DOES, a locale is who it is FOR — the
  standard's own "who the reader is decides what is required". English is the
  default; ms-ai-architect and okr are declared nb, named by the operator as
  Norway-only in audience. Stopword-frequency comparison over prose with code
  stripped: a Norwegian flag name in a shell example cannot decide the
  document. Fires on exactly those two, silent on all sixteen English repos.

One design correction found mid-implementation: the first version returned
SKIP when a README had too little prose to judge, which broke a passing
fixture and would have stopped any terse repo from ever reaching OK. SKIP is
for a check that could not RUN; this one ran, saw everything and found no
prose to be in the wrong language — the same shape as "no licence claim to
back". Insufficient prose is now OK, and evenly bilingual prose is the SKIP,
because there the question is live and unanswered.

Deliberately NOT built, because the evidence does not reach: any rule about
images, diagrams or terminal recordings (every such claim refuted 0-3); a
README length bound (no evidence-based target exists); a section count (would
fire on 9 of 18 — textbook "suspect the CHECK"); Mermaid source length and
#gh-dark-mode-only (zero occurrences, and the instance limit is not readable
via the API, so any threshold would be a guess).

Verified live against the operator's own forge (15.0.6+gitea-1.22.0): Mermaid
DOES render in README.md — two div.mermaid-block iframes carrying real SVG —
while #gh-dark-mode-only landed only in Gitea 1.26.0 and is unavailable here.

103 tests green, up from 92. No version bump: the catalog ref still trails at
v0.1.1 against 0.1.3, and starting a second release chain over that is the
operator's call.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TCAGKZT8h9F46ygzSkhEee
This commit is contained in:
Kjell Tore Guttormsen 2026-08-04 09:41:53 +02:00
commit dc386d4471
7 changed files with 474 additions and 2 deletions
Download patch file Download diff file Expand all files Collapse all files

33
CHANGELOG.md
View file

@ -4,6 +4,39 @@ All notable changes to this project are documented here.
Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
versioning is [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
### Added
Two presentation checks, both from adversarially-verified research rather than
taste — `docs/presentation-research-2026-08-03.md` records what the evidence
supports and, more usefully, what it refuses to support.
- `BADGE-COUNT` (`WARN`) — more than five badges. Trockman et al., ICSE 2018
(n=294,941 npm packages) measured a non-linear relationship with popularity
inflecting at five, motivated by surveyed maintainers calling over-badged
READMEs cluttered and "trying too hard". `WARN` and never `ERROR`: the
coefficient sits in an appendix without CI or p-value, so it carries "more is
not better" and cannot carry a hard limit. Counting uses a narrower rule than
the existing claim check — a screenshot or architecture diagram must not be
counted as clutter. Measured: 8 of 18 `open/` repos are past it.
- `README-LANGUAGE` (`WARN`) — the prose is not in the language this repo's
readers were declared to speak, via a new `locales` axis in the register.
English is the default; `ms-ai-architect` and `okr` are declared `nb` as
Norway-only in audience. Detection is a stopword-frequency comparison over
prose with code stripped, so a Norwegian flag name in a shell example cannot
decide the document. Evenly bilingual prose is a `SKIP` — the question is
live and unanswered. No running prose is an `OK`: nothing claims a language,
and a thin README is `checkFirstScreen`'s business. Measured: fires on
exactly the two declared repos, silent on all sixteen English ones.
Deliberately **not** built, because the evidence does not reach: any rule about
images, diagrams, screenshots or terminal recordings (every such claim was
refuted 0-3); a README length bound (no evidence-based target exists); a section
count (would fire on 9 of 18 — textbook "suspect the CHECK"); Mermaid source
length and `#gh-dark-mode-only` (zero occurrences, and the instance's actual
limit is not readable via the API, so any threshold would be a guess).
## [0.1.3] — 2026-08-03
### Fixed

2
CLAUDE.md
View file

@ -71,7 +71,7 @@ would recreate, in data, exactly the drift this plugin exists to remove.
## Commands
```bash
npm test # 84 tests
npm test # 103 tests
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
node scripts/repo-standard-check.mjs --offline # no network call
node scripts/repo-standard-check.mjs --json # machine output

20
README.md
View file

@ -67,6 +67,8 @@ The repository's **class** decides what each check means:
| Relative links | a link points at a file that is not tracked |
| Licence claim | the README cites a licence the repo has no file for |
| Badges | a static image badge — any host — asserts a test, build or coverage run that nothing verifies |
| Badge count | more than five badges, the inflection measured by Trockman et al. (ICSE 2018) past which a badge row reads as clutter rather than as evidence. `WARN` only — the source cannot carry a hard limit |
| README language | the prose is not in the language this repo's readers were declared to speak |
| Boilerplate | template text nobody filled in |
| Version consistency | manifest, README badge, newest CHANGELOG entry and the git tag disagree |
| Description | empty, or past the length bound |
@ -90,6 +92,22 @@ untrusted input: a `SECURITY.md` with a real disclosure channel, and a
`## Known limitations` section. Traits live in the register and are the
maintainer's judgement, not a reading.
### Locales — a third axis
Class is structural, a trait is what the code *does*, and a locale is who the
code is *for*. English is the default and is not listed. A repo aimed only at a
Norwegian readership is declared `nb` in the register, and is then wrong in
English rather than right — the standard's own principle is that who the reader
is decides what is required, and language is the first thing that decides.
Detection is a stopword-frequency comparison over prose with code stripped, so a
Norwegian flag name in a shell example cannot decide what the document is. It
answers which language dominates and nothing else: a README can pass this and
still be badly written. Where the prose is too evenly bilingual to call, the
finding is a `SKIP` — the question is live and unanswered. Where there is no
running prose at all, it is an `OK`: nothing claims a language, and a thin README
is the first-screen check's business, not this one's.
### What is deliberately not required
`CONTRIBUTING.md`, `CODE_OF_CONDUCT.md` and `MAINTAINERS.md` are required by no
@ -149,7 +167,7 @@ without that, a raw scan turns three dead names into about twenty.
npm test
```
84 tests over the pure classifiers. The reference fixtures are measured false
103 tests over the pure classifiers. The reference fixtures are measured false
positives, each with its expected verdict — the six that produced the
three-outcome reference rule, plus the noise sources found by running the gate
against a real repository: regexes inside code spans that are markdown links to

186
docs/presentation-research-2026-08-03.md Normal file
View file

@ -0,0 +1,186 @@
# Presentation research — what "professional" can and cannot be measured as
**Date:** 2026-08-03 · **Status:** research complete, architecture decision open
**Method:** adversarial deep research (107 agents, 25 sources fetched, 124 claims
extracted, 25 verified, **11 confirmed / 14 refuted**) + direct measurement of
the 18 locally-cloned `open/` repos + live verification against the operator's
own Forgejo instance.
The headline is uncomfortable and load-bearing: **most of what people call
"professional presentation" has no evidence base as a requirement.** Half the
research question returned nothing that survived adversarial verification. That
result constrains what this plugin may honestly check.
---
## 1. Measured: the org as it stands
18 of the 19 registered `open/` repos (`.profile` is not cloned locally).
| Dimension | Measurement |
| --- | --- |
| Real images | **0.** Every `![...]` in all 18 READMEs resolves to a badge. |
| Mermaid diagrams | 3 blocks total — `voyage` 2, `llm-security` 1, `graceful-handoff` 1. |
| README length | 1,239 → 8,725 words. 7× spread. |
| H2 sections | 5 → 26. Nine repos sit above the empirical IQR ceiling of 12. |
| Badges | 0 → 8. **Six repos carry 7–8.** |
| Language | `ms-ai-architect` ships Norwegian body prose *and* a Norwegian changelog table; `okr` ships a Norwegian changelog table. Both are public. |
| Heading casing | Mixed *within one file* in 5/18 — "What Is This?" beside "What it does". |
| First H2 | `Install` in 15/18. The org's strongest existing coherence signal. |
`linkedin-studio` was flagged by the crude language heuristic and cleared on
inspection — a false positive, discarded before it reached this table.
## 2. Confirmed by the evidence
Peer-reviewed, adversarially survived. **Every number here is descriptive of
what typical repos do, not prescriptive of what an excellent repo should do.**
Inverting them into requirements contradicts the authors' own conclusions — the
research flagged this "descriptive-to-normative trap" as its single biggest
methodological hazard.
1. **Structure: median 7 sections, middle 50% between 5 and 12**
(Prana et al., *Empirical Software Engineering* 2019, doi
10.1007/s10664-018-9660-3; n=393 repos / 4,226 annotated sections). The
metric is **sections, not words** — the paper reports no length measurement.
Population filtered to ≥2 KB English `.md` software repos.
2. **Content skew, and the one normative finding in the whole set.** What 97.0%,
How 88.5%, References 60.8%, Who 52.9%, Contribution 27.8%, **Why 25.7%,
When/status 21.4%**. The authors argue in their own voice that supplying
*Why* and *When* is how a repository differentiates itself and signals
maturity. This is the only recommendation sourced from authors rather than
from inverting a statistic.
3. **No evidence-based README length target exists.** The only measured
baselines (Gaughan et al., CHASE 2025) are of *first versions*, which are
stubs — median reading time 14.79 s, 39% readable in under 10 seconds. The
authors frame this as a deficiency, not a norm. Any length rule is taste.
4. **Badges: a measured inflection at 5.** Trockman et al., ICSE 2018 (doi
10.1145/3180155.3180209; n=294,941 npm packages): a non-linear relationship
with popularity inflecting at 5 badges, motivated by survey respondents
calling over-badged READMEs cluttered and "trying too hard". The paper also
splits badges into **assessment signals** (a service actually ran an
analysis) versus **conventional signals** (a lookup or a static statement —
"cheap to produce, therefore easy to fake"), and advises limiting the latter.
*This is the only presentation anti-pattern in the entire evidence set with a
measured basis.* It aligns exactly with this plugin's existing
`BADGE-STATIC-CLAIM` invariant. The coefficient sits in an appendix without
CI or p-value, so it supports "badge count is not monotonically beneficial"
and **cannot** support a hard "never exceed 5".
5. **Prose readability is machine-measurable.** Flesch Reading Ease, McAlpine
EFLAW, and Linsear Write via Python `textstat`; published medians 50.86 /
16.50 / 7.14. Linsear Write is the one designed for technical documents.
EFLAW scores accessibility to non-native English readers — it does **not**
measure language choice, and confidence here is medium (single source, and
the medians describe sparse first drafts).
## 3. Refuted — evidence you might expect to have, and do not
Fourteen claims were killed 0–3 or 1–2. The consequential ones:
- **Every claim about visual content failed.** That images are rare and
therefore differentiating; that static images correlate with popularity while
GIFs and video do not; that outbound links are the strongest structural
feature. **There is currently no evidential basis for any rule about
screenshots, diagrams, GIFs, or terminal recordings.** Adding visuals may
still be right — but it is taste, and must be argued as taste.
- **Nothing survived on cross-repo coherence mechanisms.** No evidence on shared
skeletons, org profile pages, catalog hubs, naming taxonomies, or which
mechanisms are GitHub-specific. This is the central half of the question and
it is unresearched, not settled.
- **Nothing survived on named exemplar organisations.**
- **Nothing survived on English-only language policy.** One incidental
datapoint: Prana et al. excluded 48 of 1,193 inspected repos as non-English
(~4%).
- **Nothing survived on prose/markdown linting tooling.** Vale, markdownlint,
textlint and write-good have no evidenced effect on perceived quality. Only
the three `textstat` metrics are evidenced, and only as instruments.
- **Nothing survived on AI-generated-prose tells** beyond badge overload —
arguably the most consequential 2026 anti-pattern, and entirely unevidenced.
## 4. Platform ground truth — verified live, not inferred
Measured against `git.fromaitochitta.com`, **Forgejo `15.0.6+gitea-1.22.0`**:
- **Mermaid renders in `README.md`. Confirmed on the real instance.** The
published `voyage` README yields two `div.mermaid-block` containers, each
holding an `iframe.markup-render` whose `srcdoc` contains real SVG (26,634 and
24,713 bytes), laid out at 191 px and 1,459 px. The server marks the fences
`class="chroma language-mermaid display"`; Forgejo's own frontend then
replaces them. Diagrams therefore need no committed image — and no
user-authored `iframe`, since Forgejo injects its own.
- **GitHub's `#gh-dark-mode-only` / `#gh-light-mode-only` fragments will NOT
work here.** That support landed upstream in **Gitea 1.26.0** (PR #36922,
merged 2026-03-21, released 2026-04-18). This instance is built on
**gitea-1.22.0** — four minor versions below. Both variants would stack.
Any dark-mode image strategy is unavailable today.
- **A rendered caution:** the second `voyage` diagram occupies 1,459 px of
vertical space and wraps node text mid-word. Mermaid rendering *working* and
Mermaid rendering *well* are separate questions.
- `MERMAID_MAX_SOURCE_CHARACTERS` is a per-instance limit; both current diagrams
are under it. A longer diagram degrades silently to a plain code block.
## 5. What this permits, and what it forbids
Cross-referencing the evidence against this plugin's own invariants
(`No hook until the rule is precise`; `When a check fires many times in one
repo, suspect the CHECK`; `a gate that fails every repo teaches people to switch
the gate off`; `the gate sees ONE repo`):
**Defensible as engine checks** — precise, measurable, low false-positive risk:
- Badge count above the measured inflection (as `WARN`, never `ERROR` — the
source cannot carry a hard threshold).
- Non-English prose on a public README. Machine-detectable, and the two hits
found are unambiguous. Needs the operator's policy decision first: this is a
coherence choice, not a research finding.
- Section count outside 5–12, as information rather than a failure.
**Not defensible as engine checks:**
- Anything about images, diagrams, screenshots — no evidence base.
- README length — no evidence base.
- Tone, voice, "reads as AI-generated" — no evidence base, and irreducibly
judgement.
- Readability thresholds — the instruments exist, the *thresholds* do not, and
a wrong threshold is precisely the false-positive class that gets gates
switched off.
**Out of scope for this plugin by its own charter:** everything about
recognisability *across* repos. The gate sees one repo. A shared skeleton, an
org profile page, a consistent description register, a catalog hub — these are
org-level questions, and the research found no evidence to guide them anyway.
## 6. Open decisions
1. **Language policy.** Is English-only the standard for public `open/` repos?
Research offers nothing; this is the operator's call. Two repos fail it today
if the answer is yes.
2. **Where does cross-repo coherence live?** Not here. Deciding *where* is not
the same as deciding *who builds it* — that requires a session in the owning
repo.
3. **Is a visual standard wanted at all**, given that it can only ever be taste?
Mermaid is confirmed available; that is capability, not justification.
## Verification log
| Claim | Source |
| --- | --- |
| Median 7 sections, IQR 5–12 | Prana et al., EMSE 2019, doi 10.1007/s10664-018-9660-3 |
| Why 25.7% / When 21.4%; authors' differentiation argument | ibid., Table 3 + §7 Implications |
| No evidence-based length target; first-version stubs | Gaughan et al., CHASE 2025, arXiv:2502.18440 |
| Badge inflection at 5; assessment vs conventional signals | Trockman et al., ICSE 2018, doi 10.1145/3180155.3180209 |
| Flesch / EFLAW / Linsear via `textstat` | Gaughan et al., CHASE 2025 |
| `#gh-dark-mode-only` landed in Gitea 1.26.0 | go-gitea/gitea PR #36922, issue #35545, release history |
| Mermaid renders in this instance's README | Live DOM inspection, `git.fromaitochitta.com/open/voyage` |
| Instance version `15.0.6+gitea-1.22.0` | `GET /api/v1/version` |
| All §1 org measurements | Direct measurement of 18 cloned repos |
**Not verified:** whether Forgejo carries Gitea PR #36922 in any later release
(only that this instance predates it); whether the markdown sanitizer strips
user-authored `iframe` (two claims failed verification, so embedded terminal
recordings remain unestablished); any claim about exemplar organisations,
linting tools, or English-only policy.

19
register/repos.json
View file

@ -122,6 +122,25 @@
}
},
"$comment_locales": [
"A THIRD axis, orthogonal to both class and trait. Class is structural, a",
"trait is what the code DOES — this is who the code is FOR, which is the",
"standard's own stated principle: who the reader is decides what is",
"required. English is the default and is not listed. A repo aimed ONLY at a",
"Norwegian readership is `nb`, and is then WRONG in English, not right.",
"",
"PROPOSED BY THE OPERATOR, NOT MEASURED — like `traits`, this list is a",
"judgement and no remote can report it. Both entries below were named by",
"the operator on 2026-08-04 as Norway-only in their audience.",
"",
"Detection is a stopword-frequency comparison over prose with code stripped.",
"It answers WHICH language dominates, never whether the prose is any good."
],
"locales": {
"ms-ai-architect": "nb",
"okr": "nb"
},
"description_max_codepoints": 180,
"$comment_length": [
"180 codepoints, not bytes and not UTF-16 units. The same string measures 248",

99
scripts/repo-standard-check.mjs
View file

@ -434,14 +434,30 @@ export function checkVersionConsistency({ pluginVersion, readmeBadge, changelogT
// word loses no real detection.
const CLAIM_BADGE = /(tests?|build|ci|coverage|passing)/i;
// Counting badges needs a NARROWER rule than detecting a dishonest one. The
// claim check reads any image, any host, on purpose. Here the opposite error
// matters: counting a screenshot or an architecture diagram as clutter would
// punish exactly the visual work this standard wants more of.
const BADGE_URL = /shields\.io|badgen\.net|\/badges?[/.]/i;
// Trockman et al., ICSE 2018 (doi 10.1145/3180155.3180209, n=294,941 npm
// packages): badge count relates to popularity non-linearly with a predicted
// inflection at five, and surveyed maintainers called over-badged READMEs
// cluttered and "trying too hard". WARN, never ERROR — the coefficient sits in
// an appendix with no CI or p-value, so it carries "more is not better" and
// cannot carry a hard limit.
const BADGE_INFLECTION = 5;
export function checkBadges({ readme }) {
const findings = [];
let badgeCount = 0;
for (const line of String(readme ?? '').split('\n')) {
// Any image, any host. Restricting this to img.shields.io would have missed
// a self-hosted SVG asserting exactly the same unverified thing.
for (const m of line.matchAll(/(\[)?!\[([^\]]*)\]\(([^)\s]+)\)(\])?/g)) {
const linked = m[1] === '[' && m[4] === ']';
const label = `${m[2]} ${m[3]}`;
if (BADGE_URL.test(m[3])) badgeCount++;
if (!linked && CLAIM_BADGE.test(label)) {
findings.push({
level: 'WARN',
@ -452,10 +468,92 @@ export function checkBadges({ readme }) {
}
}
}
if (badgeCount > BADGE_INFLECTION) {
findings.push({
level: 'WARN',
code: 'BADGE-COUNT',
bucket: 'weakening',
msg: `${badgeCount} badges — past the measured inflection of ${BADGE_INFLECTION}, where a badge row starts reading as clutter rather than as evidence (Trockman et al., ICSE 2018). Keep the ones a reader acts on.`,
});
}
if (findings.length === 0) findings.push({ level: 'OK', code: 'BADGES', msg: 'no static badge asserts an unverified run' });
return findings;
}
// Which language a README is written in is not a property of the code, so no
// remote can report it — it is a property of the READER, and the operator owns
// it. English is the default; a repo aimed only at a Norwegian readership is
// declared `nb` in the register and is then wrong in English, not right.
//
// Detection is a stopword-frequency comparison rather than a dependency: both
// word sets below are chosen to have NO member that is also a common word in
// the other language, which is why `at` and `for` (Norwegian and English both)
// are deliberately absent from each.
const STOPWORDS = {
nb: ['og', 'ikke', 'som', 'det', 'den', 'er', 'på', 'til', 'av', 'med', 'om',
'har', 'kan', 'skal', 'blir', 'være', 'etter', 'når', 'også', 'hvis',
'eller', 'men', 'fra', 'ved', 'mot', 'uten', 'hver', 'alle', 'andre',
'seg', 'dette', 'disse', 'mellom', 'gjennom', 'siden', 'fordi', 'derfor'],
en: ['the', 'and', 'of', 'to', 'in', 'is', 'that', 'with', 'this', 'are',
'be', 'from', 'by', 'as', 'an', 'or', 'not', 'you', 'your', 'we', 'our',
'it', 'on', 'which', 'when', 'what', 'how', 'its', 'they', 'their', 'has',
'can', 'will', 'should', 'each', 'between', 'because', 'therefore'],
};
// Below this there is not enough running prose for a frequency count to mean
// anything, and below the ratio the document is genuinely mixed. Both say so
// rather than guessing — a wrong verdict on language is worse than no verdict.
const LANG_MIN_HITS = 10;
const LANG_MIN_RATIO = 1.5;
function countStopwords(text, words) {
const lower = text.toLowerCase();
let n = 0;
for (const w of words) {
const m = lower.match(new RegExp(`(^|[^\\p{L}])${w}([^\\p{L}]|$)`, 'gu'));
if (m) n += m.length;
}
return n;
}
export function checkReadmeLanguage({ readme, name }, register) {
const declared = register?.locales?.[name] ?? 'en';
const other = declared === 'nb' ? 'en' : 'nb';
// Same discipline as the link and boilerplate checks: a Norwegian flag name
// in a shell example must not decide what language the DOCUMENT is in.
const prose = stripCode(String(readme ?? ''));
const hits = { nb: countStopwords(prose, STOPWORDS.nb), en: countStopwords(prose, STOPWORDS.en) };
// Not a SKIP. SKIP is for a check that could not RUN — the catalog was
// unreachable, the file unreadable. This one ran, saw everything, and found
// no prose to be in the wrong language, the same shape as "no licence claim
// to back". A thin README is a real problem, and it is checkFirstScreen's;
// routing it here would stop any terse repo from ever reaching OK.
if (hits[declared] + hits[other] < LANG_MIN_HITS) {
return [{
level: 'OK',
code: 'LANGUAGE',
msg: `no running prose to judge (${hits[declared] + hits[other]} marker words) — nothing claims a language`,
}];
}
if (hits[other] >= hits[declared] * LANG_MIN_RATIO) {
return [{
level: 'WARN',
code: 'README-LANGUAGE',
bucket: 'weakening',
msg: `README reads as \`${other}\` but this repo is declared \`${declared}\` (${hits[other]} vs ${hits[declared]} marker words). Who the reader is decides the language — fix the prose, or fix \`locales\` in the register.`,
}];
}
if (hits[declared] < hits[other] * LANG_MIN_RATIO) {
return [{
level: 'SKIP',
code: 'README-LANGUAGE-UNDECIDABLE',
msg: `README mixes languages too evenly to call (${hits.nb} nb vs ${hits.en} en) — declared \`${declared}\`, unverified`,
}];
}
return [{ level: 'OK', code: 'LANGUAGE', msg: `README reads as \`${declared}\`, as declared` }];
}
// Template text that was never filled in. A visible unfinished template costs
// more trust than the missing document would have.
const FIXME_RE = /FIXME/;
@ -689,6 +787,7 @@ export function classifyRepo(
...checkInternalLinks({ files, present }),
...checkLicenseClaim({ readme, present }),
...checkBadges({ readme }),
...checkReadmeLanguage({ readme, name }, register),
...checkBoilerplate({ files }),
...checkVersionConsistency({ pluginVersion, readmeBadge, changelogTop, tags }),
...checkDescription(description, register),

117
scripts/repo-standard-check.test.mjs
View file

@ -20,6 +20,7 @@ import {
checkVersionConsistency,
checkHeadings,
checkBadges,
checkReadmeLanguage,
checkBoilerplate,
checkLicenseClaim,
checkInternalLinks,
@ -563,6 +564,122 @@ test('a build- or CI-status badge is still caught — only bare "status" was too
assert.equal(ci.some((x) => x.code === 'BADGE-STATIC-CLAIM'), true);
});
// ----------------------------------------------------------- badge crowding
// Trockman et al., ICSE 2018 (n=294,941 npm packages) measured a non-linear
// relationship between badge count and popularity with a predicted inflection
// at five, motivated by survey respondents calling over-badged READMEs
// cluttered and "trying too hard". The coefficient sits in an appendix with no
// CI or p-value, so this is a WARN and the threshold is the measured inflection
// — not a rounder number that would read as invented.
const badges = (n, wrap = (s) => s) =>
Array.from({ length: n }, (_, i) => wrap(`![B${i}](https://img.shields.io/badge/b${i}-x-blue)`)).join('\n');
test('more than five badges is a finding — the measured inflection point', () => {
const f = checkBadges({ readme: badges(6) });
const hit = f.find((x) => x.code === 'BADGE-COUNT');
assert.ok(hit, 'six badges should produce BADGE-COUNT');
assert.equal(hit.level, 'WARN');
assert.equal(hit.bucket, 'weakening');
});
test('exactly five badges is not a finding — the source cannot carry a harder rule', () => {
assert.equal(checkBadges({ readme: badges(5) }).some((x) => x.code === 'BADGE-COUNT'), false);
});
test('a linked badge still counts toward the total — linking answers honesty, not clutter', () => {
const readme = badges(6, (s) => `[${s}](https://forge.example/run)`);
assert.equal(checkBadges({ readme }).some((x) => x.code === 'BADGE-COUNT'), true);
});
test('content images are not badges, however many there are', () => {
const readme = Array.from({ length: 9 }, (_, i) => `![Architecture ${i}](docs/img/arch${i}.png)`).join('\n');
assert.equal(checkBadges({ readme }).some((x) => x.code === 'BADGE-COUNT'), false);
});
// ---------------------------------------------------------- README language
// The operator owns this axis, exactly as they own `traits`. English is the
// default; a repo aimed ONLY at a Norwegian readership is declared `nb` and is
// then WRONG in English, not right. Measured 2026-08-03: `ms-ai-architect` and
// `okr` are the two such repos, and both currently carry English prose.
const LOCALE_REGISTER = { ...REGISTER, locales: { okr: 'nb' } };
const NB_PROSE = [
'# okr',
'',
'Dette er en plugin som ikke gjør noe annet enn å måle mål og resultater.',
'Den kan kjøres fra Claude Code, og den skal være tilgjengelig når du',
'trenger den. Hvis du vil ha mer, se dokumentasjonen. Alle kommandoer',
'blir kjørt fra en aktiv sesjon, og ingenting av dette krever en server.',
].join('\n');
const EN_PROSE = [
'# okr',
'',
'This is a plugin that does nothing other than measure objectives and',
'results. It can be run from Claude Code, and it is always available when',
'you need it. If you want more, see the documentation. All of the commands',
'are run from an active session, and none of this requires a server.',
].join('\n');
test('a repo declared Norwegian that ships English prose is a finding', () => {
const f = checkReadmeLanguage({ readme: EN_PROSE, name: 'okr' }, LOCALE_REGISTER);
const hit = f.find((x) => x.code === 'README-LANGUAGE');
assert.ok(hit, 'declared nb + English prose should produce README-LANGUAGE');
assert.equal(hit.level, 'WARN');
assert.equal(hit.bucket, 'weakening');
});
test('a repo declared Norwegian that ships Norwegian prose passes', () => {
const f = checkReadmeLanguage({ readme: NB_PROSE, name: 'okr' }, LOCALE_REGISTER);
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
});
test('an undeclared repo defaults to English, so Norwegian prose is the finding', () => {
const f = checkReadmeLanguage({ readme: NB_PROSE, name: 'repo-standard' }, LOCALE_REGISTER);
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), true);
});
test('an undeclared repo shipping English prose passes', () => {
const f = checkReadmeLanguage({ readme: EN_PROSE, name: 'repo-standard' }, LOCALE_REGISTER);
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
});
// Same discipline as the link and boilerplate checks: a Norwegian identifier in
// a shell example must not decide what language the DOCUMENT is written in.
test('code blocks do not decide the language', () => {
const readme = [
EN_PROSE,
'',
'```bash',
'kjør --og --ikke --som --det --den --er --på --til --av --med --om',
'kjør --har --kan --skal --blir --etter --når --også --hvis --eller',
'```',
].join('\n');
const f = checkReadmeLanguage({ readme, name: 'repo-standard' }, LOCALE_REGISTER);
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
});
// A README with no running prose has no language to be wrong about — the check
// ran and found nothing, which is not the same as a check that could not run.
// Routing it to SKIP would mean no terse repo could ever classify OK.
test('no running prose is an OK, not a SKIP — nothing claims a language', () => {
const f = checkReadmeLanguage({ readme: '# thing\n\nA tool.\n', name: 'repo-standard' }, LOCALE_REGISTER);
assert.equal(f.some((x) => x.code === 'README-LANGUAGE'), false);
assert.equal(f[0].level, 'OK');
});
// A document with real prose that the gate genuinely cannot call IS a SKIP —
// here the question is live and unanswered, unlike the empty case above.
test('prose that mixes languages too evenly is a SKIP, never a pass', () => {
const readme = [NB_PROSE, EN_PROSE].join('\n\n');
const f = checkReadmeLanguage({ readme, name: 'repo-standard' }, LOCALE_REGISTER);
const hit = f.find((x) => x.code === 'README-LANGUAGE-UNDECIDABLE');
assert.ok(hit, 'an evenly bilingual README cannot be called');
assert.equal(hit.level, 'SKIP');
});
// -------------------------------------------------------------- boilerplate
test('unfinished template text is a finding', () => {