repo-standard/CLAUDE.md
Kjell Tore Guttormsen f4017bcfe1 feat(engine)!: a skip says which of the two things it is
`notChecked` merged "deliberately not judged" with "could not run". 0.7.0 took
SKIP out of the severity lattice, which fixed "clean repos look skipped"; this
fixes the same defect one level down, where it was saying so more quietly.
Measured across 19 clones, 35 of 39 skips are deliberate — so
`portfolio-optimiser — OK · 11 not checked` announced eleven unread checks when
all eleven were fixture and out-of-repo links.

Every SKIP finding now carries `skip: 'byDesign' | 'notRun'`, set at the
emission site rather than looked up from its code: VERSION-TAG is emitted at
SKIP with no tags and at ERROR with the wrong one, so a code→kind map has to
re-derive a reason the emitter already had. Untagged falls to `notRun`, the
loud side. Results carry `skips: { byDesign, notRun }`; `notChecked` stays a
number and stays the total, because a consumer doing `notChecked > 0` against
an object gets a silent false.

The summary line names only `notRun` — the deliberate ones keep their own
`NOT JUDGED` heading in the detail output. Landing the split in `--json` alone
would have left the symptom exactly where it was.

VERSION-UNAVAILABLE stays `notRun` by decision, not by default: it is arguably
"no subject to judge", the shape checkReadmeLanguage answers with OK, but
re-levelling it moves a repo's status and is a second behaviour change.

Verified: 169 tests. 19 clones swept with both engines offline — 220 judged
findings, every status, notChecked and buckets identical, no finding differing
beyond the new tag. The exhaustiveness test scans the engine SOURCE, not a
finding set: a corpus test only sees the sites it triggers, and the next
emission site added would take the reader's default invisibly.

BREAKING CHANGE: the summary line's coverage qualifier reads `· N not run` and
counts only un-runnable skips, where it read `· N not checked` over all of
them. `--json` gains `skips`; `notChecked` is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DHaQTcaD3w2C6PqVsAmrcv
2026-08-09 21:52:00 +02:00

177 lines
11 KiB
Markdown

# repo-standard
Per-repo gate for the `open/` presentation standard, packaged as a marketplace
plugin.
## Context
Two components, one boundary:
- **Engine (`scripts/repo-standard-check.mjs`)** — pure classifiers with all I/O
resolved into their input, mirroring the marketplace's `check-versions.mjs`.
Findings are `ERROR`/`WARN`/`SKIP`/`OK`; exit 1 on `ERROR`. Pinned by
`scripts/repo-standard-check.test.mjs` (`npm test`).
- **Skill (`skills/repo-standard/`)** — the judgement the script cannot encode.
No checking logic lives here; it calls the engine.
`register/repos.json` is the single taxonomy register (name → class, per-class
requirements, and the known non-repo names). Central by design: per-repo copies
would recreate, in data, exactly the drift this plugin exists to remove.
## Invariants
- **This repo has a PUBLIC remote.** `STATE.md` is LOCAL-ONLY and gitignored
from the first commit. Repos on a private remote track theirs — do not carry
that habit across in either direction.
- **The gate sees ONE repo.** Anything needing a view across the whole org
(topic coverage, competing install forms, catalog-vs-forge divergence) does
not belong here. It is measured where the org is enumerated.
- **It records, it does not fix.** Findings first, remediation afterwards.
Patching while measuring is how the inconsistency it detects was produced.
- **`SKIP` is never a pass — and never a severity either.** A check that could
not run says so and names why. But a skip is the ABSENCE of a verdict, so it
cannot be the worst of a set that contains real ones: a repo's `status` is the
worst *judged* finding, and `SKIP` only when nothing was judged (unregistered,
or an empty finding set). It used to outrank `OK`, which made five repos with
0 ERROR, 0 WARN and a dozen OK headline as "skipped" in org-ops census 05 —
`okr`, with the most OK in the org, read as unread. Fixing it in the status
alone would have traded that for the worse direction, "skipped checks look
clean", so coverage rides its own axis: `notChecked` in `--json`, a `· N not
checked` qualifier on the summary line. Absent `notChecked` means an older
engine and prints the pre-0.7.0 line — it is not zero.
- **A skip is one of TWO facts, and merging them says the same wrong thing more
quietly.** `byDesign` (the check saw it and declined — out-of-repo links,
fixture paths; it can never become a verdict, nobody has an action) versus
`notRun` (a re-run or an operator action resolves it). Measured across 19
clones, 35 of 39 skips were the first kind, which is why `portfolio-optimiser
— OK · 11 not checked` read as eleven unread checks. The kind is set at the
EMISSION site, never looked up from the code: `VERSION-TAG` is emitted at
`SKIP` with no tags and at `ERROR` with the wrong one, so a code→kind map has
to re-derive a reason the emission site already had. Untagged falls to
`notRun`, the loud side. The summary line names only `notRun`; the deliberate
ones keep their own `NOT JUDGED` heading, because the wanted side effect is
exposure, not silence — landing the split in `--json` alone would have left
the symptom untouched. The test that holds this is a scan of the engine
SOURCE, not of a finding set: a corpus test only sees the sites it manages to
trigger, and the next site added would take the reader's default invisibly.
- **When a check fires many times in one repo, suspect the CHECK.** The first
link pass produced ~30 findings against `llm-security` and all were noise —
regexes in code spans, `file:` URLs, relative paths resolved against the wrong
directory. Strip code before scanning text; resolve links against the file
they sit in. A gate that is wrong this often gets switched off.
- **Two axes on every finding.** Level (`ERROR`/`WARN`/`SKIP`/`OK`) and bucket
(`broken`/`missing`/`weakening`). They are independent — a `weakening` finding
can be an `ERROR`.
- **Class is structural, traits are judgement.** Class is read off the catalog
and the remotes. A trait (`security`) says what the code does, which no remote
knows, and the operator owns that list.
- **Who the reader is decides what is required.** Contributor-facing documents
are required by no class — solo-maintained, and the published stance says so.
Consumer-facing ones (`SECURITY.md`, `LICENSE`, non-goals, limitations) are
untouched by that. This is not a rule against having the others.
- **No CI badge, because there is no CI.** The forge has zero Actions runners
registered (measured). The substitute is one command from a clean clone, said
plainly. A static badge asserting a run is the anti-pattern this gate flags —
and an early draft of this README carried one.
- **Three outcomes on references.** "No match" and "match on a known non-repo"
must stay distinct findings. Collapsing them hides real loss inside correct
text — the exact defect class this gate exists to catch.
- **Two API calls per invocation, anonymous, with 429 retry.** The org listing
(description + topics) is one; the catalog's `marketplace.json` for
INSTALL-TRUTH is the other (added after this used to say "one call" — that
line went stale and stayed stale until a 13-repo shell loop trusted it and
tripped the rate limiter at 26 requests). Both go through `fetchWithRetry`,
which retries HTTP 429 rather than silently reporting SKIP. Both are
anonymous — no token, confirmed no different with one — so the gate works
for any reader, not only someone holding one. A sweep across every repo
still does not belong here: it needs the listing fetched once, not once per
invocation, which is a different shape of caller (org-ops), not a flag on
this engine.
**The "13 calls in a loop" explanation was incomplete** (2026-08-04): the
forge's nginx never sends `Retry-After` on its 429s (measured directly), so
`fetchWithRetry` always falls back to exponential backoff — the
`Retry-After` branch is live code with no live path yet. The limit is also
smaller than "loop of 13" implied: 20 concurrent requests from one IP
reproduced it directly, no loop needed, and a single well-formed 2-call
invocation can still lose if something else on the same IP is calling the
forge at the same moment (other repos' hooks, another session). The block
is a leaky bucket, not a fixed ban — a 20-25 request burst took up to ~15s
to fully drain. `fetchWithRetry` defaults to `retries: 5` /
`maxDelayMs: 8000` (23s worst case) to cover that.
- **Codepoints, not bytes, not UTF-16 units.** Use `[...s].length`. An em-dash
exposes only the byte layer; astral characters expose the rest.
- **The reader decides a link's level, not just what is required.** Root
documents are the shop window — a dead link there is an `ERROR`. Below the
root it is a `WARN`: that is where session plans, agent working files and
path-traversal fixtures with deliberately invalid targets live. Measured, 30
of 43 findings were down there and all were `ERROR`s.
- **A fixture-path dead link is `SKIP`, not `WARN` — and never silently
dropped.** `test/`, `tests/`, `fixtures/` (exact segment) and `*golden*`
(substring) mark a path as presumed intentional; the finding still fires as
`LINK-INTERNAL-FIXTURE` with its file and line, it just isn't judged.
Grounded in `nav-golden-escape/bundle/index.md`'s deliberate
`../../../../etc/passwd` escape: the deep `..` pops the whole base path
rather than resolving to `null`, so it read as a genuine `WARN` — third tool
in the org to hit this exact pattern, which is the signal the check was at
fault. Measured before shipping: 16 findings before, 16 after, across all 20
local clones — every one converted 1:1, none disappeared.
- **A repo's name is its remote, not its directory.** `catalog/` holds
`ktg-plugin-marketplace`. The basename left it unregistered with zero checks
run, against the one repo every catalog rule depends on.
- **A decision needs somewhere to live, or the gate repeats itself forever.**
The engine already held that a human README title is the operator's call —
and still warned about it every round, because a YES could not be recorded.
Six warnings, unchanged across censuses 05, 06 and 07. `titles` in the
register is that record: set, the H1 matching it is `OK`; unset, the `WARN`
stands. What the gate must never do is make "we decided this" and "nobody
looked" the same output. The wanted side effect is exposure, not silence —
`ai-psychosis` is deliberately unregistered so it stands alone.
- **An exemption is a finding, not a deletion.** `readme_desc_match: false`
turns off README-DESC equality for a class, and the check still emits an `OK`
naming why. An exception nobody can see reads exactly like a check that
silently stopped running.
- **Class rules live in the register, never as a class name in the engine.**
The `org-profile` exemption is a flag on the class, not `if (klass ===
'org-profile')`. Per-repo copies of a rule are the drift this plugin exists
to remove; a class name hardcoded in a classifier is the same defect one
level up.
- **The version names a file; only the sha names the code.** `engineVersion`
was added because a stale cache served an old engine silently — but a feature
and its version bump are two commits, so a worktree carries new behaviour
under the old number for a window. Measured: a sweep stamped 18 raw files
`0.4.0`, four of them holding findings from a check that only exists in
`0.5.0`. `engineCommit` closes that, derived from the same checkout with no
network call. It is present-and-`null` when underivable, never absent — an
absent key means an older engine, `null` means this one ran without a HEAD.
- **No hook until the rule is precise.** A blocking gate that fails a correct
repository is the mechanism that gets gates switched off.
## Commands
```bash
npm test # 169 tests
node scripts/repo-standard-check.mjs --dir "$PWD" # gate one repo
node scripts/repo-standard-check.mjs --offline # no network call
node scripts/repo-standard-check.mjs --json # machine output
node scripts/repo-standard-check.mjs --refresh # register vs. forge
```
## Release
**Run `--refresh` before every release.** Register freshness is owned HERE, not
by the sweeps that read the register. Twice running, a newly published repo was
missing when a census ran, and the cost is not a gap — it is false ERRORs in a
*different* repo: `portfolio-optimiser` earned three `LINK-DEAD`s against a repo
that existed, in the same round it fixed its three real ones, so its status line
did not move even though the work was done. A stale register makes the gate
lie about repos that are not even the stale one. One owner, no shared duty:
consumers of the register are told not to check freshness themselves.
Polyrepo rule: a version bump is not finished until the tag `vX.Y.Z` is pushed
**and** the catalog `ref` is bumped to it. Use `release-plugin.mjs`, never a
hand-edited `ref`.
`check-versions.mjs` reads the catalog README's per-plugin label, and a **missing**
entry is a silent `null` rather than an error — `release-plugin.mjs` rewrites an
existing heading but cannot create one. A new plugin's catalog README entry has
to be added by hand once.