docs(voyage): S19 — leanness: honest test census + prune 3 redundant prose-pins
Closes devils-advocate audit §Top changes #8 (MINOR×MED). Reports the behavior-test count separately from the doc-consistency prose-pin count so the cited total no longer oversells behavior coverage, and removes the clearest reword-fragile redundancies. Conservative scope (operator-chosen): no genuine regression guard dropped. 1. Suite census (report separately). lib/util/test-census.mjs + tests/lib/test-census.test.mjs walk tests/**/*.test.mjs and bucket top-level test() declarations into behavior vs doc-consistency-pins (bucket per file, regex PIN_FILE_RE), asserting the two sum to the total so neither can drift silently. Split emitted as a t.diagnostic: behavior=601 doc-consistency-pins=69. Metric = top-level declarations; node:test's runtime total counts subtests too and is therefore >= it. 2. Conservative prune (3 tests, each provably subsumed): - "trekexecute.md still parses v1.7 plan schema" — tautological OR-chain; real coverage = the plan_version:1.7 template pin + plan-validator / plan-schema behavior tests. - "CLAUDE.md mentions all six pipeline commands" — hardcoded six-string list subsumed by the filesystem-driven "commands table mentions every commands/*.md file" structural pin. - "CLAUDE.md mentions /trekcontinue command" — same subsumption. Each removal leaves an in-place note (why + where coverage lives). 3. doc-consistency.test.mjs header now documents the structural-invariant vs prose/existence-pin distinction so new pins land in the right kind. TDD: census test written failing-first (ERR_MODULE_NOT_FOUND before lib/util/test-census.mjs existed). Suite 699->698 (696 pass / 2 skip / 0 fail): -3 prune +2 census. plugin validate passes (1 accepted warning). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
987e847ea1
commit
494a344700
4 changed files with 121 additions and 33 deletions
|
|
@ -51,6 +51,7 @@ Existing `2.0`/`2.1` briefs stay valid (forward + backward compatible), mirrorin
|
|||
### Release hygiene
|
||||
|
||||
- **Version sync** — `plugin.json`, `package.json`, README badge, and the CHANGELOG top entry all at `5.5.0`, guarded by a new version-consistency test in `tests/lib/doc-consistency.test.mjs`.
|
||||
- **Honest test count (S19)** — `lib/util/test-census.mjs` + `tests/lib/test-census.test.mjs` report the behavior-test count separately from the doc-consistency prose-pin count, so the cited total no longer oversells behavior coverage. Pruned 3 redundant/tautological prose-pins from `doc-consistency.test.mjs` (each provably subsumed by a structural invariant or behavior test). Devil's-advocate audit §Top changes #8.
|
||||
- **`claude plugin validate` passes.** The single advisory warning (root `CLAUDE.md` not loaded as consumer project context) is **accepted by design**: it is universal across all marketplace plugins, validation still passes, and the root `CLAUDE.md` is repo/maintainer context — consumer context ships via README + command/agent frontmatter.
|
||||
|
||||
## v5.1.1 — 2026-05-14 — Remediation patch (11/12 review findings closed)
|
||||
|
|
|
|||
49
lib/util/test-census.mjs
Normal file
49
lib/util/test-census.mjs
Normal file
|
|
@ -0,0 +1,49 @@
|
|||
// lib/util/test-census.mjs
|
||||
// Census of the test suite: split top-level test() declarations into
|
||||
// "behavior" tests vs "doc-consistency pins" (string/existence assertions that
|
||||
// pin documentation against a source-of-truth). Makes the cited test count
|
||||
// honest — a prose-pin is not the same coverage as a behavior test, so a single
|
||||
// conflated total oversells behavior coverage.
|
||||
// Devil's-advocate audit §Top changes #8 (S19).
|
||||
//
|
||||
// Metric: top-level `test(` declarations (static, deterministic, in-process).
|
||||
// This is distinct from node:test's runtime total, which additionally counts
|
||||
// subtests; the runtime total is therefore ≥ this declaration count.
|
||||
|
||||
import { readFileSync, readdirSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
|
||||
// Files whose tests are documentation pins, not behavior coverage. Kept as a
|
||||
// regex (not a single filename) so a future split-out (e.g. prose-pins.test.mjs)
|
||||
// is bucketed correctly without editing this module.
|
||||
export const PIN_FILE_RE = /(doc-consistency|prose-pins)\.test\.mjs$/;
|
||||
|
||||
function walk(dir) {
|
||||
const out = [];
|
||||
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = join(dir, e.name);
|
||||
if (e.isDirectory()) out.push(...walk(p));
|
||||
else if (e.isFile() && e.name.endsWith('.test.mjs')) out.push(p);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
function countTests(file) {
|
||||
return (readFileSync(file, 'utf-8').match(/^\s*test\(/gm) || []).length;
|
||||
}
|
||||
|
||||
// Returns { behavior, docPins, total, byFile } for all *.test.mjs under
|
||||
// testsRoot. behavior + docPins === total by construction.
|
||||
export function censusTests(testsRoot) {
|
||||
const files = walk(testsRoot).sort();
|
||||
const byFile = {};
|
||||
let behavior = 0;
|
||||
let docPins = 0;
|
||||
for (const f of files) {
|
||||
const n = countTests(f);
|
||||
byFile[f] = n;
|
||||
if (PIN_FILE_RE.test(f)) docPins += n;
|
||||
else behavior += n;
|
||||
}
|
||||
return { behavior, docPins, total: behavior + docPins, byFile };
|
||||
}
|
||||
|
|
@ -4,6 +4,19 @@
|
|||
//
|
||||
// When this test fails, fix the source-of-truth — do NOT rewrite the test to
|
||||
// hide drift. Borrowed pattern from llm-security commit 97c5c9d.
|
||||
//
|
||||
// Two kinds of pin live here (devil's-advocate audit §Top changes #8, S19):
|
||||
// • STRUCTURAL cross-file invariants — derive one side from a source-of-truth
|
||||
// (file counts, dimension counts from `### N.` headers, profile tables from
|
||||
// the yaml, version-sync) and survive harmless rewording. These are the
|
||||
// load-bearing ones; keep and grow them.
|
||||
// • PROSE / EXISTENCE pins — assert a literal string or file fact. Most still
|
||||
// guard a real wiring (CLI shims, annotate.mjs, anti-stale CHANGELOG,
|
||||
// anti-false-claim), but they break on reword. Prefer a structural form
|
||||
// when one exists; do not add a prose-pin that a structural test subsumes.
|
||||
// All tests in this file count as "doc-consistency pins" in the suite census
|
||||
// (lib/util/test-census.mjs) so the cited test total stays honest about how much
|
||||
// is behavior coverage vs documentation pinning.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
|
|
@ -129,16 +142,12 @@ test('templates/plan-template.md declares plan_version: 1.7', () => {
|
|||
assert.match(tpl, /plan_version:\s*['"]?1\.7['"]?/);
|
||||
});
|
||||
|
||||
test('commands/trekexecute.md still parses v1.7 plan schema', () => {
|
||||
const cmd = read('commands/trekexecute.md');
|
||||
const tpl = read('templates/plan-template.md');
|
||||
const tplVersion = (tpl.match(/plan_version:\s*['"]?([\d.]+)['"]?/) || [])[1];
|
||||
assert.ok(tplVersion, 'templates/plan-template.md missing plan_version');
|
||||
assert.ok(
|
||||
cmd.includes(`plan_version`) || cmd.includes(`Step N:`) || cmd.includes('### Step '),
|
||||
'commands/trekexecute.md should reference v1.7 plan-schema parsing',
|
||||
);
|
||||
});
|
||||
// S19 prune (audit §Top changes #8): removed a tautological prose-pin —
|
||||
// "commands/trekexecute.md still parses v1.7 plan schema" was an OR-chain
|
||||
// (`includes('plan_version') || includes('Step N:') || includes('### Step ')`)
|
||||
// that is ~always true and asserts almost nothing. The real v1.7 coverage lives
|
||||
// in the structural pin "templates/plan-template.md declares plan_version: 1.7"
|
||||
// (above) + the plan-validator / plan-schema behavior tests.
|
||||
|
||||
test('settings.json has only known top-level scopes after Spor 0 cleanup', () => {
|
||||
const cfg = JSON.parse(read('settings.json'));
|
||||
|
|
@ -156,21 +165,11 @@ test('settings.json no longer carries vestigial exploration block', () => {
|
|||
'agentTeam block was vestigial — should be deleted in v3.1.0 Spor 0');
|
||||
});
|
||||
|
||||
test('CLAUDE.md mentions all six pipeline commands', () => {
|
||||
// v4.1 Step 21 — added /trekcontinue to coverage (was 5/6 before).
|
||||
// v5.0.0 — /trekrevise removed (bespoke playground retired); back to six.
|
||||
const md = read('CLAUDE.md');
|
||||
for (const c of [
|
||||
'/trekbrief',
|
||||
'/trekresearch',
|
||||
'/trekplan',
|
||||
'/trekexecute',
|
||||
'/trekreview',
|
||||
'/trekcontinue',
|
||||
]) {
|
||||
assert.ok(md.includes(c), `CLAUDE.md missing reference to ${c}`);
|
||||
}
|
||||
});
|
||||
// S19 prune (audit §Top changes #8): removed "CLAUDE.md mentions all six
|
||||
// pipeline commands" — a hardcoded six-string list fully subsumed by the
|
||||
// structural pin "CLAUDE.md commands table mentions every commands/*.md file"
|
||||
// above, which derives the command set from the filesystem and auto-covers any
|
||||
// new command. The hardcoded list was the weaker, reword-fragile form.
|
||||
|
||||
test('HANDOVER-CONTRACTS.md contains Handover 6 section', () => {
|
||||
const text = read('docs/HANDOVER-CONTRACTS.md');
|
||||
|
|
@ -248,14 +247,10 @@ test('HANDOVER-CONTRACTS.md Handover 7 § Lifecycle names --cleanup and produced
|
|||
);
|
||||
});
|
||||
|
||||
test('CLAUDE.md mentions /trekcontinue command', () => {
|
||||
const md = read('CLAUDE.md');
|
||||
assert.ok(
|
||||
md.includes('/trekcontinue') || md.includes('trekcontinue'),
|
||||
'CLAUDE.md should document /trekcontinue in the Commands table ' +
|
||||
'(added in v3.3.0 alongside the new command file)',
|
||||
);
|
||||
});
|
||||
// S19 prune (audit §Top changes #8): removed "CLAUDE.md mentions /trekcontinue
|
||||
// command" — subsumed by "CLAUDE.md commands table mentions every commands/*.md
|
||||
// file" (commands/trekcontinue.md exists, so that loop already asserts the
|
||||
// /trekcontinue reference). Single-command existence pin → redundant.
|
||||
|
||||
test('rule-catalogue has exactly 12 entries', async () => {
|
||||
const mod = await import('../../lib/review/rule-catalogue.mjs');
|
||||
|
|
|
|||
43
tests/lib/test-census.test.mjs
Normal file
43
tests/lib/test-census.test.mjs
Normal file
|
|
@ -0,0 +1,43 @@
|
|||
// tests/lib/test-census.test.mjs
|
||||
// Behavior test for the suite census + the honest-count invariant from the
|
||||
// devil's-advocate audit §Top changes #8 (S19): the behavior-test count must
|
||||
// be reported SEPARATELY from the doc-consistency prose-pin count. A prose-pin
|
||||
// (a string/existence assertion against documentation) is not the same coverage
|
||||
// as a behavior test, so a single conflated total oversells the behavior
|
||||
// coverage. This census makes the split explicit and drift-proof.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { censusTests } from '../../lib/util/test-census.mjs';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const TESTS_ROOT = join(HERE, '..');
|
||||
|
||||
test('suite census splits behavior tests from doc-consistency pins (S19)', (t) => {
|
||||
const c = censusTests(TESTS_ROOT);
|
||||
// Honest-count invariant: the two buckets must account for every top-level
|
||||
// test() declaration — no silent drift between behavior and pin counts.
|
||||
assert.equal(c.behavior + c.docPins, c.total,
|
||||
'census buckets must sum to the total declaration count');
|
||||
assert.ok(c.docPins > 0, 'doc-consistency pin bucket must be non-empty (regex/glob sanity)');
|
||||
assert.ok(c.behavior > 0, 'behavior bucket must be non-empty (regex/glob sanity)');
|
||||
// Report the split so the cited count is honest (audit §Top changes #8).
|
||||
// Metric = top-level test() declarations; node:test's runtime total counts
|
||||
// subtests too and is therefore ≥ this number.
|
||||
t.diagnostic(
|
||||
`behavior=${c.behavior} doc-consistency-pins=${c.docPins} ` +
|
||||
`total=${c.total} (top-level test() declarations)`,
|
||||
);
|
||||
});
|
||||
|
||||
test("doc-consistency.test.mjs is the suite's prose-pin home (S19)", () => {
|
||||
const c = censusTests(TESTS_ROOT);
|
||||
const pinFiles = Object.keys(c.byFile)
|
||||
.filter((f) => /(doc-consistency|prose-pins)\.test\.mjs$/.test(f));
|
||||
assert.ok(
|
||||
pinFiles.some((f) => f.endsWith('doc-consistency.test.mjs')),
|
||||
'doc-consistency.test.mjs must exist as the canonical prose-pin bucket',
|
||||
);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue