feat(review): the review is a gate - verdict in the file, CC engine, project memory
Ordre 5 (wave 1, P11 + P5), review_version 1.1. - K7: `verdict: ALLOW | WARN | BLOCK` is required in review.md (strict error, soft warning for 1.0 reviews). New CLI lib/review/verdict-gate.mjs: 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not decide (never green); a verdict greener than the file's own BLOCKERs or JSON block is exit 2. Realpath main guard (the old file:// guard exits 0 under a symlink - measured). - No finding is struck for length: succinctness trims only MINOR/SUGGESTION text (catalogue tier), keeps BLOCKER/MAJOR whole, counts trims; the coordinator lists what it declined to judge. - Claude Code's /code-review and /security-review run as the engine via the Skill tool (never --fix/--comment/--post/ultra); lib/review/engine-findings.mjs turns their findings into a reviewer payload; two catalogue keys (CODE_REVIEW_DEFECT, SECURITY_REVIEW_VULNERABILITY), catalogue 12 -> 14. An engine that did not run forbids ALLOW. - Every BLOCKER is written as a numbered `### Step R<n>` with its source finding (strict error REVIEW_MISSING_REMEDIATION_STEP). - P5: lib/review/recurrence.mjs - a finding class in two review.md files (current + git history, no new store) becomes one proposal, routed by the catalogue's memory_route; written only between Voyage markers in .claude/rules/voyage-review-lessons.md under a size budget, CLAUDE.md gets a pointer only; the operator approves every line (Phase 8.7). - Workflow substrate: catalogue enum in step; a missing reviewer or engine forbids ALLOW there too (it returned ALLOW on zero findings before). - Fixtures review-run-A/B said WARN over a BLOCKER; now BLOCK. Borrowed and credited in code: Superpowers (declined-to-judge), ai-native-sdlc (cap only nits; mistake/fix/check), Spec Kit (managed block; converge steps), Pocock (retro routing), Kiro (verdict file + gate; evidence per lesson), Cherny (engine -> brief control order; review -> memory; budget). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
2c89a6b580
commit
d88245c06e
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
28 changed files with 2052 additions and 110 deletions
4
tests/fixtures/trekreview/review-run-A.md
vendored
4
tests/fixtures/trekreview/review-run-A.md
vendored
|
|
@ -9,7 +9,7 @@ brief_path: .claude/projects/2026-05-01-jwt-auth/brief.md
|
|||
scope_sha_start: 0123456789abcdef0123456789abcdef01234567
|
||||
scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
|
||||
reviewed_files_count: 3
|
||||
verdict: WARN
|
||||
verdict: BLOCK
|
||||
findings:
|
||||
- d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232
|
||||
- 7861519c326c207aabf17072db51c469bebc217b
|
||||
|
|
@ -23,7 +23,7 @@ findings:
|
|||
## Executive Summary
|
||||
|
||||
Implementation hits the brief's core success criteria (login + refresh + logout) but
|
||||
has one BLOCKER and four MAJOR/MINOR issues. Verdict: **WARN** — fix the BLOCKER
|
||||
has one BLOCKER and four MAJOR/MINOR issues. Verdict: **BLOCK** — fix the BLOCKER
|
||||
before merge; the MAJORs should land in a follow-up plan.
|
||||
|
||||
This is a SYNTHETIC v1.0 fixture for testing the Jaccard determinism pipeline. It is
|
||||
|
|
|
|||
4
tests/fixtures/trekreview/review-run-B.md
vendored
4
tests/fixtures/trekreview/review-run-B.md
vendored
|
|
@ -9,7 +9,7 @@ brief_path: .claude/projects/2026-05-01-jwt-auth/brief.md
|
|||
scope_sha_start: 0123456789abcdef0123456789abcdef01234567
|
||||
scope_sha_end: fedcba9876543210fedcba9876543210fedcba98
|
||||
reviewed_files_count: 3
|
||||
verdict: WARN
|
||||
verdict: BLOCK
|
||||
findings:
|
||||
- d2d0e27875ae9ef0d818cb08bb6f14e6d33c4232
|
||||
- 7861519c326c207aabf17072db51c469bebc217b
|
||||
|
|
@ -25,7 +25,7 @@ findings:
|
|||
|
||||
Same diff as Run A, re-reviewed independently to test determinism. This run found
|
||||
the same 5 findings plus one extra (a placeholder TODO in logout.mjs that Run A
|
||||
missed). Verdict: **WARN** — same as Run A; the extra finding is MAJOR but does
|
||||
missed). Verdict: **BLOCK** — same as Run A; the extra finding is MAJOR but does
|
||||
not change the merge gate.
|
||||
|
||||
This is a SYNTHETIC v1.0 fixture for testing the Jaccard determinism pipeline.
|
||||
|
|
|
|||
|
|
@ -127,20 +127,43 @@ test('reasonablenessFilter — citation-less is REFUTED, unknown rule_key is UNV
|
|||
|
||||
// ---- Pass 2 — judge --------------------------------------------------------
|
||||
|
||||
test('judgeFilter — over-long title and empty recommended_action are UNVERIFIED, not dropped', () => {
|
||||
// Contract change (fail-closed): both implemented Pass 2 tests read a
|
||||
// `.length` and never examine the claim, so neither refutes the finding.
|
||||
test('judgeFilter — an over-long MAJOR title is kept whole; an empty recommended_action is UNVERIFIED', () => {
|
||||
// Length never removes a finding (ordre 5, P11). Actionability reads a
|
||||
// `.length` and never examines the claim, so it cannot refute the finding.
|
||||
// `dropped` is empty here on purpose — the refuting Pass 2 filter (Accuracy)
|
||||
// is the one this deterministic subset excludes.
|
||||
const longTitle = 'x'.repeat(101);
|
||||
const j = judgeFilter([
|
||||
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'x'.repeat(101) }, // too long → unverified
|
||||
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: longTitle }, // MAJOR → kept whole
|
||||
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok', recommended_action: ' ' }, // empty action → unverified
|
||||
{ file: 'x.mjs', line: 3, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'ok' }, // keep (no action field is fine)
|
||||
]);
|
||||
assert.equal(j.kept.length, 1);
|
||||
assert.equal(j.kept.length, 2);
|
||||
assert.equal(j.kept[0].title, longTitle, 'a MAJOR keeps every character');
|
||||
assert.equal(j.dropped.length, 0);
|
||||
assert.equal(j.unverified.length, 2);
|
||||
assert.deepEqual(j.unverified.map((f) => f.suppressed_reason), ['succinctness:title', 'actionability:empty']);
|
||||
assert.equal(j.trimmed.length, 0);
|
||||
assert.deepEqual(j.unverified.map((f) => f.suppressed_reason), ['actionability:empty']);
|
||||
});
|
||||
|
||||
test('judgeFilter — only MINOR/SUGGESTION text is trimmed, and every trim is counted', () => {
|
||||
const j = judgeFilter([
|
||||
{ file: 'x.mjs', line: 1, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', title: 't'.repeat(150), detail: 'd'.repeat(1200) },
|
||||
{ file: 'x.mjs', line: 2, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', title: 'short', detail: 'short' },
|
||||
]);
|
||||
assert.equal(j.kept.length, 2, 'a trimmed finding is still kept');
|
||||
assert.equal(j.trimmed.length, 1);
|
||||
assert.ok(j.kept[0].title.length <= 100 && j.kept[0].detail.length <= 800);
|
||||
assert.deepEqual(j.kept[0].trimmed_fields, ['title', 'detail']);
|
||||
assert.ok(j.kept[0].detail.endsWith('[…trimmed]'), 'the cut is visible in the text');
|
||||
});
|
||||
|
||||
test('judgeFilter — the trim decision reads the catalogue tier, not the reviewer label', () => {
|
||||
// MISSING_TEST is MAJOR in the catalogue; a reviewer calling it MINOR must
|
||||
// not get its text cut.
|
||||
const detail = 'd'.repeat(900);
|
||||
const j = judgeFilter([{ file: 'x.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MINOR', title: 't', detail }]);
|
||||
assert.equal(j.trimmed.length, 0);
|
||||
assert.equal(j.kept[0].detail, detail);
|
||||
});
|
||||
|
||||
// ---- ingest ----------------------------------------------------------------
|
||||
|
|
@ -229,19 +252,34 @@ test('computeVerdict — a reviewer that did not report forbids ALLOW', () => {
|
|||
assert.ok(r.allow_blocked_by.some((x) => x.includes('brief-conformance-reviewer')));
|
||||
});
|
||||
|
||||
test('runContract — a BLOCKER dropped for an over-long title cannot yield ALLOW', () => {
|
||||
// Pass 2 succinctness reads `.length`. It never examines the claim, so it
|
||||
// cannot establish the finding is unreal -- it is unverified, not refuted.
|
||||
test('runContract — a BLOCKER with an over-long title is kept and yields BLOCK', () => {
|
||||
// Before ordre 5 this finding went to `unverified` on `succinctness:title`
|
||||
// (the fail-closed floor kept it from ALLOW, but it fell out of the list).
|
||||
const result = runContract([
|
||||
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||
{ file: 'lib/auth/jwt.mjs', line: 19, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER', title: 'x'.repeat(101), detail: 'algo taken from the JWT header' },
|
||||
] },
|
||||
]);
|
||||
assert.notEqual(result.verdict, 'ALLOW', 'an unsubstantiated BLOCKER must never clear the review');
|
||||
assert.equal(result.findings.length, 0, 'it is still not a kept finding');
|
||||
assert.equal(result.unverified.length, 1);
|
||||
assert.equal(result.unverified[0].suppressed_reason, 'succinctness:title');
|
||||
assert.equal(result.suppressed.length, 1, 'suppressed stays the union of dropped + unverified');
|
||||
assert.equal(result.verdict, 'BLOCK');
|
||||
assert.equal(result.findings.length, 1);
|
||||
assert.equal(result.unverified.length, 0);
|
||||
assert.equal(result.suppressed.length, 0);
|
||||
});
|
||||
|
||||
test('runContract — declined_to_judge lists every unverified finding with its reason; trimmed_count counts trims', () => {
|
||||
const result = runContract([
|
||||
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||
{ file: 'a.mjs', line: 1, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'empty action', recommended_action: '' },
|
||||
{ file: 'b.mjs', line: 2, rule_key: 'MISSING_ERROR_HANDLING', severity: 'MINOR', title: 'long', detail: 'z'.repeat(850), recommended_action: 'Guard it.' },
|
||||
] },
|
||||
]);
|
||||
assert.equal(result.declined_to_judge.length, 1);
|
||||
assert.deepEqual(
|
||||
{ file: result.declined_to_judge[0].file, reason: result.declined_to_judge[0].reason },
|
||||
{ file: 'a.mjs', reason: 'actionability:empty' },
|
||||
);
|
||||
assert.equal(result.trimmed_count, 1);
|
||||
assert.equal(result.counts.MINOR, 1, 'the trimmed MINOR still counts in its tier');
|
||||
});
|
||||
|
||||
test('runContract — a schema-invalid payload cannot yield ALLOW (an unread reviewer is an absent one)', () => {
|
||||
|
|
@ -333,3 +371,25 @@ test('runContract — an anonymous invalid payload forbids ALLOW on its own, wit
|
|||
assert.equal(result.unattributable_payloads, 1);
|
||||
assert.notEqual(result.verdict, 'ALLOW');
|
||||
});
|
||||
|
||||
// ---- No finding is struck for length (ordre 5, P11) --------------------------
|
||||
//
|
||||
// The defect (STATE, Åpne funn; S118): a finding whose `detail` ran past 800
|
||||
// characters went to `unverified` on `succinctness:detail` alone, and S118's most
|
||||
// serious finding fell out of the Handover 6 list that way. Length is packaging,
|
||||
// not a claim: only MINOR/SUGGESTION text is trimmed, and every trim is counted.
|
||||
|
||||
test('runContract — a BLOCKER with a 900-character detail survives and yields BLOCK', () => {
|
||||
const detail = 'The refresh token is accepted after logout because the revocation list is never consulted. '.repeat(10).slice(0, 900);
|
||||
assert.equal(detail.length, 900);
|
||||
const result = runContract([
|
||||
{ reviewer: 'code-correctness-reviewer', findings: [
|
||||
{ file: 'lib/auth/refresh.mjs', line: 31, rule_key: 'SECURITY_INJECTION', severity: 'BLOCKER',
|
||||
title: 'Revoked refresh token still accepted', detail, recommended_action: 'Check the revocation list before issuing.' },
|
||||
] },
|
||||
]);
|
||||
assert.equal(result.verdict, 'BLOCK');
|
||||
assert.equal(result.findings.length, 1, 'the long BLOCKER is a kept finding');
|
||||
assert.equal(result.findings[0].detail, detail, 'its detail is not cut');
|
||||
assert.equal(result.unverified.length, 0);
|
||||
});
|
||||
|
|
|
|||
|
|
@ -209,11 +209,12 @@ test('HANDOVER-CONTRACTS.md contains Handover 7 section (session-state)', () =>
|
|||
});
|
||||
|
||||
test('review-validator has CLI shim', () => {
|
||||
// Ordre 5: the realpath form (no-op-free under a symlink or a space); the run
|
||||
// through a symlink is proven in tests/validators/review-validator.test.mjs.
|
||||
const text = read('lib/validators/review-validator.mjs');
|
||||
assert.ok(
|
||||
text.includes('import.meta.url === '),
|
||||
'lib/validators/review-validator.mjs should expose the standard CLI shim ' +
|
||||
'(if (import.meta.url === `file://${process.argv[1]}`)) so commands can call it from Bash',
|
||||
text.includes('realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url))'),
|
||||
'lib/validators/review-validator.mjs should expose the realpath CLI shim so commands can call it from Bash',
|
||||
);
|
||||
});
|
||||
|
||||
|
|
@ -268,12 +269,12 @@ test('HANDOVER-CONTRACTS.md Handover 7 § Lifecycle names --cleanup and produced
|
|||
// file" (commands/trekcontinue.md exists, so that loop already asserts the
|
||||
// /trekcontinue reference). Single-command existence pin → redundant.
|
||||
|
||||
test('rule-catalogue has exactly 12 entries', async () => {
|
||||
test('rule-catalogue has exactly 14 entries', async () => {
|
||||
const mod = await import('../../lib/review/rule-catalogue.mjs');
|
||||
assert.strictEqual(
|
||||
mod.RULE_CATALOGUE.length,
|
||||
12,
|
||||
'lib/review/rule-catalogue.mjs RULE_CATALOGUE size invariant: must be 12 (v1.0 baseline)',
|
||||
14,
|
||||
'lib/review/rule-catalogue.mjs RULE_CATALOGUE size invariant: must be 14 (12 v1.0 baseline + 2 engine keys, ordre 5)',
|
||||
);
|
||||
});
|
||||
|
||||
|
|
|
|||
77
tests/lib/engine-findings.test.mjs
Normal file
77
tests/lib/engine-findings.test.mjs
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
// tests/lib/engine-findings.test.mjs
|
||||
// Claude Code's /code-review and /security-review as the engine (ordre 5, P11):
|
||||
// their transcribed findings become a coordinator payload, and meet Voyage's
|
||||
// brief control in the same dedup + verdict.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { toReviewerPayload, ENGINES } from '../../lib/review/engine-findings.mjs';
|
||||
import { runContract } from '../../lib/review/coordinator-contract.mjs';
|
||||
import { validateFindings } from '../../lib/review/findings-schema.mjs';
|
||||
|
||||
test('engine-findings — a /code-review finding becomes a MAJOR CODE_REVIEW_DEFECT', () => {
|
||||
const r = toReviewerPayload({ engine: 'code-review', findings: [
|
||||
{ file: './lib/a.mjs', line: 12, summary: 'Off-by-one drops the last page', failure_scenario: '10 items, page size 5 → page 2 empty', category: 'correctness' },
|
||||
] });
|
||||
assert.equal(r.ok, true, r.errors.join('\n'));
|
||||
assert.equal(r.payload.reviewer, ENGINES['code-review'].reviewer);
|
||||
const [f] = r.payload.findings;
|
||||
assert.deepEqual([f.rule_key, f.severity, f.file, f.line], ['CODE_REVIEW_DEFECT', 'MAJOR', 'lib/a.mjs', 12]);
|
||||
assert.match(f.detail, /page 2 empty/);
|
||||
assert.equal(validateFindings(r.payload).valid, true);
|
||||
});
|
||||
|
||||
test('engine-findings — a /security-review finding becomes a BLOCKER SECURITY_REVIEW_VULNERABILITY', () => {
|
||||
const r = toReviewerPayload({ engine: 'security-review', findings: [
|
||||
{ file: 'lib/q.mjs', line: 4, summary: 'SQL built from request body' },
|
||||
] });
|
||||
assert.equal(r.ok, true);
|
||||
assert.equal(r.payload.findings[0].severity, 'BLOCKER');
|
||||
});
|
||||
|
||||
test('engine-findings — a narrower catalogue key given by the orchestrator is kept; an unknown one falls back', () => {
|
||||
const r = toReviewerPayload({ engine: 'code-review', findings: [
|
||||
{ file: 'a.mjs', line: 1, summary: 's', rule_key: 'MISSING_TEST' },
|
||||
{ file: 'b.mjs', line: 1, summary: 's', rule_key: 'MADE_UP' },
|
||||
] });
|
||||
assert.deepEqual(r.payload.findings.map((f) => f.rule_key), ['MISSING_TEST', 'CODE_REVIEW_DEFECT']);
|
||||
});
|
||||
|
||||
test('engine-findings — a file outside the review scope is listed, never dropped silently', () => {
|
||||
const r = toReviewerPayload({ engine: 'security-review', findings: [
|
||||
{ file: 'in.mjs', line: 1, summary: 'a' },
|
||||
{ file: 'old/out.mjs', line: 1, summary: 'b' },
|
||||
] }, { scopeFiles: ['in.mjs'] });
|
||||
assert.equal(r.payload.findings.length, 1);
|
||||
assert.equal(r.out_of_scope.length, 1);
|
||||
assert.equal(r.out_of_scope[0].file, 'old/out.mjs');
|
||||
});
|
||||
|
||||
test('engine-findings — empty file or summary is an input error, unknown engine too', () => {
|
||||
assert.equal(toReviewerPayload({ engine: 'code-review', findings: [{ file: '', summary: 'x' }] }).ok, false);
|
||||
assert.equal(toReviewerPayload({ engine: 'code-review', findings: [{ file: 'a', summary: '' }] }).ok, false);
|
||||
assert.equal(toReviewerPayload({ engine: 'simplify', findings: [] }).ok, false);
|
||||
});
|
||||
|
||||
test('engine-findings — engine and brief control meet in one verdict; engine + Voyage on the same triplet collapse', () => {
|
||||
const engine = toReviewerPayload({ engine: 'code-review', findings: [
|
||||
{ file: 'lib/a.mjs', line: 12, summary: 'No test covers the new branch', rule_key: 'MISSING_TEST' },
|
||||
] }).payload;
|
||||
const conformance = { reviewer: 'brief-conformance-reviewer', findings: [
|
||||
{ file: 'lib/a.mjs', line: 12, rule_key: 'MISSING_TEST', severity: 'MAJOR', title: 'SC2 untested' },
|
||||
{ file: 'lib/b.mjs', line: 3, rule_key: 'UNIMPLEMENTED_CRITERION', severity: 'BLOCKER', title: 'SC3 missing' },
|
||||
] };
|
||||
const result = runContract([engine, conformance],
|
||||
{ expectedReviewers: [engine.reviewer, 'brief-conformance-reviewer'] });
|
||||
assert.equal(result.verdict, 'BLOCK', 'the brief control stops');
|
||||
assert.equal(result.findings.length, 2);
|
||||
const a = result.findings.find((f) => f.file === 'lib/a.mjs');
|
||||
assert.deepEqual([...a.raised_by].sort(), ['brief-conformance-reviewer', 'claude-code-review']);
|
||||
});
|
||||
|
||||
test('engine-findings — an engine that was expected but never reported forbids ALLOW', () => {
|
||||
const result = runContract([{ reviewer: 'brief-conformance-reviewer', findings: [] }],
|
||||
{ expectedReviewers: ['brief-conformance-reviewer', ENGINES['security-review'].reviewer] });
|
||||
assert.equal(result.verdict, 'WARN');
|
||||
assert.deepEqual(result.missing_reviewers, ['claude-security-review']);
|
||||
});
|
||||
|
|
@ -25,7 +25,7 @@ const RUN_B = fixture('review-run-B.md');
|
|||
|
||||
test('parseReviewArtifact — extracts verdict + finding IDs from frontmatter', () => {
|
||||
const a = parseReviewArtifact(RUN_A);
|
||||
assert.equal(a.verdict, 'WARN');
|
||||
assert.equal(a.verdict, 'BLOCK', 'one BLOCKER → BLOCK (the fixture said WARN until ordre 5; the threshold table never allowed that)');
|
||||
assert.ok(Array.isArray(a.findingIds));
|
||||
assert.equal(a.findingIds.length, 5);
|
||||
for (const id of a.findingIds) assert.match(id, /^[0-9a-f]{40}$/);
|
||||
|
|
@ -71,7 +71,8 @@ test('fidelityDiff — A vs B (A ⊂ B): same verdict, jaccard 5/6, equivalent a
|
|||
|
||||
test('fidelityDiff — divergent verdict makes it non-equivalent even at jaccard 1.0', () => {
|
||||
const a = parseReviewArtifact(RUN_A);
|
||||
const flipped = RUN_A.replace('verdict: WARN', 'verdict: BLOCK');
|
||||
const flipped = RUN_A.replace('verdict: BLOCK', 'verdict: WARN');
|
||||
assert.notEqual(flipped, RUN_A, 'fixture control: the verdict line was flipped');
|
||||
const d = fidelityDiff(RUN_A, flipped);
|
||||
assert.equal(d.verdictMatch, false);
|
||||
assert.equal(d.jaccard, 1);
|
||||
|
|
|
|||
128
tests/lib/review-prose-pins.test.mjs
Normal file
128
tests/lib/review-prose-pins.test.mjs
Normal file
|
|
@ -0,0 +1,128 @@
|
|||
// tests/lib/review-prose-pins.test.mjs
|
||||
// Prose pins for the review process (ordre 5, bølge 1: P11 + P5). Counted as
|
||||
// doc-pins by lib/util/test-census.mjs. Each pin names the behaviour it guards
|
||||
// so a failing pin says what broke, not just which string moved.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { readFileSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { RULE_KEYS } from '../../lib/review/rule-catalogue.mjs';
|
||||
import { UNVERIFIED_REASONS } from '../../lib/review/coordinator-contract.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const read = (rel) => readFileSync(join(ROOT, rel), 'utf-8');
|
||||
|
||||
function section(text, heading) {
|
||||
const start = text.indexOf(heading);
|
||||
assert.ok(start >= 0, `heading missing: ${heading}`);
|
||||
const next = text.indexOf('\n## ', start + heading.length);
|
||||
return text.slice(start, next === -1 ? undefined : next);
|
||||
}
|
||||
|
||||
const CMD = read('commands/trekreview.md');
|
||||
const COORD = read('agents/review-coordinator.md');
|
||||
|
||||
// ---- P11: Claude Code's review as the engine --------------------------------
|
||||
|
||||
test('trekreview — frontmatter allows the Skill tool (the engine is started through it)', () => {
|
||||
const fm = CMD.slice(0, CMD.indexOf('\n---', 4));
|
||||
assert.match(fm, /allowed-tools:.*\bSkill\b/);
|
||||
});
|
||||
|
||||
test('trekreview — Phase 4.7 starts /code-review and /security-review through the Skill tool', () => {
|
||||
const p = section(CMD, '## Phase 4.7');
|
||||
assert.ok(p.includes('Skill("code-review"'), 'code-review via Skill');
|
||||
assert.ok(p.includes('Skill("security-review")'), 'security-review via Skill');
|
||||
assert.ok(p.includes('lib/review/engine-findings.mjs'), 'engine findings go through the adapter');
|
||||
});
|
||||
|
||||
test('trekreview — the engine never edits code, posts or uploads (no --fix/--comment/--post/ultra)', () => {
|
||||
const p = section(CMD, '## Phase 4.7');
|
||||
assert.match(p, /\*\*Never\*\* pass `--fix`, `--comment`, `--post` or `ultra`/);
|
||||
const call = p.match(/Skill\("code-review", args: "([^"]*)"\)/);
|
||||
assert.ok(call, 'the code-review call is written out');
|
||||
for (const banned of ['--fix', '--comment', '--post', 'ultra']) {
|
||||
assert.ok(!call[1].includes(banned), `the engine call must not carry ${banned}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('trekreview — an engine that did not run forbids ALLOW (fail-closed), and engine payloads stay out of the reviewers', () => {
|
||||
const p = section(CMD, '## Phase 4.7');
|
||||
assert.match(p, /forbids `ALLOW`/);
|
||||
assert.match(p, /NOT shown to the Phase 5\s+reviewers/);
|
||||
});
|
||||
|
||||
test('trekreview — the engine claims cite the live docs they were verified in', () => {
|
||||
const p = section(CMD, '## Phase 4.7');
|
||||
assert.ok(p.includes('v2.1.246'), 'model-invocation floor for /code-review');
|
||||
assert.ok(p.includes("origin's"), '/security-review base is origin default branch');
|
||||
});
|
||||
|
||||
// ---- P11: no finding struck for length; declined-to-judge ----------------------
|
||||
|
||||
test('coordinator — succinctness never removes a finding; only MINOR/SUGGESTION text is trimmed', () => {
|
||||
assert.match(COORD, /\| Succinctness \|[^\n]*never removed/);
|
||||
assert.ok(!UNVERIFIED_REASONS.includes('succinctness:detail'), 'length is no longer an unverified reason');
|
||||
assert.ok(!UNVERIFIED_REASONS.includes('succinctness:title'));
|
||||
});
|
||||
|
||||
test('coordinator — writes ## Declined to Judge with the trim count', () => {
|
||||
assert.ok(COORD.includes('`## Declined to Judge` — REQUIRED'));
|
||||
assert.ok(COORD.includes('Trimmed: N'));
|
||||
});
|
||||
|
||||
// ---- K7: the verdict in the file, and the gate -------------------------------
|
||||
|
||||
test('coordinator — review_version 1.1 frontmatter carries verdict', () => {
|
||||
assert.ok(COORD.includes('review_version: "1.1"'));
|
||||
assert.match(COORD, /\nverdict: \{BLOCK \| WARN \| ALLOW\}\n/);
|
||||
});
|
||||
|
||||
test('trekreview — Phase 8 decides with lib/review/verdict-gate.mjs and calls exit 2 not green', () => {
|
||||
const p = section(CMD, '## Phase 8 —');
|
||||
assert.ok(p.includes('lib/review/verdict-gate.mjs'));
|
||||
assert.match(p, /never\s+green/);
|
||||
});
|
||||
|
||||
test('trekreview — a BLOCKER is written as a numbered Step R<n> with Source finding (Handover 6)', () => {
|
||||
assert.ok(CMD.includes('### Step R<n>'));
|
||||
assert.ok(COORD.includes('**Source finding:**'));
|
||||
const h6 = section(read('docs/HANDOVER-CONTRACTS.md'), '## Handover 6');
|
||||
assert.ok(h6.includes('Remediation Steps'), 'Handover 6 documents the remediation steps');
|
||||
assert.match(h6, /\| `verdict` \| string \| yes/, 'Handover 6 lists verdict as required');
|
||||
assert.ok(h6.includes('verdict-gate.mjs'), 'Handover 6 names the gate');
|
||||
});
|
||||
|
||||
// ---- P5: project memory --------------------------------------------------------
|
||||
|
||||
test('trekreview — Phase 8.7 asks the operator per line (AskUserQuestion) before recurrence.mjs apply', () => {
|
||||
const p = section(CMD, '## Phase 8.7');
|
||||
const ask = p.indexOf('AskUserQuestion');
|
||||
const apply = p.indexOf('recurrence.mjs apply');
|
||||
assert.ok(ask >= 0 && apply > ask, 'approval comes before the write');
|
||||
assert.match(p, /over-budget/);
|
||||
assert.match(p, /belong in a skill/);
|
||||
});
|
||||
|
||||
test('trekreview — the write-outside-project_dir exception names only the marked block and the pointer', () => {
|
||||
const rules = section(CMD, '## Hard rules');
|
||||
assert.ok(rules.includes('.claude/rules/voyage-review-lessons.md'));
|
||||
assert.ok(rules.includes('only after the operator approved that exact line'));
|
||||
});
|
||||
|
||||
// ---- the Workflow substrate stays in step -----------------------------------------
|
||||
|
||||
test('armB workflow — its inlined rule-key enum equals the catalogue (no drift)', () => {
|
||||
const src = read('scripts/trekreview-armB.workflow.mjs');
|
||||
const block = src.slice(src.indexOf('const RULE_KEYS = ['), src.indexOf('];', src.indexOf('const RULE_KEYS = [')));
|
||||
const keys = [...block.matchAll(/'([A-Z_]+)'/g)].map((m) => m[1]).sort();
|
||||
assert.deepEqual(keys, [...RULE_KEYS].sort());
|
||||
});
|
||||
|
||||
test('armB workflow — a missing reviewer or engine forbids ALLOW on the Workflow path too', () => {
|
||||
const src = read('scripts/trekreview-armB.workflow.mjs');
|
||||
assert.ok(src.includes("if (verdict === 'ALLOW' && missingReviewers > 0) verdict = 'WARN';"));
|
||||
assert.ok(src.includes('enginePayloads'));
|
||||
});
|
||||
239
tests/lib/review-recurrence.test.mjs
Normal file
239
tests/lib/review-recurrence.test.mjs
Normal file
|
|
@ -0,0 +1,239 @@
|
|||
// tests/lib/review-recurrence.test.mjs
|
||||
// P5 — the project's memory (ordre 5). A finding class (the same rule_key) that
|
||||
// shows up in two review.md files in the target repo becomes ONE proposal; one
|
||||
// sighting is no proposal. Proposals are read from the review.md files the repo
|
||||
// already has (no new store), and an approved one is written only between
|
||||
// Voyage's own markers, under a size budget, with CLAUDE.md getting a pointer.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, mkdirSync, writeFileSync, readFileSync, rmSync, existsSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const CLI = join(ROOT, 'lib', 'review', 'recurrence.mjs');
|
||||
|
||||
const mod = existsSync(CLI) ? await import('../../lib/review/recurrence.mjs') : null;
|
||||
|
||||
function reviewText({ slug, sha, findings }) {
|
||||
const json = JSON.stringify({
|
||||
verdict: 'WARN',
|
||||
counts: { BLOCKER: 0, MAJOR: findings.length, MINOR: 0, SUGGESTION: 0 },
|
||||
findings,
|
||||
}, null, 2);
|
||||
return `---
|
||||
type: trekreview
|
||||
review_version: "1.1"
|
||||
task: "${slug}"
|
||||
slug: ${slug}
|
||||
project_dir: .claude/projects/2026-10-0${sha.length % 9}-${slug}/
|
||||
brief_path: .claude/projects/x/brief.md
|
||||
scope_sha_end: ${sha}
|
||||
reviewed_files_count: 1
|
||||
verdict: WARN
|
||||
findings:
|
||||
${findings.map((f) => ` - ${f.id}`).join('\n')}
|
||||
---
|
||||
|
||||
# Review: ${slug}
|
||||
|
||||
## Executive Summary
|
||||
|
||||
x
|
||||
|
||||
## Coverage
|
||||
|
||||
x
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
x
|
||||
|
||||
\`\`\`json
|
||||
${json}
|
||||
\`\`\`
|
||||
`;
|
||||
}
|
||||
|
||||
function finding(id, rule_key, extra = {}) {
|
||||
return {
|
||||
id: id.padEnd(40, '0'), severity: 'MAJOR', rule_key, file: 'lib/a.mjs', line: 3,
|
||||
title: `${rule_key} title ${id}`, detail: 'd', recommended_action: `Fix it (${id}).`, ...extra,
|
||||
};
|
||||
}
|
||||
|
||||
function repoWith(reviews) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'recurrence-'));
|
||||
reviews.forEach((r, i) => {
|
||||
const p = join(dir, '.claude', 'projects', `2026-10-0${i + 1}-${r.slug}`);
|
||||
mkdirSync(p, { recursive: true });
|
||||
writeFileSync(join(p, 'review.md'), reviewText(r));
|
||||
});
|
||||
return dir;
|
||||
}
|
||||
|
||||
test('recurrence — the module exists', () => {
|
||||
assert.ok(mod, 'lib/review/recurrence.mjs must exist (P5)');
|
||||
});
|
||||
|
||||
test('recurrence — two review.md with the same finding class → exactly one proposal', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_TEST', { file: 'lib/b.mjs' })] },
|
||||
]);
|
||||
try {
|
||||
const { proposals } = mod.proposeLessons(repo);
|
||||
assert.equal(proposals.length, 1);
|
||||
assert.equal(proposals[0].rule_key, 'MISSING_TEST');
|
||||
assert.equal(proposals[0].evidence.length, 2, 'the proposal carries both findings as its evidence');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — one review.md with the class → no proposal', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'PLACEHOLDER_IN_CODE')] },
|
||||
]);
|
||||
try {
|
||||
assert.deepEqual(mod.proposeLessons(repo).proposals, []);
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — the same class twice inside ONE review is still one sighting → no proposal', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST'), finding('a2', 'MISSING_TEST', { line: 9 })] },
|
||||
]);
|
||||
try {
|
||||
assert.deepEqual(mod.proposeLessons(repo).proposals, []);
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — routing: mechanical → a check, judgement → a .claude/rules/ file', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'PLACEHOLDER_IN_CODE'), finding('a2', 'PLAN_EXECUTE_DRIFT')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'PLACEHOLDER_IN_CODE'), finding('b2', 'PLAN_EXECUTE_DRIFT')] },
|
||||
]);
|
||||
try {
|
||||
const byKey = Object.fromEntries(mod.proposeLessons(repo).proposals.map((p) => [p.rule_key, p]));
|
||||
assert.equal(byKey.PLACEHOLDER_IN_CODE.route, 'check');
|
||||
assert.equal(byKey.PLAN_EXECUTE_DRIFT.route, 'rule');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — the proposed line has the three parts: mistake · fix · how to check', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_TEST')] },
|
||||
]);
|
||||
try {
|
||||
const [p] = mod.proposeLessons(repo).proposals;
|
||||
assert.match(p.line, /Mistake:/);
|
||||
assert.match(p.line, /Fix:/);
|
||||
assert.match(p.line, /How to check:/);
|
||||
assert.equal(mod.validateLessonLine(p.line).valid, true);
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — review-hygiene classes never become target-repo lessons', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_BRIEF_REF'), finding('a2', 'COVERAGE_SILENT_SKIP')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_BRIEF_REF'), finding('b2', 'COVERAGE_SILENT_SKIP')] },
|
||||
]);
|
||||
try {
|
||||
assert.deepEqual(mod.proposeLessons(repo).proposals, []);
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — apply writes only between Voyage markers and gives CLAUDE.md a pointer only', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_TEST')] },
|
||||
]);
|
||||
try {
|
||||
writeFileSync(join(repo, 'CLAUDE.md'), '# Project\n\nOperator text stays.\n');
|
||||
mkdirSync(join(repo, '.claude', 'rules'), { recursive: true });
|
||||
writeFileSync(join(repo, '.claude', 'rules', mod.LESSONS_FILE_NAME), 'Operator preface.\n');
|
||||
const r = mod.applyLesson(repo, 'MISSING_TEST');
|
||||
assert.equal(r.written, true, JSON.stringify(r));
|
||||
|
||||
const rules = readFileSync(join(repo, '.claude', 'rules', mod.LESSONS_FILE_NAME), 'utf-8');
|
||||
assert.ok(rules.startsWith('Operator preface.\n'), 'text outside the markers is untouched');
|
||||
const start = rules.indexOf(mod.MARK_START);
|
||||
const end = rules.indexOf(mod.MARK_END);
|
||||
assert.ok(start >= 0 && end > start);
|
||||
assert.match(rules.slice(start, end), /MISSING_TEST/);
|
||||
|
||||
const claude = readFileSync(join(repo, 'CLAUDE.md'), 'utf-8');
|
||||
assert.ok(claude.startsWith('# Project\n\nOperator text stays.\n'));
|
||||
assert.ok(claude.includes(mod.LESSONS_FILE_NAME), 'CLAUDE.md gets a pointer to the rules file');
|
||||
assert.ok(!claude.includes('Mistake:'), 'CLAUDE.md gets ONLY the pointer, never the lesson');
|
||||
|
||||
assert.deepEqual(mod.proposeLessons(repo).proposals, [], 'an applied class is not proposed again');
|
||||
const again = mod.applyLesson(repo, 'MISSING_TEST');
|
||||
assert.equal(again.written, false, 'applying twice writes nothing');
|
||||
const claude2 = readFileSync(join(repo, 'CLAUDE.md'), 'utf-8');
|
||||
assert.equal(claude2, claude, 'the pointer is written once');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — apply refuses past the size budget (the lesson belongs in a skill then)', () => {
|
||||
const keys = ['MISSING_TEST', 'PLACEHOLDER_IN_CODE', 'PLAN_EXECUTE_DRIFT', 'SCOPE_CREEP_BUILT'];
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: keys.map((k, i) => finding(`a${i}`, k)) },
|
||||
{ slug: 'two', sha: 'bbb', findings: keys.map((k, i) => finding(`b${i}`, k)) },
|
||||
]);
|
||||
try {
|
||||
const r = mod.applyLesson(repo, 'MISSING_TEST', { budgetEntries: 0 });
|
||||
assert.equal(r.written, false);
|
||||
assert.equal(r.reason, 'over-budget');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — an edited line without the three parts is refused', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_TEST')] },
|
||||
]);
|
||||
try {
|
||||
const r = mod.applyLesson(repo, 'MISSING_TEST', { line: 'just do better' });
|
||||
assert.equal(r.written, false);
|
||||
assert.equal(r.reason, 'invalid-line');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — an earlier committed version of the same review.md counts as history (git, no new store)', () => {
|
||||
// review.md is overwritten per iteration; git keeps the earlier one.
|
||||
const repo = repoWith([{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'PLAN_EXECUTE_DRIFT')] }]);
|
||||
const g = (...a) => spawnSync('git', ['-C', repo, ...a], { encoding: 'utf-8' });
|
||||
try {
|
||||
assert.equal(g('init', '-q').status, 0);
|
||||
g('config', 'user.email', 't@example.invalid');
|
||||
g('config', 'user.name', 't');
|
||||
g('add', '.claude');
|
||||
assert.equal(g('commit', '-q', '-m', 'review 1').status, 0);
|
||||
assert.deepEqual(mod.proposeLessons(repo).proposals, [], 'one iteration is one sighting');
|
||||
|
||||
const p = join(repo, '.claude', 'projects', '2026-10-01-one', 'review.md');
|
||||
writeFileSync(p, reviewText({ slug: 'one', sha: 'bbbb', findings: [finding('b1', 'PLAN_EXECUTE_DRIFT', { line: 40 })] }));
|
||||
const { proposals } = mod.proposeLessons(repo);
|
||||
assert.equal(proposals.length, 1, 'the committed iteration plus the current one recur');
|
||||
assert.equal(proposals[0].rule_key, 'PLAN_EXECUTE_DRIFT');
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('recurrence — CLI propose prints JSON and exits 0; a missing repo exits 2', () => {
|
||||
const repo = repoWith([
|
||||
{ slug: 'one', sha: 'aaa', findings: [finding('a1', 'MISSING_TEST')] },
|
||||
{ slug: 'two', sha: 'bbb', findings: [finding('b1', 'MISSING_TEST')] },
|
||||
]);
|
||||
try {
|
||||
const r = spawnSync(process.execPath, [CLI, 'propose', '--repo', repo], { encoding: 'utf-8' });
|
||||
assert.equal(r.status, 0, r.stderr);
|
||||
assert.equal(JSON.parse(r.stdout).proposals.length, 1);
|
||||
const missing = spawnSync(process.execPath, [CLI, 'propose', '--repo', join(repo, 'nope')], { encoding: 'utf-8' });
|
||||
assert.equal(missing.status, 2);
|
||||
} finally { rmSync(repo, { recursive: true, force: true }); }
|
||||
});
|
||||
|
|
@ -6,6 +6,7 @@ import {
|
|||
SEVERITY_VALUES,
|
||||
CATEGORY_VALUES,
|
||||
getRule,
|
||||
MEMORY_ROUTES,
|
||||
} from '../../lib/review/rule-catalogue.mjs';
|
||||
|
||||
test('RULE_CATALOGUE — every entry has all 4 required fields', () => {
|
||||
|
|
@ -38,9 +39,24 @@ test('RULE_CATALOGUE — all category values within enum', () => {
|
|||
}
|
||||
});
|
||||
|
||||
test('RULE_KEYS.size === RULE_CATALOGUE.length (== 12) — pinned by doc-consistency', () => {
|
||||
test('RULE_KEYS.size === RULE_CATALOGUE.length (== 14) — pinned by doc-consistency', () => {
|
||||
assert.equal(RULE_KEYS.size, RULE_CATALOGUE.length);
|
||||
assert.equal(RULE_CATALOGUE.length, 12);
|
||||
assert.equal(RULE_CATALOGUE.length, 14);
|
||||
});
|
||||
|
||||
test('RULE_CATALOGUE — the two Claude Code engine keys exist at their tiers (ordre 5)', () => {
|
||||
assert.equal(getRule('CODE_REVIEW_DEFECT')?.severity, 'MAJOR');
|
||||
assert.equal(getRule('SECURITY_REVIEW_VULNERABILITY')?.severity, 'BLOCKER');
|
||||
});
|
||||
|
||||
test('RULE_CATALOGUE — every entry declares a memory_route (check | rule | null)', () => {
|
||||
for (const entry of RULE_CATALOGUE) {
|
||||
assert.ok('memory_route' in entry, `${entry.rule_key} has no memory_route`);
|
||||
assert.ok(entry.memory_route === null || MEMORY_ROUTES.includes(entry.memory_route),
|
||||
`${entry.rule_key} has invalid memory_route: ${entry.memory_route}`);
|
||||
}
|
||||
assert.equal(getRule('MISSING_BRIEF_REF').memory_route, null, 'review hygiene is never project memory');
|
||||
assert.equal(getRule('COVERAGE_SILENT_SKIP').memory_route, null, 'review hygiene is never project memory');
|
||||
});
|
||||
|
||||
test('getRule — returns frozen entry on hit, null on miss, null on bad input', () => {
|
||||
|
|
|
|||
153
tests/lib/verdict-gate.test.mjs
Normal file
153
tests/lib/verdict-gate.test.mjs
Normal file
|
|
@ -0,0 +1,153 @@
|
|||
// tests/lib/verdict-gate.test.mjs
|
||||
// K7 — the review verdict gate (ordre 5, P11). One command decides on the
|
||||
// verdict written in review.md: 0 = ALLOW/WARN, 1 = BLOCK, 2 = could not read.
|
||||
// Exit 2 is never green: a missing file, a missing verdict, or a verdict the
|
||||
// file contradicts all land there.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, writeFileSync, rmSync, existsSync, symlinkSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
|
||||
|
||||
function review({ verdict, jsonVerdict, severities = [] }) {
|
||||
const ids = severities.map((_, i) => String(i + 1).padStart(40, 'a'));
|
||||
const fmVerdict = verdict === undefined ? '' : `verdict: ${verdict}\n`;
|
||||
const findings = severities.map((s, i) => ({
|
||||
id: ids[i], severity: s, rule_key: 'MISSING_TEST', file: 'a.mjs', line: i + 1, title: 't', detail: 'd', recommended_action: 'Add a test.',
|
||||
}));
|
||||
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||
for (const s of severities) counts[s] += 1;
|
||||
const json = JSON.stringify({ verdict: jsonVerdict ?? verdict, counts, findings }, null, 2);
|
||||
return `---
|
||||
type: trekreview
|
||||
review_version: "1.1"
|
||||
task: "t"
|
||||
slug: s
|
||||
project_dir: .claude/projects/2026-10-06-s/
|
||||
brief_path: .claude/projects/2026-10-06-s/brief.md
|
||||
scope_sha_end: abc
|
||||
reviewed_files_count: 1
|
||||
${fmVerdict}findings:
|
||||
${ids.map((id) => ` - ${id}`).join('\n')}
|
||||
---
|
||||
|
||||
# Review: t
|
||||
|
||||
## Executive Summary
|
||||
|
||||
x
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| a.mjs | summary-only | default |
|
||||
|
||||
## Remediation Summary
|
||||
|
||||
x
|
||||
|
||||
\`\`\`json
|
||||
${json}
|
||||
\`\`\`
|
||||
`;
|
||||
}
|
||||
|
||||
function runGate(text, { name = 'review.md', write = true } = {}) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'verdict-gate-'));
|
||||
try {
|
||||
const p = join(dir, name);
|
||||
if (write) writeFileSync(p, text);
|
||||
const r = spawnSync(process.execPath, [GATE, p], { encoding: 'utf-8' });
|
||||
return r;
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
test('verdict-gate — the CLI exists', () => {
|
||||
assert.ok(existsSync(GATE), 'lib/review/verdict-gate.mjs must exist (K7)');
|
||||
});
|
||||
|
||||
test('verdict-gate — BLOCK → exit 1', () => {
|
||||
const r = runGate(review({ verdict: 'BLOCK', severities: ['BLOCKER'] }));
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — ALLOW → exit 0', () => {
|
||||
const r = runGate(review({ verdict: 'ALLOW', severities: ['MINOR'] }));
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — WARN → exit 0', () => {
|
||||
const r = runGate(review({ verdict: 'WARN', severities: ['MAJOR'] }));
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — missing file → exit 2', () => {
|
||||
const r = runGate('', { write: false });
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — a review.md without verdict → exit 2, never ALLOW', () => {
|
||||
const r = runGate(review({ verdict: undefined, jsonVerdict: 'ALLOW', severities: ['MINOR'] }));
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — an unknown verdict value → exit 2', () => {
|
||||
const r = runGate(review({ verdict: 'PASS', severities: ['MINOR'] }));
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — frontmatter ALLOW over a BLOCKER finding is a contradiction → exit 2', () => {
|
||||
const r = runGate(review({ verdict: 'ALLOW', severities: ['BLOCKER'] }));
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — frontmatter and trailing JSON disagree → exit 2', () => {
|
||||
const r = runGate(review({ verdict: 'WARN', jsonVerdict: 'BLOCK', severities: ['MAJOR'] }));
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — not a review (wrong type) → exit 2', () => {
|
||||
const r = runGate(review({ verdict: 'ALLOW', severities: ['MINOR'] }).replace('type: trekreview', 'type: trekbrief'));
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('verdict-gate — --json prints the decision as JSON', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'verdict-gate-'));
|
||||
try {
|
||||
const p = join(dir, 'review.md');
|
||||
writeFileSync(p, review({ verdict: 'BLOCK', severities: ['BLOCKER'] }));
|
||||
const r = spawnSync(process.execPath, [GATE, '--json', p], { encoding: 'utf-8' });
|
||||
assert.equal(r.status, 1);
|
||||
const out = JSON.parse(r.stdout);
|
||||
assert.equal(out.verdict, 'BLOCK');
|
||||
assert.equal(out.exit, 1);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('verdict-gate — run through a symlink and a path with a space it still decides (realpath main guard)', () => {
|
||||
// The old `file://${argv[1]}` guard is a no-op under a symlink or a space:
|
||||
// the CLI would exit 0 without reading anything — a silent green.
|
||||
const dir = mkdtempSync(join(tmpdir(), 'verdict gate '));
|
||||
try {
|
||||
const link = join(dir, 'gate link.mjs');
|
||||
symlinkSync(GATE, link);
|
||||
const p = join(dir, 'my review.md');
|
||||
writeFileSync(p, review({ verdict: 'BLOCK', severities: ['BLOCKER'] }));
|
||||
const r = spawnSync(process.execPath, [link, p], { encoding: 'utf-8' });
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.match(r.stdout, /BLOCK/);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
|
@ -1,7 +1,14 @@
|
|||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, writeFileSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { validateReviewContent } from '../../lib/validators/review-validator.mjs';
|
||||
|
||||
const VALIDATOR = join(dirname(fileURLToPath(import.meta.url)), '..', '..', 'lib', 'validators', 'review-validator.mjs');
|
||||
|
||||
const GOOD_REVIEW = `---
|
||||
type: trekreview
|
||||
review_version: "1.0"
|
||||
|
|
@ -13,6 +20,7 @@ brief_path: .claude/projects/2026-04-30-jwt-auth/brief.md
|
|||
scope_sha_start: abc123
|
||||
scope_sha_end: def456
|
||||
reviewed_files_count: 7
|
||||
verdict: ALLOW
|
||||
findings:
|
||||
- 0123456789abcdef0123456789abcdef01234567
|
||||
- fedcba9876543210fedcba9876543210fedcba98
|
||||
|
|
@ -112,3 +120,88 @@ test('validateReview — empty findings array is acceptable (no findings = ALLOW
|
|||
const r = validateReviewContent(t);
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
});
|
||||
|
||||
// ---- K7: the verdict is required in review.md (ordre 5) ----------------------
|
||||
|
||||
const NO_VERDICT = GOOD_REVIEW.replace(/^verdict: .*\n/m, '');
|
||||
|
||||
test('validateReview — strict mode rejects a review.md without verdict (REVIEW_MISSING_VERDICT)', () => {
|
||||
assert.ok(!/^verdict:/m.test(NO_VERDICT), 'fixture control: the verdict line is gone');
|
||||
const r = validateReviewContent(NO_VERDICT, { strict: true });
|
||||
assert.equal(r.valid, false, 'a review without a verdict must not pass the write-time check');
|
||||
assert.ok(r.errors.find(e => e.code === 'REVIEW_MISSING_VERDICT'), JSON.stringify(r.errors));
|
||||
});
|
||||
|
||||
test('validateReview — soft (read-time) mode only warns on a missing verdict (N-1 window for 1.0 reviews)', () => {
|
||||
const r = validateReviewContent(NO_VERDICT, { strict: false });
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
assert.ok(r.warnings.find(w => w.code === 'REVIEW_MISSING_VERDICT'));
|
||||
});
|
||||
|
||||
test('validateReview — a verdict outside ALLOW|WARN|BLOCK is an error in both modes', () => {
|
||||
const t = GOOD_REVIEW.replace('verdict: ALLOW', 'verdict: PASS');
|
||||
for (const strict of [true, false]) {
|
||||
const r = validateReviewContent(t, { strict });
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.find(e => e.code === 'REVIEW_BAD_VERDICT'));
|
||||
}
|
||||
});
|
||||
|
||||
// ---- A BLOCKER is written as a plan step Handover 6 can append (ordre 5) ----
|
||||
|
||||
const BLOCKER_ID = '763d174e6c519fafbadcba5d1706708479e36e61';
|
||||
const WITH_BLOCKER = GOOD_REVIEW.replace('verdict: ALLOW', 'verdict: BLOCK') + `
|
||||
\`\`\`json
|
||||
{"verdict": "BLOCK", "counts": {"BLOCKER": 1, "MAJOR": 0, "MINOR": 0, "SUGGESTION": 0},
|
||||
"findings": [{"id": "${BLOCKER_ID}", "severity": "BLOCKER", "rule_key": "UNIMPLEMENTED_CRITERION", "file": "lib/login.mjs", "line": 23}]}
|
||||
\`\`\`
|
||||
`;
|
||||
const STEP = `
|
||||
## Remediation Steps
|
||||
|
||||
### Step R1: Return 401 on a bad password
|
||||
|
||||
- **Files:** \`lib/login.mjs\`
|
||||
- **Source finding:** \`${BLOCKER_ID}\`
|
||||
- **Verify:** re-run /trekreview → finding \`${BLOCKER_ID}\` absent
|
||||
`;
|
||||
|
||||
test('validateReview — strict: a BLOCKER without a numbered remediation step is an error', () => {
|
||||
const r = validateReviewContent(WITH_BLOCKER, { strict: true });
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.find(e => e.code === 'REVIEW_MISSING_REMEDIATION_STEP'), JSON.stringify(r.errors));
|
||||
});
|
||||
|
||||
test('validateReview — strict: a BLOCKER with its "Step R1 … Source finding" passes', () => {
|
||||
const t = WITH_BLOCKER.replace('## Remediation Summary', `${STEP}\n## Remediation Summary`);
|
||||
const r = validateReviewContent(t, { strict: true });
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
});
|
||||
|
||||
test('validateReview — soft: a missing remediation step only warns', () => {
|
||||
const r = validateReviewContent(WITH_BLOCKER, { strict: false });
|
||||
assert.equal(r.valid, true);
|
||||
assert.ok(r.warnings.find(w => w.code === 'REVIEW_MISSING_REMEDIATION_STEP'));
|
||||
});
|
||||
|
||||
test('validateReview — a clean review with zero findings validates with `findings: []` (a bare `findings:` does not)', () => {
|
||||
const ids = /findings:\n(?: - [0-9a-f]{40}\n)+/;
|
||||
assert.match(GOOD_REVIEW, ids, 'fixture control');
|
||||
assert.equal(validateReviewContent(GOOD_REVIEW.replace(ids, 'findings: []\n'), { strict: true }).valid, true);
|
||||
assert.equal(validateReviewContent(GOOD_REVIEW.replace(ids, 'findings:\n'), { strict: true }).valid, false);
|
||||
});
|
||||
|
||||
test('review-validator CLI — run through a symlink and a path with a space it still validates (exit 1, not a silent 0)', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'review validator '));
|
||||
try {
|
||||
const link = join(dir, 'validator link.mjs');
|
||||
symlinkSync(VALIDATOR, link);
|
||||
const p = join(dir, 'no verdict.md');
|
||||
writeFileSync(p, NO_VERDICT);
|
||||
const r = spawnSync(process.execPath, [link, p], { encoding: 'utf-8' });
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.match(r.stderr, /REVIEW_MISSING_VERDICT/);
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue