open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

fix(validate): a check that can never run is caught when the brief or plan is written

Browse source 
voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9:
11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief;
a plan `git grep` criterion passed validator, gate and three critic rounds,
then felled the run in Phase 7; the End-to-end field and Phase 7's text
search never met; a prose Verify left the test command undefined.

- criteria-runner: runnableCheck() screens a declared check without running
  it (same allowlist; an npm run script is not looked up at write time);
  /bin/bash tests/<script> allowed; the plan report carries end_to_end,
  verified when a passed criterion runs a step's End-to-end command (the
  older text form still counts).
- plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE,
  PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE,
  PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them).
- brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a
  tests/ script; brief-reviewer no longer proposes unrunnable commands.
- trekexecute: Phase 7 copies end_to_end from the runner; Verify's test
  command is its first backticked span.

Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN),
and the plan stops on V5 git grep and the unlisted End-to-end command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-08 06:42:33 +02:00
commit c56f852ab6
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 422 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

8
tests/validators/plan-validator-v18.test.mjs
View file

@ -31,7 +31,7 @@ test('a complete 1.8 architectural plan validates strict', () => {
});
test('1.8: a step without `Test first:` is rejected', () => {
const r = validatePlanContent(buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'node src/cli.mjs' }] }));
const r = validatePlanContent(buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'bash tests/e2e.sh' }] }));
assert.equal(r.valid, false);
assert.ok(codes(r).includes('PLAN_STEP_NO_TEST_FIRST'), codes(r).join(','));
assert.match(r.errors.find(e => e.code === 'PLAN_STEP_NO_TEST_FIRST').message, /Step 1/);
@ -57,10 +57,10 @@ test('1.8: `End-to-end:` without a backticked command does not count', () => {
test('1.8: K4 fields are optional, but an empty one is rejected', () => {
const withFields = buildPlan({ steps: [
{ interfaceOut: '`run1(): number` read by Step 2', reviewFocus: 'empty input — pinned by `tests/mod1.test.mjs`' },
{ interfaceIn: '`run1()` from Step 1', endToEnd: 'node src/cli.mjs' },
{ interfaceIn: '`run1()` from Step 1', endToEnd: 'bash tests/e2e.sh' },
] });
assert.deepEqual(validatePlanContent(withFields).errors, []);
const empty = buildPlan({ steps: [{ interfaceIn: '' }, { endToEnd: 'node src/cli.mjs' }] });
const empty = buildPlan({ steps: [{ interfaceIn: '' }, { endToEnd: 'bash tests/e2e.sh' }] });
assert.ok(codes(validatePlanContent(empty)).includes('PLAN_STEP_FIELD_EMPTY'));
});
@ -236,7 +236,7 @@ test('CLI --strict exits 1 on a 1.8 plan missing Test first; exits 0 through a s
const dir = mkdtempSync(join(tmpdir(), 'plan v18 '));
try {
const bad = join(dir, 'bad.md');
writeFileSync(bad, buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'node x' }] }));
writeFileSync(bad, buildPlan({ steps: [{ testFirst: false }, { endToEnd: 'bash tests/e2e.sh' }] }));
let code = 0;
try { execFileSync('node', [VALIDATOR, '--strict', '--json', bad], { stdio: 'pipe' }); } catch (e) { code = e.status; }
assert.equal(code, 1);