open/voyage
1
0
Fork 0
Code Issues Releases 13 Activity

fix(validate): a check that can never run is caught when the brief or plan is written

Browse source 
voyage-lekerepo end-to-end trial (2026-10-06), report points 2, 3, 4, 9:
11 of 13 brief criteria were NOT RUN at review with no warning in /trekbrief;
a plan `git grep` criterion passed validator, gate and three critic rounds,
then felled the run in Phase 7; the End-to-end field and Phase 7's text
search never met; a prose Verify left the test command undefined.

- criteria-runner: runnableCheck() screens a declared check without running
  it (same allowlist; an npm run script is not looked up at write time);
  /bin/bash tests/<script> allowed; the plan report carries end_to_end,
  verified when a passed criterion runs a step's End-to-end command (the
  older text form still counts).
- plan-validator 1.8: PLAN_VERIFICATION_NOT_RUNNABLE,
  PLAN_STEP_VERIFY_NOT_RUNNABLE, PLAN_END_TO_END_NOT_RUNNABLE,
  PLAN_END_TO_END_NOT_VERIFIED (the gate inherits them).
- brief-validator: BRIEF_SC_NOT_RUNNABLE warning; /trekbrief offers a
  tests/ script; brief-reviewer no longer proposes unrunnable commands.
- trekexecute: Phase 7 copies end_to_end from the runner; Verify's test
  command is its first backticked span.

Measured on the trial's own brief and plan: 11 warnings (the 11 NOT RUN),
and the plan stops on V5 git grep and the unlisted End-to-end command.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-10-08 06:42:33 +02:00
commit c56f852ab6
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
14 changed files with 422 additions and 38 deletions
Download patch file Download diff file Expand all files Collapse all files

21
lib/validators/brief-validator.mjs
View file

@ -11,6 +11,7 @@ import { fileURLToPath } from 'node:url';
import { parseDocument } from '../util/frontmatter.mjs';
import { issue, ok, fail } from '../util/result.mjs';
import { BASE_ALLOWED_MODELS } from './profile-validator.mjs';
import { parseSuccessCriteria, runnableCheck } from '../verification/criteria-runner.mjs';
export const BRIEF_REQUIRED_FRONTMATTER = ['type', 'brief_version', 'task', 'slug', 'research_topics', 'research_status'];
export const REVIEW_AS_BRIEF_REQUIRED_FRONTMATTER = ['type', 'task', 'slug', 'project_dir', 'findings'];
@ -262,6 +263,26 @@ export function validateBriefContent(text, opts = {}) {
}
}
// A criterion that opens with a command the review's criteria runner can
// never run is reported NOT RUN at /trekreview, long after anyone could fix
// it: 11 of 13 in the voyage-lekerepo trial (2026-10-06), and nothing in
// /trekbrief warned. A warning, not an error: a criterion may be one a human
// judges by reading, and that stays allowed. Prose-only criteria (no
// command) are not flagged.
if (fm.type !== 'trekreview') {
for (const c of parseSuccessCriteria(body)) {
if (!c.command) continue;
const r = runnableCheck(c, { cwd: opts.cwd });
if (r.runnable) continue;
warnings.push(issue(
'BRIEF_SC_NOT_RUNNABLE',
`Success criterion ${c.label} opens with \`${c.command}\`, which the review's criteria runner will never run (${r.reason})`,
'Wrap the check in a script under tests/ and write the criterion as `bash tests/<script>.sh` exits 0, ' +
'or name a test runner (`npm test`, `node --test <file>`); otherwise it is reported NOT RUN at review.',
));
}
}
if (typeof fm.brief_version === 'string' || typeof fm.brief_version === 'number') {
const m = String(fm.brief_version).match(/^(\d+)\.(\d+)$/);
if (!m) {