open/voyage
1
0
Fork 0
Code Issues Releases 5 Activity

fix(execute): run the plan's Verification on the single-session path (D-03)

Browse source 
A trekplan's `## Verification` section is where the brief's success criteria
land. Phase 7 opened with "**Skip for trekplans.**", and only the multi-session
wave path (Phase 2.6 Step 3) ran master verification. A plan executed in ONE
session therefore reported `completed` without ever running the criteria it was
measured against — the executor's own belief was the only evidence.

The check now exists as code, not as an instruction:

- `lib/verification/criteria-runner.mjs` parses the criteria an artifact
  DECLARES (a plan's `## Verification`, a brief's `## Success Criteria`), runs
  each command, and returns a verdict built from exit codes. Fail-closed
  throughout: a placeholder, a prose-only criterion, or an unavailable screen
  is `unrunnable`/`blocked`, never `passed`. A plan with no `## Verification`
  section exits 1 — a plan that promises no end-to-end check cannot be reported
  as verified.
- Every command is screened through the plugin's own PreToolUse denylist
  (`hooks/scripts/pre-bash-executor.mjs`) before it reaches a shell. Chose
  invoking that hook over its documented stdin protocol rather than copying its
  rules, because a command spawned from node never passes through the Bash tool
  and so the hook cannot fire by itself — this keeps exactly one denylist.
- Phase 7 is now "Exit / verification check": session specs run the exit
  condition, trekplans run the criteria runner. Phase 4's entry-condition skip
  for trekplans stands — a plan carries no entry condition; the exit side is
  not symmetrical.
- A failing criterion FELLS the run: `plan_verification.status != "passed"`
  forbids `result: completed`. That is clause 2 of the stop-signal contract,
  now enforced on the single-session path too.

Red first: `tests/lib/criteria-runner.test.mjs` (26 tests) against two committed
fixture plans, one of which declares a criterion that fails on purpose. The
doc pin in `tests/lib/doc-consistency.test.mjs` guards the wiring — a capability
no phase calls is the same defect wearing a lib/ file; verified red against the
pre-fix Phase 7 (skip present, runner absent, no fell-the-run clause).

Suite: 1108 (1106/0/2), up 27 from 1081.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 01:30:57 +02:00
commit e55ca9fc89
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
6 changed files with 756 additions and 6 deletions
Download patch file Download diff file Expand all files Collapse all files

102
commands/trekexecute.md
View file

@ -897,10 +897,17 @@ progress file.
"iterations_remaining": 25,
"entry_condition_checked": false,
"exit_condition_checked": false,
"plan_verification": null,
"summary": null
}
```
**`plan_verification` (additive-optional):** written by Phase 7 when the input
is a trekplan — `{status, summary, failed_criteria}`, where `status` is
`"passed" | "failed" | "not-run"`. `null` for session specs (they have an exit
condition instead) and on early stop. Unknown keys are tolerated by
`progress-validator.mjs`, so a legacy progress file without it still validates.
**`iterations_remaining` (v1.7+, additive-optional):** the remaining
recovery/retry budget against the global hard cap (unit = **iterations**, per the
brief — token/cost budgeting is out of scope). Seeded to
@ -1225,9 +1232,24 @@ Update progress: `steps.{N}.status = "passed"`, `steps.{N}.commit = {hash}`,
If mode = `step N`: after completing step N (pass or fail), skip remaining steps
and jump to Phase 8 (final report).
## Phase 7 — Exit condition check (session specs only)
## Phase 7 — Exit / verification check
**Skip for trekplans.** Run only when all steps passed (not on early stop).
**Runs only when all steps passed** (not on early stop). Which check runs
depends on what was executed:
| Input | Check |
|-------|-------|
| session spec | the `## Exit Condition` checklist |
| trekplan | the plan's `## Verification` criteria |
Phase 4's entry-condition skip for trekplans stands — a plan carries no entry
condition. The exit side is not symmetrical: a trekplan's `## Verification`
section is where the brief's success criteria land, and it used to run only on
the multi-session wave path (Phase 2.6 Step 3). A single-session run therefore
reported `completed` without ever executing the criteria it was measured
against. It no longer does.
### Session specs — exit condition checklist
Run each exit condition command from the `## Exit Condition` checklist:
@ -1240,6 +1262,62 @@ Exit condition check:
If all pass: `exit_condition_checked: true` in progress file.
If any fail: record which failed. Include in final report.
### trekplans — run the plan's `## Verification`
Run the criteria runner over the plan file parsed in Phase 1
(`{plan_path}`). It parses the `## Verification` section, screens every command
through the plugin's own PreToolUse denylist before it reaches a shell, and
runs what survives — in the FOREGROUND, never backgrounded. (A command spawned
from node does not pass through the Bash tool, so that hook cannot fire by
itself; the runner invokes it over its documented stdin protocol instead of
carrying a second copy of the denylist.)
```bash
# Resolve the plugin root ONCE. ${CLAUDE_PLUGIN_ROOT} is substituted in this
# command's text but is EMPTY in the Bash tool's process env, and a bare
# `node ${CLAUDE_PLUGIN_ROOT}/lib/…` then runs `node /lib/…`, which exits 1 —
# indistinguishable from a criterion that failed.
VOYAGE_ROOT="${CLAUDE_PLUGIN_ROOT:-}"
case "$VOYAGE_ROOT" in
/*) ;;
*) VOYAGE_ROOT="$(ls -d "$HOME"/.claude/plugins/cache/*/voyage 2>/dev/null | head -1)" ;;
esac
if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
echo "[voyage] plan verification could not run - plugin root unresolved (exit 2)."
echo " NOT a pass: report it and never emit result: completed."
exit 2
fi
node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" --plan "{plan_path}" --json
```
The exit code is the verdict, and it is not advisory:
| Exit | Meaning | Executor |
|------|---------|----------|
| 0 | every criterion passed | `plan_verification.status = "passed"`; continue |
| 1 | a criterion failed, was blocked, or could not run | `plan_verification.status = "failed"`; progress `status: "failed"` |
| 2 | the runner itself could not run | `plan_verification.status = "not-run"`; treated exactly like exit 1 |
A plan with no `## Verification` section exits 1 with
`error.code = NO_VERIFICATION_SECTION`. That is deliberate: a plan that
promises no end-to-end check cannot be reported as verified.
Record in the progress file (additive-optional; unknown keys are tolerated by
`progress-validator.mjs`, so a legacy progress file still validates):
- `plan_verification.status` — `"passed" | "failed" | "not-run"`
- `plan_verification.summary` — `{total, passed, failed, blocked, unrunnable}`
- `plan_verification.failed_criteria` — `[{label, command, exit_code}]`
**A failing criterion FELLS the run.** It is not "recorded and included in the
final report": the executor MUST NOT emit `result: completed` while
`plan_verification.status != "passed"`. This is clause 2 of the stop-signal
contract below (Phase 7.5) — "the plan's Verify command(s) passed (exit 0)" —
now enforced on the single-session path and not only on the multi-session one.
Continue to Phase 7.5 either way: the manifest audit is independent, and a
verification failure must not hide a manifest drift.
## Phase 7.5 — Manifest audit (independent)
**Runs for all modes except dry-run.** This is the last-line-of-defense
@ -1301,7 +1379,11 @@ Record in progress file:
> 0)** / lint clean / an explicit **DONE token emitted AFTER the Phase 7.5
> audit ran** (never before, never in place of it). A transcript that merely
> "feels done" is NOT a stop-signal — victory-declaration bias is the exact
> failure mode this gate exists to defeat.
> failure mode this gate exists to defeat. For a trekplan this clause is
> **Phase 7's** `plan_verification.status == "passed"` — a real exit code
> from `lib/verification/criteria-runner.mjs`, on the single-session path as
> well as the multi-session one. `"failed"` or `"not-run"` OVERRIDES
> `completed` to `failed`, exactly as the audit-override does.
> 3. **Counter liveness (deterministic, defeats "never decremented")** — the gate
> reconciles the budget counter against the ground-truth counters the Phase 7.5
> audit already derives:
@ -1524,6 +1606,12 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
| 2 | {desc} | FAIL | 3 | — | — |
| 3 | {desc} | — | 0 | — | — |
### Plan Verification (Phase 7, trekplans only)
- **Status:** {passed | failed | not-run | n/a}
- **Criteria:** {passed}/{total} passed, {failed} failed, {blocked} blocked, {unrunnable} not run
- **Failed criteria:** {label + command + exit code, one per line; empty when passed}
### Manifest Audit (Phase 7.5)
- **Status:** {pass | drift}
@ -1540,8 +1628,8 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
- Not reached: {N}
- Blocked (sandbox): {N}
{if all passed + exit condition passed}:
All steps completed. Exit condition: PASS.
{if all passed + exit condition passed + plan verification passed}:
All steps completed. Exit condition: PASS. Plan verification: PASS.
{if failed/stopped}:
### Failure Details
@ -1559,7 +1647,8 @@ To resume: /trekexecute --resume {path}
```
**Result vocabulary (v1.7, strict):**
- `completed` — all steps passed AND Phase 7.5 manifest audit passed
- `completed` — all steps passed AND Phase 7.5 manifest audit passed AND
(for trekplans) Phase 7 plan verification passed
- `partial` — steps passed per executor but Phase 7.5 found drift, OR
Phase 7.6 recovery incomplete
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
@ -1583,6 +1672,7 @@ To resume: /trekexecute --resume {path}
"steps_blocked": 0,
"failed_at_step": null,
"exit_condition": "{pass | fail | skipped | n/a}",
"plan_verification": "{passed | failed | not-run | n/a}",
"manifest_audit": "{pass | drift | n/a}",
"drift_details": [],
"recovery_dispatched": false,

308
lib/verification/criteria-runner.mjs Normal file
View file

@ -0,0 +1,308 @@
#!/usr/bin/env node
// lib/verification/criteria-runner.mjs
// Runs the falsifiable checks a pipeline artifact DECLARES, and reports a
// verdict built from exit codes rather than from a reader's impression.
//
// Two artifacts declare such checks:
// plan `## Verification` — `- [ ] `cmd` -> expected: ...` (V1..Vn)
// brief `## Success Criteria` — `- text: `cmd` ...` (SC1..SCn)
//
// Why this is code and not prose: `/trekexecute` must be able to FELL a
// single-session run on a criterion that does not hold, and `/trekreview` must
// hand `brief-conformance-reviewer` a real result instead of asking a
// Read/Glob/Grep agent to judge whether a command "passes". Neither is
// checkable while it lives only as an instruction.
//
// Fail-closed everywhere: a criterion that cannot run — placeholder text, no
// command, a screen that is unavailable — is NEVER reported as passed.
//
// Screening: every command is screened through the plugin's own PreToolUse
// denylist (hooks/scripts/pre-bash-executor.mjs) before it reaches a shell.
// Commands spawned from here do not pass through the Bash tool, so that hook
// would not otherwise fire; the hook is invoked over its documented stdin
// protocol rather than copied, so there is exactly one denylist.
//
// Usage:
// node lib/verification/criteria-runner.mjs --plan <plan.md> [--json] [--cwd D] [--timeout MS]
// node lib/verification/criteria-runner.mjs --brief <brief.md> [--json] [--cwd D] [--timeout MS]
// Exit: 0 = ok, 1 = a criterion failed / was blocked / could not run, 2 = usage
// or read error (deliberately distinct from "a criterion failed").
import { readFileSync } from 'node:fs';
import { resolve, dirname, join } from 'node:path';
import { fileURLToPath } from 'node:url';
import { spawnSync } from 'node:child_process';
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
export const PLAN_HEADING = '## Verification';
export const BRIEF_HEADING = '## Success Criteria';
export const DEFAULT_HOOK = join(REPO_ROOT, 'hooks', 'scripts', 'pre-bash-executor.mjs');
export const DEFAULT_TIMEOUT_MS = 900_000;
export const DEFAULT_MAX_OUTPUT = 4000;
// --- parsing ----------------------------------------------------------------
// The lines from `heading` up to the next `## ` heading (exclusive).
function sectionLines(markdown, heading) {
const lines = markdown.split('\n');
const start = lines.findIndex((l) => l.trim() === heading);
if (start === -1) return null;
let end = lines.length;
for (let i = start + 1; i < lines.length; i++) {
if (lines[i].startsWith('## ')) { end = i; break; }
}
return lines.slice(start + 1, end);
}
// `- [ ] text`, `- [x] text`, `- text`, `* text`, `1. text` -> text
const BULLET = /^\s*(?:[-*]|\d+\.)\s+(?:\[[ xX]\]\s+)?(.*)$/;
// The first backtick-delimited span on the line is the command by convention —
// both templates put it first and a second span holds the expected output.
function firstCommand(text) {
const m = text.match(/`([^`]+)`/);
if (!m) return { command: null, reason: 'no-command' };
const raw = m[1].trim();
// Template placeholders (`{exact command}`) are not commands.
if (raw === '' || /^\{.*\}$/.test(raw)) return { command: null, reason: 'placeholder' };
return { command: raw, reason: '' };
}
function parseSection(markdown, heading, prefix) {
const lines = sectionLines(markdown, heading);
if (lines === null) return [];
const criteria = [];
for (const line of lines) {
const bullet = line.match(BULLET);
if (!bullet) continue;
const text = bullet[1].trim();
if (text === '') continue;
const { command, reason } = firstCommand(text);
criteria.push({ label: `${prefix}${criteria.length + 1}`, text, command, reason });
}
return criteria;
}
/** Criteria declared in a plan's `## Verification` section, labelled V1..Vn. */
export function parsePlanVerification(markdown) {
return parseSection(markdown, PLAN_HEADING, 'V');
}
/** Criteria declared in a brief's `## Success Criteria` section, labelled SC1..SCn. */
export function parseSuccessCriteria(markdown) {
return parseSection(markdown, BRIEF_HEADING, 'SC');
}
// --- screening --------------------------------------------------------------
/**
* Screen one command through the plugin's PreToolUse denylist.
* Hook protocol: stdin JSON, exit 2 = block, exit 0 = allow. Any other exit
* means the screen did not render a verdict — that denies, it never allows.
*/
export function screenCommand(command, opts = {}) {
const hookPath = opts.hookPath ?? DEFAULT_HOOK;
const r = spawnSync(process.execPath, [hookPath], {
input: JSON.stringify({ tool_name: 'Bash', tool_input: { command } }),
encoding: 'utf8',
timeout: 30_000,
});
if (r.status === 0) return { allowed: true, rule: '' };
const first = String(r.stderr || '').split('\n')[0].trim();
if (r.status === 2) {
return { allowed: false, rule: first.replace(/^\[voyage\]\s*BLOCKED:\s*/, '') || 'blocked by the executor denylist' };
}
return {
allowed: false,
rule: `screen unavailable (${hookPath} exited ${r.status === null ? 'on signal' : r.status}) — denied, never assumed safe`,
};
}
// --- running ----------------------------------------------------------------
function defaultExec(command, { cwd, timeoutMs }) {
const r = spawnSync('/bin/sh', ['-c', command], { cwd, encoding: 'utf8', timeout: timeoutMs });
return { status: r.status, stdout: r.stdout ?? '', stderr: r.stderr ?? '' };
}
function cap(text, max) {
const t = String(text ?? '');
return t.length <= max ? t : `${t.slice(0, max)}\n… [truncated: ${t.length} chars total]`;
}
/**
* Run each criterion that has a command. Returns one result per criterion:
* passed the command exited 0
* failed the command exited non-zero (or was killed by the timeout)
* blocked the screen denied it — it never reached a shell
* unrunnable there is no command to run (placeholder or prose-only criterion)
*/
export function runCriteria(criteria, opts = {}) {
const exec = opts.exec ?? defaultExec;
const screen = opts.screen ?? screenCommand;
const cwd = opts.cwd ?? process.cwd();
const timeoutMs = opts.timeoutMs ?? DEFAULT_TIMEOUT_MS;
const maxOutput = opts.maxOutput ?? DEFAULT_MAX_OUTPUT;
const hookPath = opts.hookPath ?? DEFAULT_HOOK;
return criteria.map((c) => {
const base = { label: c.label, text: c.text, command: c.command };
if (!c.command) {
return { ...base, status: 'unrunnable', exitCode: null, output: `not runnable: ${c.reason || 'no command'}` };
}
const verdict = screen(c.command, { hookPath });
if (!verdict.allowed) {
return { ...base, status: 'blocked', exitCode: null, output: `blocked: ${verdict.rule}` };
}
const r = exec(c.command, { cwd, timeoutMs });
const output = cap(`${r.stdout ?? ''}${r.stderr ?? ''}`, maxOutput);
return {
...base,
status: r.status === 0 ? 'passed' : 'failed',
exitCode: r.status,
output,
};
});
}
/**
* The verdict. `requireCommand` distinguishes the two callers: a plan promises
* an exact command per criterion, so an unrunnable one is a defect in the plan;
* a brief may hold criteria a human judges by reading, so there an unrunnable
* criterion is reported and left to the reviewer.
*/
export function summarize(results, opts = {}) {
const requireCommand = opts.requireCommand ?? true;
const count = (s) => results.filter((r) => r.status === s).length;
const summary = {
total: results.length,
passed: count('passed'),
failed: count('failed'),
blocked: count('blocked'),
unrunnable: count('unrunnable'),
};
summary.ok =
summary.failed === 0 &&
summary.blocked === 0 &&
(!requireCommand || (summary.unrunnable === 0 && summary.total > 0));
return summary;
}
function report(kind, source, criteria, opts, error) {
const requireCommand = kind === 'plan';
const results = error ? [] : runCriteria(criteria, opts);
return {
kind,
source,
heading: kind === 'plan' ? PLAN_HEADING : BRIEF_HEADING,
results,
summary: summarize(results, { requireCommand }),
error: error ?? null,
};
}
function readOrThrow(path) {
try {
return readFileSync(path, 'utf8');
} catch (err) {
const e = new Error(`cannot read ${path}: ${err.message}`);
e.code = 'READ_ERROR';
throw e;
}
}
/**
* Run a plan's `## Verification` criteria. A plan with no such section is NOT
* ok: a plan that promises no end-to-end check cannot be verified, and that
* must read as red rather than as nothing to do.
*/
export function runPlanVerification(planPath, opts = {}) {
const md = readOrThrow(planPath);
if (sectionLines(md, PLAN_HEADING) === null) {
return report('plan', planPath, [], opts, {
code: 'NO_VERIFICATION_SECTION',
message: `${planPath} has no "${PLAN_HEADING}" section — nothing to verify, so the run cannot be called verified`,
});
}
return report('plan', planPath, parsePlanVerification(md), opts);
}
/**
* Run a brief's `## Success Criteria` commands. Used by /trekreview to hand the
* conformance reviewer a real result per criterion.
*/
export function runSuccessCriteriaChecks(briefPath, opts = {}) {
const md = readOrThrow(briefPath);
if (sectionLines(md, BRIEF_HEADING) === null) {
return report('brief', briefPath, [], opts, {
code: 'NO_SUCCESS_CRITERIA_SECTION',
message: `${briefPath} has no "${BRIEF_HEADING}" section`,
});
}
return report('brief', briefPath, parseSuccessCriteria(md), opts);
}
// --- rendering --------------------------------------------------------------
const MARK = { passed: 'PASS', failed: 'FAILED', blocked: 'BLOCKED', unrunnable: 'NOT RUN' };
export function render(rep) {
const out = [];
const s = rep.summary;
out.push(`criteria-runner: ${s.ok ? 'OK' : 'NOT OK'} — ${rep.source} ${rep.heading}`);
if (rep.error) out.push(` error: ${rep.error.code} — ${rep.error.message}`);
out.push(
` ${s.passed} passed · ${s.failed} failed · ${s.blocked} blocked · ${s.unrunnable} not run (of ${s.total})`
);
for (const r of rep.results) {
const code = r.exitCode === null ? '' : ` (exit ${r.exitCode})`;
out.push(` [${MARK[r.status]}] ${r.label}: \`${r.command ?? r.text}\`${code}`);
if (r.status !== 'passed' && r.output) {
for (const line of r.output.split('\n').slice(0, 20)) out.push(` ${line}`);
}
}
return out.join('\n');
}
// --- CLI --------------------------------------------------------------------
const USAGE =
'usage: criteria-runner.mjs (--plan <plan.md> | --brief <brief.md>) [--json] [--cwd <dir>] [--timeout <ms>]';
export function main(argv) {
let mode = null;
let path = null;
let json = false;
const opts = {};
for (let i = 0; i < argv.length; i++) {
const a = argv[i];
if ((a === '--plan' || a === '--brief') && argv[i + 1]) {
if (mode) { process.stderr.write(`criteria-runner: one mode at a time\n${USAGE}\n`); return 2; }
mode = a.slice(2);
path = argv[++i];
} else if (a === '--json') json = true;
else if (a === '--cwd' && argv[i + 1]) opts.cwd = resolve(argv[++i]);
else if (a === '--timeout' && argv[i + 1]) opts.timeoutMs = Number(argv[++i]);
else { process.stderr.write(`criteria-runner: unknown argument ${a}\n${USAGE}\n`); return 2; }
}
if (!mode) { process.stderr.write(`criteria-runner: no artifact given\n${USAGE}\n`); return 2; }
if (opts.timeoutMs !== undefined && !Number.isFinite(opts.timeoutMs)) {
process.stderr.write(`criteria-runner: --timeout must be a number\n${USAGE}\n`);
return 2;
}
let rep;
try {
rep = mode === 'plan' ? runPlanVerification(path, opts) : runSuccessCriteriaChecks(path, opts);
} catch (err) {
process.stderr.write(`criteria-runner: ${err.message}\n`);
return 2;
}
process.stdout.write((json ? JSON.stringify(rep, null, 2) : render(rep)) + '\n');
return rep.summary.ok ? 0 : 1;
}
if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
process.exitCode = main(process.argv.slice(2));
}

25
tests/fixtures/plan-verification-fails.md vendored Normal file
View file

@ -0,0 +1,25 @@
---
task: criteria-runner fixture — one verification criterion fails
slug: criteria-runner-fixture-fail
---
# Plan: criteria-runner fixture (one criterion FAILS on purpose)
Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`. It is the
falsifying case for D-03: a plan whose declared success criterion does not
hold must fell the single-session run instead of being noted and passed over.
## Steps
### Step 1 — nothing
Fixtures have no steps worth running.
## Verification
- [ ] `true` -> expected: exit 0
- [ ] `false` -> expected: exit 0 (FAILS on purpose — exit 1)
## Estimated Scope
- **Files to modify:** 0

24
tests/fixtures/plan-verification-passes.md vendored Normal file
View file

@ -0,0 +1,24 @@
---
task: criteria-runner fixture — every verification criterion passes
slug: criteria-runner-fixture-pass
---
# Plan: criteria-runner fixture (all criteria pass)
Fixture only. Consumed by `tests/lib/criteria-runner.test.mjs`; never executed
by the pipeline. The commands are deliberately trivial and side-effect free.
## Steps
### Step 1 — nothing
Fixtures have no steps worth running.
## Verification
- [ ] `true` -> expected: exit 0
- [ ] `printf ok` -> expected: `ok` on stdout, exit 0
## Estimated Scope
- **Files to modify:** 0

280
tests/lib/criteria-runner.test.mjs Normal file
View file

@ -0,0 +1,280 @@
// tests/lib/criteria-runner.test.mjs
// The criteria runner is what makes a declared check a RUN check: it parses the
// falsifiable criteria a plan (`## Verification`) or a brief (`## Success
// Criteria`) declares, screens each command through the executor's own
// PreToolUse denylist, runs it, and returns a verdict built from exit codes.
//
// Fail-closed is the whole point: a criterion that cannot run (placeholder, no
// command, screen unavailable) must never read as "passed".
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import { spawnSync } from 'node:child_process';
import { join, dirname } from 'node:path';
import { fileURLToPath } from 'node:url';
import { mkdtempSync, writeFileSync } from 'node:fs';
import { tmpdir } from 'node:os';
import {
parsePlanVerification,
parseSuccessCriteria,
screenCommand,
runCriteria,
summarize,
runPlanVerification,
render,
} from '../../lib/verification/criteria-runner.mjs';
const HERE = dirname(fileURLToPath(import.meta.url));
const ROOT = join(HERE, '..', '..');
const CLI = join(ROOT, 'lib', 'verification', 'criteria-runner.mjs');
const FIX = join(ROOT, 'tests', 'fixtures');
const HOOK = join(ROOT, 'hooks', 'scripts', 'pre-bash-executor.mjs');
// An exec double: maps a command string to {status, stdout, stderr}.
function execDouble(table) {
const calls = [];
const exec = (command) => {
calls.push(command);
return table[command] ?? { status: 127, stdout: '', stderr: 'not in table' };
};
exec.calls = calls;
return exec;
}
// A screen double that allows everything, so exec behaviour can be tested alone.
const allowAll = () => ({ allowed: true, rule: '' });
// --- parsing ---------------------------------------------------------------
test('parsePlanVerification: checkbox bullets become V1..Vn with their command', () => {
const md = [
'# Plan',
'',
'## Verification',
'',
'- [ ] `npm test` -> expected: exit 0',
'- [x] `node --test tests/lib/x.test.mjs` -> expected: 3 passing',
'',
'## Estimated Scope',
'',
'- [ ] `not-a-criterion` (outside the section)',
].join('\n');
const criteria = parsePlanVerification(md);
assert.equal(criteria.length, 2);
assert.deepEqual(criteria.map((c) => c.label), ['V1', 'V2']);
assert.equal(criteria[0].command, 'npm test');
assert.equal(criteria[1].command, 'node --test tests/lib/x.test.mjs');
assert.match(criteria[0].text, /expected: exit 0/);
});
test('parsePlanVerification: a template placeholder is unrunnable, not a command', () => {
const md = '## Verification\n\n- [ ] `{exact command}` -> expected: `{exact output}`\n';
const criteria = parsePlanVerification(md);
assert.equal(criteria.length, 1);
assert.equal(criteria[0].command, null);
assert.equal(criteria[0].reason, 'placeholder');
});
test('parsePlanVerification: a missing section yields no criteria', () => {
assert.deepEqual(parsePlanVerification('# Plan\n\n## Steps\n\n- do a thing\n'), []);
});
test('parseSuccessCriteria: bullets become SC1..SCn and take the FIRST backticked span', () => {
const md = [
'## Success Criteria',
'',
'- All existing tests pass: `npm test` exits 0',
'- Endpoint returns 200: `curl -s localhost:3000/health` -> `"ok"`',
'- No new runtime dependencies are introduced',
'',
'## Research Plan',
].join('\n');
const criteria = parseSuccessCriteria(md);
assert.deepEqual(criteria.map((c) => c.label), ['SC1', 'SC2', 'SC3']);
assert.equal(criteria[0].command, 'npm test');
assert.equal(criteria[1].command, 'curl -s localhost:3000/health');
assert.equal(criteria[2].command, null);
assert.equal(criteria[2].reason, 'no-command');
});
// --- screening -------------------------------------------------------------
test('screenCommand: the real executor denylist blocks a catastrophic command', () => {
const verdict = screenCommand('rm -rf ~', { hookPath: HOOK });
assert.equal(verdict.allowed, false);
assert.match(verdict.rule, /rm -rf|destruction/i);
});
test('screenCommand: an ordinary command passes the real denylist', () => {
assert.equal(screenCommand('npm test', { hookPath: HOOK }).allowed, true);
});
test('screenCommand: an unavailable screen denies (fail-closed, never silently allows)', () => {
const verdict = screenCommand('npm test', { hookPath: join(ROOT, 'hooks', 'scripts', 'no-such-hook.mjs') });
assert.equal(verdict.allowed, false);
assert.match(verdict.rule, /screen unavailable/i);
});
// --- running ---------------------------------------------------------------
test('runCriteria: exit 0 passes, a non-zero exit fails, output is captured', () => {
const criteria = parsePlanVerification(
'## Verification\n\n- [ ] `good`\n- [ ] `bad`\n'
);
const exec = execDouble({
good: { status: 0, stdout: 'all green\n', stderr: '' },
bad: { status: 1, stdout: '', stderr: '1 failing\n' },
});
const results = runCriteria(criteria, { exec, screen: allowAll });
assert.deepEqual(results.map((r) => r.status), ['passed', 'failed']);
assert.equal(results[0].exitCode, 0);
assert.equal(results[1].exitCode, 1);
assert.match(results[1].output, /1 failing/);
assert.deepEqual(exec.calls, ['good', 'bad']);
});
test('runCriteria: a blocked command is marked blocked and is NEVER executed', () => {
const criteria = parsePlanVerification('## Verification\n\n- [ ] `rm -rf ~`\n');
const exec = execDouble({});
const results = runCriteria(criteria, {
exec,
screen: () => ({ allowed: false, rule: 'Filesystem root/home destruction' }),
});
assert.equal(results[0].status, 'blocked');
assert.equal(results[0].exitCode, null);
assert.match(results[0].output, /Filesystem root\/home destruction/);
assert.deepEqual(exec.calls, [], 'a blocked command must not reach the shell');
});
test('runCriteria: a criterion with no command is unrunnable, not passed', () => {
const criteria = parseSuccessCriteria('## Success Criteria\n\n- No new dependencies\n');
const results = runCriteria(criteria, { exec: execDouble({}), screen: allowAll });
assert.equal(results[0].status, 'unrunnable');
assert.equal(results[0].exitCode, null);
});
test('runCriteria: output is capped so a verbose command cannot flood a prompt', () => {
const criteria = parsePlanVerification('## Verification\n\n- [ ] `loud`\n');
const exec = execDouble({ loud: { status: 0, stdout: 'x'.repeat(10000), stderr: '' } });
const results = runCriteria(criteria, { exec, screen: allowAll, maxOutput: 200 });
assert.ok(results[0].output.length < 400, `capped, got ${results[0].output.length}`);
assert.match(results[0].output, /truncated/);
});
// --- the verdict -----------------------------------------------------------
test('summarize: one failing criterion makes the run NOT ok', () => {
const results = [
{ status: 'passed' }, { status: 'failed' }, { status: 'passed' },
];
const s = summarize(results, { requireCommand: true });
assert.equal(s.ok, false);
assert.equal(s.failed, 1);
assert.equal(s.passed, 2);
assert.equal(s.total, 3);
});
test('summarize: in plan mode an unrunnable criterion makes the run NOT ok', () => {
const s = summarize([{ status: 'passed' }, { status: 'unrunnable' }], { requireCommand: true });
assert.equal(s.ok, false);
assert.equal(s.unrunnable, 1);
});
test('summarize: in brief mode an unrunnable criterion is reported, not failed', () => {
const s = summarize([{ status: 'passed' }, { status: 'unrunnable' }], { requireCommand: false });
assert.equal(s.ok, true);
assert.equal(s.unrunnable, 1);
});
test('summarize: a blocked criterion is never ok, in either mode', () => {
for (const requireCommand of [true, false]) {
assert.equal(summarize([{ status: 'blocked' }], { requireCommand }).ok, false);
}
});
test('summarize: zero criteria is NOT ok in plan mode (a plan that promises nothing)', () => {
assert.equal(summarize([], { requireCommand: true }).ok, false);
});
// --- the single-session path, end to end -----------------------------------
test('runPlanVerification: a plan whose success criterion FAILS fells the run', () => {
const report = runPlanVerification(join(FIX, 'plan-verification-fails.md'));
assert.equal(report.kind, 'plan');
assert.equal(report.summary.ok, false);
assert.equal(report.summary.failed, 1);
const failed = report.results.find((r) => r.status === 'failed');
assert.ok(failed, 'the failing criterion is reported by label');
assert.match(failed.label, /^V\d+$/);
});
test('runPlanVerification: a plan whose criteria all pass is ok', () => {
const report = runPlanVerification(join(FIX, 'plan-verification-passes.md'));
assert.equal(report.summary.ok, true);
assert.equal(report.summary.failed, 0);
assert.equal(report.summary.total, 2);
});
test('runPlanVerification: a plan with no ## Verification section is NOT ok', () => {
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
const p = join(dir, 'plan.md');
writeFileSync(p, '# Plan\n\n## Steps\n\n- do a thing\n');
const report = runPlanVerification(p);
assert.equal(report.summary.ok, false);
assert.equal(report.error.code, 'NO_VERIFICATION_SECTION');
});
test('render: the report names every non-passing criterion and its exit code', () => {
const report = runPlanVerification(join(FIX, 'plan-verification-fails.md'));
const text = render(report);
assert.match(text, /FAILED/);
assert.match(text, /exit 1/);
});
// --- the CLI ---------------------------------------------------------------
function cli(args) {
return spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8', cwd: ROOT });
}
test('CLI: --plan exits 1 when a criterion fails', () => {
const r = cli(['--plan', join(FIX, 'plan-verification-fails.md')]);
assert.equal(r.status, 1, r.stderr);
assert.match(r.stdout, /FAILED/);
});
test('CLI: --plan exits 0 when every criterion passes', () => {
const r = cli(['--plan', join(FIX, 'plan-verification-passes.md')]);
assert.equal(r.status, 0, r.stderr + r.stdout);
});
test('CLI: --json emits a parseable report with the summary', () => {
const r = cli(['--plan', join(FIX, 'plan-verification-fails.md'), '--json']);
assert.equal(r.status, 1);
const out = JSON.parse(r.stdout);
assert.equal(out.summary.ok, false);
assert.equal(out.results.length, out.summary.total);
});
test('CLI: a missing file exits 2 — a read error is not a failed criterion', () => {
const r = cli(['--plan', join(FIX, 'no-such-plan.md')]);
assert.equal(r.status, 2);
assert.match(r.stderr, /criteria-runner/);
});
test('CLI: an unknown argument exits 2 with usage', () => {
const r = cli(['--nope']);
assert.equal(r.status, 2);
assert.match(r.stderr, /usage/);
});
test('CLI: no mode flag exits 2 — it never guesses which artifact it was given', () => {
const r = cli([]);
assert.equal(r.status, 2);
assert.match(r.stderr, /usage/);
});

23
tests/lib/doc-consistency.test.mjs
View file

@ -1829,3 +1829,26 @@ test('D-07: trekresearch launch rules inject the resolved model instead of a bla
);
assert.match(rules, /phase_signal_result\.model/, 'the launch rules must name the resolved model the spawn sites inject');
});
// End-state defect D-03: a trekplan's `## Verification` is where the brief's success
// criteria land. Phase 7 used to say "Skip for trekplans", so on the single-session path
// the criteria were never run and `completed` meant "the executor believed it". The
// behaviour lives in lib/verification/criteria-runner.mjs; this pin guards the WIRING —
// a capability no phase calls is the same defect wearing a lib/ file. Fix the SOURCE.
test('D-03: trekexecute Phase 7 runs the plan Verification on the single-session path', () => {
const t = read('commands/trekexecute.md');
const phase7 = (t.split('\n## Phase 7 — ')[1] || '').split('\n## ')[0];
assert.ok(phase7.length > 0, 'trekexecute.md must still carry a Phase 7 section');
assert.ok(
!/^\*\*Skip for trekplans\.\*\*/m.test(phase7),
'Phase 7 may no longer skip trekplans — that skip IS defect D-03',
);
assert.match(
phase7, /criteria-runner\.mjs" --plan/,
'Phase 7 must invoke the criteria runner over the plan file, not describe the check in prose',
);
assert.match(
phase7, /MUST NOT emit `result: completed`/,
'a failing criterion must fell the run, not be noted in the final report',
);
});