fix(execute): run the plan's Verification on the single-session path (D-03)
A trekplan's `## Verification` section is where the brief's success criteria land. Phase 7 opened with "**Skip for trekplans.**", and only the multi-session wave path (Phase 2.6 Step 3) ran master verification. A plan executed in ONE session therefore reported `completed` without ever running the criteria it was measured against — the executor's own belief was the only evidence. The check now exists as code, not as an instruction: - `lib/verification/criteria-runner.mjs` parses the criteria an artifact DECLARES (a plan's `## Verification`, a brief's `## Success Criteria`), runs each command, and returns a verdict built from exit codes. Fail-closed throughout: a placeholder, a prose-only criterion, or an unavailable screen is `unrunnable`/`blocked`, never `passed`. A plan with no `## Verification` section exits 1 — a plan that promises no end-to-end check cannot be reported as verified. - Every command is screened through the plugin's own PreToolUse denylist (`hooks/scripts/pre-bash-executor.mjs`) before it reaches a shell. Chose invoking that hook over its documented stdin protocol rather than copying its rules, because a command spawned from node never passes through the Bash tool and so the hook cannot fire by itself — this keeps exactly one denylist. - Phase 7 is now "Exit / verification check": session specs run the exit condition, trekplans run the criteria runner. Phase 4's entry-condition skip for trekplans stands — a plan carries no entry condition; the exit side is not symmetrical. - A failing criterion FELLS the run: `plan_verification.status != "passed"` forbids `result: completed`. That is clause 2 of the stop-signal contract, now enforced on the single-session path too. Red first: `tests/lib/criteria-runner.test.mjs` (26 tests) against two committed fixture plans, one of which declares a criterion that fails on purpose. The doc pin in `tests/lib/doc-consistency.test.mjs` guards the wiring — a capability no phase calls is the same defect wearing a lib/ file; verified red against the pre-fix Phase 7 (skip present, runner absent, no fell-the-run clause). Suite: 1108 (1106/0/2), up 27 from 1081. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
8d1669ef51
commit
e55ca9fc89
6 changed files with 756 additions and 6 deletions
|
|
@ -897,10 +897,17 @@ progress file.
|
|||
"iterations_remaining": 25,
|
||||
"entry_condition_checked": false,
|
||||
"exit_condition_checked": false,
|
||||
"plan_verification": null,
|
||||
"summary": null
|
||||
}
|
||||
```
|
||||
|
||||
**`plan_verification` (additive-optional):** written by Phase 7 when the input
|
||||
is a trekplan — `{status, summary, failed_criteria}`, where `status` is
|
||||
`"passed" | "failed" | "not-run"`. `null` for session specs (they have an exit
|
||||
condition instead) and on early stop. Unknown keys are tolerated by
|
||||
`progress-validator.mjs`, so a legacy progress file without it still validates.
|
||||
|
||||
**`iterations_remaining` (v1.7+, additive-optional):** the remaining
|
||||
recovery/retry budget against the global hard cap (unit = **iterations**, per the
|
||||
brief — token/cost budgeting is out of scope). Seeded to
|
||||
|
|
@ -1225,9 +1232,24 @@ Update progress: `steps.{N}.status = "passed"`, `steps.{N}.commit = {hash}`,
|
|||
If mode = `step N`: after completing step N (pass or fail), skip remaining steps
|
||||
and jump to Phase 8 (final report).
|
||||
|
||||
## Phase 7 — Exit condition check (session specs only)
|
||||
## Phase 7 — Exit / verification check
|
||||
|
||||
**Skip for trekplans.** Run only when all steps passed (not on early stop).
|
||||
**Runs only when all steps passed** (not on early stop). Which check runs
|
||||
depends on what was executed:
|
||||
|
||||
| Input | Check |
|
||||
|-------|-------|
|
||||
| session spec | the `## Exit Condition` checklist |
|
||||
| trekplan | the plan's `## Verification` criteria |
|
||||
|
||||
Phase 4's entry-condition skip for trekplans stands — a plan carries no entry
|
||||
condition. The exit side is not symmetrical: a trekplan's `## Verification`
|
||||
section is where the brief's success criteria land, and it used to run only on
|
||||
the multi-session wave path (Phase 2.6 Step 3). A single-session run therefore
|
||||
reported `completed` without ever executing the criteria it was measured
|
||||
against. It no longer does.
|
||||
|
||||
### Session specs — exit condition checklist
|
||||
|
||||
Run each exit condition command from the `## Exit Condition` checklist:
|
||||
|
||||
|
|
@ -1240,6 +1262,62 @@ Exit condition check:
|
|||
If all pass: `exit_condition_checked: true` in progress file.
|
||||
If any fail: record which failed. Include in final report.
|
||||
|
||||
### trekplans — run the plan's `## Verification`
|
||||
|
||||
Run the criteria runner over the plan file parsed in Phase 1
|
||||
(`{plan_path}`). It parses the `## Verification` section, screens every command
|
||||
through the plugin's own PreToolUse denylist before it reaches a shell, and
|
||||
runs what survives — in the FOREGROUND, never backgrounded. (A command spawned
|
||||
from node does not pass through the Bash tool, so that hook cannot fire by
|
||||
itself; the runner invokes it over its documented stdin protocol instead of
|
||||
carrying a second copy of the denylist.)
|
||||
|
||||
```bash
|
||||
# Resolve the plugin root ONCE. ${CLAUDE_PLUGIN_ROOT} is substituted in this
|
||||
# command's text but is EMPTY in the Bash tool's process env, and a bare
|
||||
# `node ${CLAUDE_PLUGIN_ROOT}/lib/…` then runs `node /lib/…`, which exits 1 —
|
||||
# indistinguishable from a criterion that failed.
|
||||
VOYAGE_ROOT="${CLAUDE_PLUGIN_ROOT:-}"
|
||||
case "$VOYAGE_ROOT" in
|
||||
/*) ;;
|
||||
*) VOYAGE_ROOT="$(ls -d "$HOME"/.claude/plugins/cache/*/voyage 2>/dev/null | head -1)" ;;
|
||||
esac
|
||||
if [ ! -f "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" ]; then
|
||||
echo "[voyage] plan verification could not run - plugin root unresolved (exit 2)."
|
||||
echo " NOT a pass: report it and never emit result: completed."
|
||||
exit 2
|
||||
fi
|
||||
|
||||
node "$VOYAGE_ROOT/lib/verification/criteria-runner.mjs" --plan "{plan_path}" --json
|
||||
```
|
||||
|
||||
The exit code is the verdict, and it is not advisory:
|
||||
|
||||
| Exit | Meaning | Executor |
|
||||
|------|---------|----------|
|
||||
| 0 | every criterion passed | `plan_verification.status = "passed"`; continue |
|
||||
| 1 | a criterion failed, was blocked, or could not run | `plan_verification.status = "failed"`; progress `status: "failed"` |
|
||||
| 2 | the runner itself could not run | `plan_verification.status = "not-run"`; treated exactly like exit 1 |
|
||||
|
||||
A plan with no `## Verification` section exits 1 with
|
||||
`error.code = NO_VERIFICATION_SECTION`. That is deliberate: a plan that
|
||||
promises no end-to-end check cannot be reported as verified.
|
||||
|
||||
Record in the progress file (additive-optional; unknown keys are tolerated by
|
||||
`progress-validator.mjs`, so a legacy progress file still validates):
|
||||
|
||||
- `plan_verification.status` — `"passed" | "failed" | "not-run"`
|
||||
- `plan_verification.summary` — `{total, passed, failed, blocked, unrunnable}`
|
||||
- `plan_verification.failed_criteria` — `[{label, command, exit_code}]`
|
||||
|
||||
**A failing criterion FELLS the run.** It is not "recorded and included in the
|
||||
final report": the executor MUST NOT emit `result: completed` while
|
||||
`plan_verification.status != "passed"`. This is clause 2 of the stop-signal
|
||||
contract below (Phase 7.5) — "the plan's Verify command(s) passed (exit 0)" —
|
||||
now enforced on the single-session path and not only on the multi-session one.
|
||||
Continue to Phase 7.5 either way: the manifest audit is independent, and a
|
||||
verification failure must not hide a manifest drift.
|
||||
|
||||
## Phase 7.5 — Manifest audit (independent)
|
||||
|
||||
**Runs for all modes except dry-run.** This is the last-line-of-defense
|
||||
|
|
@ -1301,7 +1379,11 @@ Record in progress file:
|
|||
> 0)** / lint clean / an explicit **DONE token emitted AFTER the Phase 7.5
|
||||
> audit ran** (never before, never in place of it). A transcript that merely
|
||||
> "feels done" is NOT a stop-signal — victory-declaration bias is the exact
|
||||
> failure mode this gate exists to defeat.
|
||||
> failure mode this gate exists to defeat. For a trekplan this clause is
|
||||
> **Phase 7's** `plan_verification.status == "passed"` — a real exit code
|
||||
> from `lib/verification/criteria-runner.mjs`, on the single-session path as
|
||||
> well as the multi-session one. `"failed"` or `"not-run"` OVERRIDES
|
||||
> `completed` to `failed`, exactly as the audit-override does.
|
||||
> 3. **Counter liveness (deterministic, defeats "never decremented")** — the gate
|
||||
> reconciles the budget counter against the ground-truth counters the Phase 7.5
|
||||
> audit already derives:
|
||||
|
|
@ -1524,6 +1606,12 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
|||
| 2 | {desc} | FAIL | 3 | — | — |
|
||||
| 3 | {desc} | — | 0 | — | — |
|
||||
|
||||
### Plan Verification (Phase 7, trekplans only)
|
||||
|
||||
- **Status:** {passed | failed | not-run | n/a}
|
||||
- **Criteria:** {passed}/{total} passed, {failed} failed, {blocked} blocked, {unrunnable} not run
|
||||
- **Failed criteria:** {label + command + exit code, one per line; empty when passed}
|
||||
|
||||
### Manifest Audit (Phase 7.5)
|
||||
|
||||
- **Status:** {pass | drift}
|
||||
|
|
@ -1540,8 +1628,8 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
|||
- Not reached: {N}
|
||||
- Blocked (sandbox): {N}
|
||||
|
||||
{if all passed + exit condition passed}:
|
||||
All steps completed. Exit condition: PASS.
|
||||
{if all passed + exit condition passed + plan verification passed}:
|
||||
All steps completed. Exit condition: PASS. Plan verification: PASS.
|
||||
|
||||
{if failed/stopped}:
|
||||
### Failure Details
|
||||
|
|
@ -1559,7 +1647,8 @@ To resume: /trekexecute --resume {path}
|
|||
```
|
||||
|
||||
**Result vocabulary (v1.7, strict):**
|
||||
- `completed` — all steps passed AND Phase 7.5 manifest audit passed
|
||||
- `completed` — all steps passed AND Phase 7.5 manifest audit passed AND
|
||||
(for trekplans) Phase 7 plan verification passed
|
||||
- `partial` — steps passed per executor but Phase 7.5 found drift, OR
|
||||
Phase 7.6 recovery incomplete
|
||||
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
|
||||
|
|
@ -1583,6 +1672,7 @@ To resume: /trekexecute --resume {path}
|
|||
"steps_blocked": 0,
|
||||
"failed_at_step": null,
|
||||
"exit_condition": "{pass | fail | skipped | n/a}",
|
||||
"plan_verification": "{passed | failed | not-run | n/a}",
|
||||
"manifest_audit": "{pass | drift | n/a}",
|
||||
"drift_details": [],
|
||||
"recovery_dispatched": false,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue