feat(execute): one fresh helper per step, a controller on the diff, proof by command
/trekexecute no longer builds steps itself. Each plan step is built by one fresh helper (new agent step-implementer: opus, effort high, no Agent tool) and judged by one controller (a general helper on opus) that reads the diff from a recorded start_sha and gives two verdicts, spec and quality. Fix rounds go up, maximum 5: three fixes by a fresh helper, then back to the plan, then to the operator. No cheaper-model ladder. Proof is a command, never the model's word: - lib/verification/red-proof.mjs: the step's test must fail at the start point and pass at the result (exit 0/1/2; 2 is never green). Reproduces the02243c6red-first measurement (6 of 55) through the shim fixture. - lib/verification/test-integrity.mjs: the floor guard on the diff (removed asserts, skipped/deleted tests, muted checks, stubs, loosened thresholds). Over the last 150 commits: 149 clean, 1 real finding. - lib/verification/done-line.mjs: DONE step N only when the test command exits 0. Test first is required from plan_version 1.8. The test lock (K8) is used through its CLI and reported as not-locked while it is missing. A run stops only on a closed list of four classes; every other call goes into a decision log with its cost if wrong. Without a passing run-the-thing criterion the result is not verified end to end, never green. Handover 5 gains optional per-step start_sha, report_path, red_proof, test_lock, fix_rounds (0-5) and run-level decisions and end_to_end; old progress files still validate and resume. No isolation on the helper: measured, a worktree-isolated subagent branches from the default branch (02f320f), not the executor's HEAD (2c89a6b), so it would miss the earlier steps. Borrowings credited in commands/trekexecute.md ## Credits (Superpowers, ai-native-sdlc, Kiro, GitHub Spec Kit, agent-skills, Cherny). Tests: 1411 -> 1457 (1455 pass, 0 fail, 2 skipped). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
2c89a6b580
commit
abc3083c1e
28 changed files with 2052 additions and 103 deletions
|
|
@ -29,7 +29,7 @@ Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execut
|
||||||
| `/trekbrief` | Brief — interactive interview produces a task brief with explicit research plan; ends with the operator approving `## Intent` + `## Goal` (`--approve <dir>` does only that, then writes the trekbrief record); optionally orchestrates the pipeline | session |
|
| `/trekbrief` | Brief — interactive interview produces a task brief with explicit research plan; ends with the operator approving `## Intent` + `## Goal` (`--approve <dir>` does only that, then writes the trekbrief record); optionally orchestrates the pipeline | session |
|
||||||
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default) | session |
|
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default) | session |
|
||||||
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`; halts unless the brief's intent is approved (hash-bound, stale on edit). Auto-discovers `architecture/overview.md` if present | session |
|
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`; halts unless the brief's intent is approved (hash-bound, stale on edit). Auto-discovers `architecture/overview.md` if present | session |
|
||||||
| `/trekexecute` | Execute — disciplined plan/session-spec executor with failure recovery | session |
|
| `/trekexecute` | Execute — disciplined plan/session-spec executor: one fresh `step-implementer` helper per plan step, one controller that reads the diff (spec + quality verdicts), proof of red / floor guard / done line as commands, fix rounds that go up (max 5: helper → plan → operator), closed stop list + decision log in the final report | session |
|
||||||
| `/trekreview` | Review — independent post-hoc review of delivered code against the brief. Produces `review.md` with severity-tagged findings (Handover 6) | session |
|
| `/trekreview` | Review — independent post-hoc review of delivered code against the brief. Produces `review.md` with severity-tagged findings (Handover 6) | session |
|
||||||
| `/trekcontinue` | Continue — resumes the next session of a multi-session voyage project. Reads `.session-state.local.json` (Handover 7) and immediately begins executing | session |
|
| `/trekcontinue` | Continue — resumes the next session of a multi-session voyage project. Reads `.session-state.local.json` (Handover 7) and immediately begins executing | session |
|
||||||
| `/trekendsession` | End-session — mark the current session complete and write session-state pointing at the next session. Helper for informal multi-session flows | session |
|
| `/trekendsession` | End-session — mark the current session complete and write session-state pointing at the next session. Helper for informal multi-session flows | session |
|
||||||
|
|
@ -74,15 +74,16 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
|
||||||
| plan-critic | opus | Adversarial plan review (10 dimensions) |
|
| plan-critic | opus | Adversarial plan review (10 dimensions) |
|
||||||
| scope-guardian | opus | Scope alignment (creep + gaps) |
|
| scope-guardian | opus | Scope alignment (creep + gaps) |
|
||||||
| session-decomposer | opus | Splits plans into headless sessions with dependency graph |
|
| session-decomposer | opus | Splits plans into headless sessions with dependency graph |
|
||||||
|
| step-implementer | opus | `/trekexecute` step helper — one fresh helper per plan step: test first, proof of red, test lock, then exactly the step's Changes; ran / saw / did-NOT-check report (no Agent tool) |
|
||||||
| synthesis-agent | opus | Distills Phase-5/7 exploration outputs into a findings digest (NW3 PoC — **dormant**, not wired; delegating Phase 7 alone yields Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`) |
|
| synthesis-agent | opus | Distills Phase-5/7 exploration outputs into a findings digest (NW3 PoC — **dormant**, not wired; delegating Phase 7 alone yields Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`) |
|
||||||
| docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) |
|
| docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) |
|
||||||
| community-researcher | opus | Community experience: issues, blogs, discussions |
|
| community-researcher | opus | Community experience: issues, blogs, discussions |
|
||||||
| security-researcher | opus | CVEs, audit history, supply chain risks |
|
| security-researcher | opus | CVEs, audit history, supply chain risks |
|
||||||
| contrarian-researcher | opus | Counter-evidence, overlooked alternatives |
|
| contrarian-researcher | opus | Counter-evidence, overlooked alternatives |
|
||||||
|
|
||||||
> **Inventory (S33 reconcile, updated v5.10.1).** 23 agent files = **20 spawnable** (one, `synthesis-agent`, ships **dormant** — Δ≈0, wired to nothing) **+ 3 orchestrator reference docs** (`planning-/research-/review-orchestrator` document the inline `/trek*` workflow, not spawnable capabilities). All 23 stay `model: opus` (operator pin `40d8742`); the glue/mechanical/retrieval/dormant roles were reconsidered for a sonnet downgrade and **kept opus** — decision record: `docs/voyage-vs-cc-balance-analysis.md` §10.
|
> **Inventory (S33 reconcile, updated wave 1).** 24 agent files = **21 spawnable** (one, `synthesis-agent`, ships **dormant** — Δ≈0, wired to nothing) **+ 3 orchestrator reference docs** (`planning-/research-/review-orchestrator` document the inline `/trek*` workflow, not spawnable capabilities). All 24 stay `model: opus` (operator pin `40d8742`); the glue/mechanical/retrieval/dormant roles were reconsidered for a sonnet downgrade and **kept opus** — decision record: `docs/voyage-vs-cc-balance-analysis.md` §10.
|
||||||
|
|
||||||
> **Model & effort.** On the Anthropic API, `opus` = Opus 5.5 and `sonnet` = Sonnet 5.5 (both default reasoning effort `medium`); `fable` = Fable 5.1 (the most capable model in Claude Code; default `high`, so xhigh must be chosen explicitly). An agent without `effort:` inherits the session level. Eight agents pin native per-spawn `effort: high` and the rest follow the session; none pins below `high` — a different axis from brief `phase_signals.effort` (orchestration shape: which agents/passes run). Per-agent table + axes → `docs/profiles.md` §Model & effort axes.
|
> **Model & effort.** On the Anthropic API, `opus` = Opus 5.5 and `sonnet` = Sonnet 5.5 (both default reasoning effort `medium`); `fable` = Fable 5.1 (the most capable model in Claude Code; default `high`, so xhigh must be chosen explicitly). An agent without `effort:` inherits the session level. Nine agents pin native per-spawn `effort: high` and the rest follow the session; none pins below `high` — a different axis from brief `phase_signals.effort` (orchestration shape: which agents/passes run). Per-agent table + axes → `docs/profiles.md` §Model & effort axes.
|
||||||
|
|
||||||
## Reference docs (read on demand)
|
## Reference docs (read on demand)
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,7 @@ Changes to this plugin must preserve:
|
||||||
| `commands/trekresearch.md` | The `/trekresearch` slash command — research orchestration |
|
| `commands/trekresearch.md` | The `/trekresearch` slash command — research orchestration |
|
||||||
| `commands/trekplan.md` | The `/trekplan` slash command — planning orchestration |
|
| `commands/trekplan.md` | The `/trekplan` slash command — planning orchestration |
|
||||||
| `commands/trekexecute.md` | The `/trekexecute` slash command — execution orchestration |
|
| `commands/trekexecute.md` | The `/trekexecute` slash command — execution orchestration |
|
||||||
| `agents/*.md` | 19 specialized agents for research, exploration, review, and orchestration |
|
| `agents/*.md` | 24 agent files (21 spawnable + 3 orchestrator reference docs) for research, exploration, review, execution, and orchestration |
|
||||||
| `templates/plan-template.md` | Structured plan output format |
|
| `templates/plan-template.md` | Structured plan output format |
|
||||||
| `templates/research-brief-template.md` | Research brief format with triangulation and confidence |
|
| `templates/research-brief-template.md` | Research brief format with triangulation and confidence |
|
||||||
| `templates/spec-template.md` | Spec file format |
|
| `templates/spec-template.md` | Spec file format |
|
||||||
|
|
|
||||||
40
README.md
40
README.md
|
|
@ -296,7 +296,25 @@ Exploration uses 6–8 agents in parallel (architecture-mapper, dependency-trace
|
||||||
|
|
||||||
Reads a plan from `/trekplan` and implements it with strict discipline. No guessing, no improvising — follows the plan exactly.
|
Reads a plan from `/trekplan` and implements it with strict discipline. No guessing, no improvising — follows the plan exactly.
|
||||||
|
|
||||||
Per step: apply Changes exactly as written → run Verify (exit code is truth) → manifest audit (expected paths, forbidden paths, commit pattern) → follow the plan's failure clause if anything fails (revert / retry / skip / escalate) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery.
|
Per step: record the start point → **one fresh helper** (`step-implementer`) writes the step's test first, proves it red, locks it, then applies Changes exactly as written → the executor proves it by command: **red at the start point, green at the result** (`red-proof.mjs`), Verify (exit code is truth), manifest audit, and the **floor guard** on the diff (`test-integrity.mjs`: removed asserts, skipped or deleted tests, muted checks, stubs, loosened thresholds) → **one controller** reads the diff — never the helper's summary — and gives two verdicts, *spec* and *quality* → a script writes the step's **done line** only when the test command exits 0 (`done-line.mjs`) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery. A plan with no passing run-the-thing criterion ends **not verified end to end**, never green.
|
||||||
|
|
||||||
|
### The step loop
|
||||||
|
|
||||||
|
| Part | What it does | Why |
|
||||||
|
|------|--------------|-----|
|
||||||
|
| Fresh helper per step | `voyage:step-implementer` (opus, effort high, no Agent tool) builds one step and writes a report: **ran · saw · did NOT check** | Each step starts from the repository and the plan, not from a long context |
|
||||||
|
| Start point | `start_sha` recorded before the step; every check reads `git diff {start_sha}` | The review package is the diff, not a story |
|
||||||
|
| Proof of red | the step's test must fail at the start point and pass at the result — run by `red-proof.mjs`, not claimed | A test that passes without the change proves nothing |
|
||||||
|
| Test lock | the helper locks its test files after the red proof (`test-lock.mjs`, when installed; otherwise "not-locked" is reported) | The code moves to meet the test, never the reverse |
|
||||||
|
| Floor guard | `test-integrity.mjs` reads the diff for weakened tests; exit 2 ("could not run") is never green | Cheating the tests is caught by a command |
|
||||||
|
| Controller | one general helper, two verdicts (spec, quality), re-checks only what a fix round changed | An independent judge of the diff |
|
||||||
|
| Done line | `done-line.mjs` appends `DONE step N` only when the test command exits 0 | "Done" is an exit code |
|
||||||
|
|
||||||
|
The helper works in your working tree, not in a worktree of its own: measured 2026-10-06, a worktree-isolated subagent branches from the default branch rather than your `HEAD`, so it would not see the earlier steps (only the operator setting `worktree.baseRef: "head"` changes that).
|
||||||
|
|
||||||
|
### Unattended: what stops a run
|
||||||
|
|
||||||
|
A run stops only on a **closed list**: something that cannot be undone · something security-sensitive · something outside the working copy (merge to main, push, publish) · a plan so broken that everything after it is guesswork. Everything else the executor decides itself and logs in the final report's **Decisions taken** table, each with what it costs if it is wrong.
|
||||||
|
|
||||||
### Modes
|
### Modes
|
||||||
|
|
||||||
|
|
@ -337,11 +355,11 @@ Use `--fg` to run every step sequentially in the current session instead, even w
|
||||||
|
|
||||||
### Failure recovery
|
### Failure recovery
|
||||||
|
|
||||||
- **3-attempt retry cap** — retries twice, then stops (never loops forever)
|
- **Fix rounds go up, maximum 5** — rounds 1–3: a fresh helper fixes exactly the controller's findings; after three failed fixes the question is the architecture: round 4 sends the step back to planning (`/trekplan` revises it, then `--resume`), round 5 to you. Never a fourth guess, never a cheaper model
|
||||||
- **On failure: revert** — undo changes, stop
|
- **On failure: revert** — fix rounds, then undo the step's changes when it goes up
|
||||||
- **On failure: retry** — try alternative approach, then revert if still failing
|
- **On failure: retry** — the alternative approach goes to the first fix round; revert when it goes up
|
||||||
- **On failure: skip** — non-critical step, continue
|
- **On failure: skip** — non-critical step, continue
|
||||||
- **On failure: escalate** — stop everything, needs human judgment
|
- **On failure: escalate** — stop at the first failed round, needs human judgment
|
||||||
|
|
||||||
### Security hardening
|
### Security hardening
|
||||||
|
|
||||||
|
|
@ -623,7 +641,7 @@ Solid arrows are the forward pipeline; dashed are conditional/internal edges (`-
|
||||||
|
|
||||||
### Agents per phase
|
### Agents per phase
|
||||||
|
|
||||||
Every command orchestrates its agents inline from the main context (all sub-agents are `model: opus`-pinned). `/trekexecute` is the exception — it spawns **no** sub-agents.
|
Every command orchestrates its agents inline from the main context (all sub-agents are `model: opus`-pinned). `/trekexecute` spawns no swarm: one fresh `step-implementer` per plan step, in order, and one controller per round.
|
||||||
|
|
||||||
```mermaid
|
```mermaid
|
||||||
flowchart TB
|
flowchart TB
|
||||||
|
|
@ -640,8 +658,8 @@ flowchart TB
|
||||||
PLC["Phase 9 — parallel · plan-critic · scope-guardian"]
|
PLC["Phase 9 — parallel · plan-critic · scope-guardian"]
|
||||||
PLG --> PLS --> PLC
|
PLG --> PLS --> PLC
|
||||||
end
|
end
|
||||||
subgraph EX["/trekexecute · no sub-agents"]
|
subgraph EX["/trekexecute · one helper per step"]
|
||||||
EXI["inline step-loop + session orders per wave (worktrees)<br/>+ Phase 7.5/7.6 manifest audit (deterministic)"]
|
EXI["per step: step-implementer → controller (spec + quality)<br/>+ red proof · floor guard · done line (commands)<br/>+ session orders per wave (worktrees) + Phase 7.5/7.6 audit"]
|
||||||
end
|
end
|
||||||
subgraph RV["/trekreview"]
|
subgraph RV["/trekreview"]
|
||||||
RVR["Phase 5 — parallel · code-correctness-reviewer · brief-conformance-reviewer*"]
|
RVR["Phase 5 — parallel · code-correctness-reviewer · brief-conformance-reviewer*"]
|
||||||
|
|
@ -803,7 +821,7 @@ Default JSONL stats stream (`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) is unchan
|
||||||
|
|
||||||
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5.1 on every phase). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
|
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5.1 on every phase). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
|
||||||
|
|
||||||
On the Anthropic API the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5** (both default to reasoning effort `medium`), and `fable` to **Fable 5.1** (the most capable model in Claude Code; default reasoning effort `high` — xhigh must be chosen explicitly, see [`docs/profiles.md`](docs/profiles.md)). Other providers resolve `opus` and `sonnet` to other versions. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` (eight agents pin `high`; the rest follow the session) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
On the Anthropic API the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5** (both default to reasoning effort `medium`), and `fable` to **Fable 5.1** (the most capable model in Claude Code; default reasoning effort `high` — xhigh must be chosen explicitly, see [`docs/profiles.md`](docs/profiles.md)). Other providers resolve `opus` and `sonnet` to other versions. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` (nine agents pin `high`; the rest follow the session) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
||||||
|
|
||||||
For per-profile cost estimates, see [`docs/profiles.md`](docs/profiles.md).
|
For per-profile cost estimates, see [`docs/profiles.md`](docs/profiles.md).
|
||||||
|
|
||||||
|
|
@ -819,8 +837,8 @@ Top-level layout:
|
||||||
|
|
||||||
```
|
```
|
||||||
trekplan/
|
trekplan/
|
||||||
├── agents/ 23 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
├── agents/ 24 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
||||||
│ └ 20 spawnable (1 dormant: synthesis-agent, Δ≈0) + 3 orchestrator reference docs (not spawned)
|
│ └ 21 spawnable (1 dormant: synthesis-agent, Δ≈0) + 3 orchestrator reference docs (not spawned)
|
||||||
├── commands/ 6 slash commands (trekbrief, trekresearch, trekplan, trekexecute, trekreview, trekcontinue) + trekendsession helper
|
├── commands/ 6 slash commands (trekbrief, trekresearch, trekplan, trekexecute, trekreview, trekcontinue) + trekendsession helper
|
||||||
├── templates/ Frontmatter templates for brief, research, plan, session
|
├── templates/ Frontmatter templates for brief, research, plan, session
|
||||||
├── hooks/ 8 hooks (pre-bash, pre-write, pre-agent-cap, session-title, post-bash-stats, pre-compact-flush, post-compact-flush, otel-export)
|
├── hooks/ 8 hooks (pre-bash, pre-write, pre-agent-cap, session-title, post-bash-stats, pre-compact-flush, post-compact-flush, otel-export)
|
||||||
|
|
|
||||||
109
agents/step-implementer.md
Normal file
109
agents/step-implementer.md
Normal file
|
|
@ -0,0 +1,109 @@
|
||||||
|
---
|
||||||
|
name: step-implementer
|
||||||
|
description: |
|
||||||
|
Use this agent when /trekexecute hands one plan step to a fresh helper: it
|
||||||
|
writes the step's test first, proves it red, locks it, implements exactly the
|
||||||
|
step's Changes, and writes a ran / saw / did-NOT-check report to a file.
|
||||||
|
model: opus
|
||||||
|
effort: high
|
||||||
|
color: green
|
||||||
|
tools: ["Read", "Write", "Edit", "Bash", "Glob", "Grep"]
|
||||||
|
---
|
||||||
|
|
||||||
|
You are the helper for ONE step of a Voyage plan. You start fresh: you know
|
||||||
|
nothing about earlier steps except what is in the repository and the files the
|
||||||
|
executor names. You do one step, report it in a file, and stop. You never start
|
||||||
|
another helper (you have no Agent tool, by design), never commit, never push,
|
||||||
|
and never touch a file the step does not list.
|
||||||
|
|
||||||
|
## What the executor gives you
|
||||||
|
|
||||||
|
- `plan` — path to the plan; `step` — the step number; the step's text verbatim
|
||||||
|
(Files, Changes, Reuses, Test first, Verify, and when present Interface in,
|
||||||
|
Interface out, Review focus).
|
||||||
|
- `start_sha` — the commit the step starts from. Everything you change is read
|
||||||
|
later as `git diff {start_sha}`, so the diff is your real report.
|
||||||
|
- `report_path` — where your report goes.
|
||||||
|
- `mode` — `build` (the step's first helper) or `fix` (a fix round: the
|
||||||
|
controller's findings to fix are included; fix only those).
|
||||||
|
- `voyage_root` — the plugin root, for the CLIs below.
|
||||||
|
|
||||||
|
## Mode `build`
|
||||||
|
|
||||||
|
1. **Test first.** Read the step's `Test first:` and write that test (and only
|
||||||
|
that test) first. Follow the `Pattern:` file's style when one is named.
|
||||||
|
Then register every new file you created so the executor's diff sees it:
|
||||||
|
`git add --intent-to-add <path>` (this stages nothing).
|
||||||
|
2. **Prove it red before you build.** Run, from the repository root:
|
||||||
|
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files>`
|
||||||
|
Exit 0 = the test fails at the start point: continue. Exit 1 = the test
|
||||||
|
already passes at the start point, so it proves nothing: rewrite the test
|
||||||
|
until it fails for the reason the step exists, and run the check again.
|
||||||
|
Exit 2 = it could not run (the Verify command is not one test-runner
|
||||||
|
invocation, or a file is missing): do not build; say so in the report and stop.
|
||||||
|
3. **Lock the tests** (when the CLI exists):
|
||||||
|
`node {voyage_root}/lib/util/test-lock.mjs lock --files <test files> --step {step}`.
|
||||||
|
If the file does not exist, write "not-locked (test-lock.mjs missing)" in the
|
||||||
|
report. Never silent. Once locked you cannot edit the tests, and that is the
|
||||||
|
point: the code changes to meet the test, never the reverse.
|
||||||
|
4. **Implement exactly the step's `Changes:`** in the files its `Files:` lists.
|
||||||
|
Read `Reuses:` first. No extra scope, no refactors on the side, no "while I
|
||||||
|
was here". If the step cannot be done as written, stop and say why in the
|
||||||
|
report: the executor sends it up to the plan; you do not improvise.
|
||||||
|
5. **Run the step's `Verify:` command** yourself, fresh. Its exit code is the
|
||||||
|
truth, not your reading of the output.
|
||||||
|
|
||||||
|
## Mode `fix`
|
||||||
|
|
||||||
|
You get the controller's findings (file, line, problem). Fix those and nothing
|
||||||
|
else. The tests stay locked: if a finding says the test itself is wrong, do not
|
||||||
|
work around it - say so in the report; that goes up to the plan. Run the
|
||||||
|
step's `Verify:` again.
|
||||||
|
|
||||||
|
## Never
|
||||||
|
|
||||||
|
- Never edit a test to make it pass: no deleted assert, no skip, no looser
|
||||||
|
threshold, no `|| true`. The executor's floor guard reads your diff for
|
||||||
|
exactly that, and finds it.
|
||||||
|
- Never write a stub ("not implemented", `TODO: implement`) as the step's body.
|
||||||
|
- Never write a `DONE` line. The executor's script writes it, and only when
|
||||||
|
the test command exits 0.
|
||||||
|
- Never commit, push, merge or tag. Never write outside the repository.
|
||||||
|
|
||||||
|
## The report (write it to `report_path`, then answer with one line)
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
## Step {N} - {build | fix round R}
|
||||||
|
|
||||||
|
### Ran
|
||||||
|
- `{command}` -> exit {code}
|
||||||
|
(every command you ran, in order, with its exit code)
|
||||||
|
|
||||||
|
### Saw
|
||||||
|
- {what the output actually showed: the failing test name at the start point,
|
||||||
|
the passing count after, a warning you did not fix}
|
||||||
|
|
||||||
|
### Did NOT check
|
||||||
|
- {what you did not verify, plainly: other platforms, the full suite, an edge
|
||||||
|
case the plan names but no test covers, anything you assumed}
|
||||||
|
|
||||||
|
### Files touched
|
||||||
|
- {path} ({new | modified})
|
||||||
|
|
||||||
|
### Test lock
|
||||||
|
- {locked | not-locked (test-lock.mjs missing) | not-locked (reason)}
|
||||||
|
```
|
||||||
|
|
||||||
|
"Did NOT check" is never empty: if you truly checked everything the step
|
||||||
|
names, say what lies outside the step that you did not look at.
|
||||||
|
|
||||||
|
Your final answer to the executor is one line: `report: {report_path}`. The
|
||||||
|
executor does not read your summary to decide anything; it reads the diff, and a
|
||||||
|
controller judges it.
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
The loop this helper lives in is adapted from obra/superpowers@8ca22db (MIT,
|
||||||
|
(c) 2025 Jesse Vincent), subagent-driven-development: a fresh helper per task,
|
||||||
|
hand-over in files. The ran / saw / did-NOT-check report contract is borrowed
|
||||||
|
from bashebr/ai-native-sdlc@11fb8d7 (MIT).
|
||||||
|
|
@ -2,8 +2,8 @@
|
||||||
name: trekexecute
|
name: trekexecute
|
||||||
description: Disciplined plan executor — single-session, or multi-session through session orders the user starts, with failure recovery
|
description: Disciplined plan executor — single-session, or multi-session through session orders the user starts, with failure recovery
|
||||||
argument-hint: "[--project <dir>] [--fg | --resume | --dry-run | --validate | --step N | --session N] [plan.md]"
|
argument-hint: "[--project <dir>] [--fg | --resume | --dry-run | --validate | --step N | --session N] [plan.md]"
|
||||||
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion, Agent
|
||||||
disallowed-tools: Agent, TeamCreate
|
disallowed-tools: TeamCreate
|
||||||
---
|
---
|
||||||
|
|
||||||
# Ultraexecute Local
|
# Ultraexecute Local
|
||||||
|
|
@ -12,9 +12,17 @@ Disciplined executor for trekplan plans. Reads a plan file, detects if it has
|
||||||
an Execution Strategy (multi-session), and either executes directly or
|
an Execution Strategy (multi-session), and either executes directly or
|
||||||
prepares one session order per session, wave by wave — all to realize one plan.
|
prepares one session order per session, wave by wave — all to realize one plan.
|
||||||
|
|
||||||
|
Each plan step is built by **one fresh helper** (`voyage:step-implementer`)
|
||||||
|
and judged by **one controller** that reads the step's diff. The executor
|
||||||
|
itself writes no product code: it records the start point, launches the
|
||||||
|
helper, runs the mechanical checks (proof of red, the floor guard, the
|
||||||
|
manifest, the done line), and decides from exit codes and the controller's
|
||||||
|
verdicts (Phase 6).
|
||||||
|
|
||||||
Voyage never starts a Claude Code session. Work that needs another session
|
Voyage never starts a Claude Code session. Work that needs another session
|
||||||
becomes an order plus a starter line the user gives a session they open
|
becomes an order plus a starter line the user gives a session they open
|
||||||
themselves (Phase 2.6).
|
themselves (Phase 2.6). Helpers inside this session are subagents, and helpers
|
||||||
|
never start helpers.
|
||||||
|
|
||||||
## Phase 1 — Parse mode and validate input
|
## Phase 1 — Parse mode and validate input
|
||||||
|
|
||||||
|
|
@ -120,15 +128,34 @@ Extract every `### Step N: {description}` heading (in order). For each step, ext
|
||||||
- **Files** — file paths to create or modify
|
- **Files** — file paths to create or modify
|
||||||
- **Changes** — what to modify
|
- **Changes** — what to modify
|
||||||
- **Reuses** — existing code to leverage (informational)
|
- **Reuses** — existing code to leverage (informational)
|
||||||
- **Test first** — test to run before implementation (optional)
|
- **Test first** — the test the step writes before it builds: its `File:`
|
||||||
- **Verify** — command to run after implementation
|
(existing | new), what it `Verifies:`, an optional `Pattern:` (required from
|
||||||
|
plan_version ≥ 1.8, see below)
|
||||||
|
- **Verify** — command to run after implementation. It is also the step's
|
||||||
|
**test command**: proof of red (Sub-step B) and the done line (Sub-step G)
|
||||||
|
run it, so it must be one test-runner invocation
|
||||||
- **On failure** — recovery action (revert/retry/skip/escalate)
|
- **On failure** — recovery action (revert/retry/skip/escalate)
|
||||||
- **Checkpoint** — git commit command after success
|
- **Checkpoint** — git commit command after success
|
||||||
- **Manifest** — YAML block following Checkpoint (v1.7+)
|
- **Manifest** — YAML block following Checkpoint (v1.7+)
|
||||||
|
- **Interface in** / **Interface out** / **Review focus** — optional (K4):
|
||||||
|
what the step consumes and produces, and what its controller must look at
|
||||||
|
hardest. Read when present and passed to the helper and the controller;
|
||||||
|
never required, and the executor tolerates their absence in every plan
|
||||||
|
version.
|
||||||
|
|
||||||
If a step is missing `On failure`, default to `escalate` and record a parse warning.
|
If a step is missing `On failure`, default to `escalate` and record a parse warning.
|
||||||
If a step is missing `Verify`, record a parse warning.
|
If a step is missing `Verify`, record a parse warning.
|
||||||
|
|
||||||
|
**`Test first:` is required from plan_version ≥ 1.8.** A ≥ 1.8 step without a
|
||||||
|
`Test first:` field (or whose `File:` is missing) is not executable: STOP with
|
||||||
|
```
|
||||||
|
Error: plan_version={v} but step {N} has no Test first: field.
|
||||||
|
Re-run /trekplan - from plan_version 1.8 every step proves red before it builds.
|
||||||
|
```
|
||||||
|
For plans below 1.8 the field stays optional: a step without it runs, its
|
||||||
|
`red_proof` is recorded as `"not-run"`, and the final report counts it among
|
||||||
|
the steps without proof of red (never hidden, never called proven).
|
||||||
|
|
||||||
### Parse the Manifest block (v1.7+)
|
### Parse the Manifest block (v1.7+)
|
||||||
|
|
||||||
Read the plan header for a `plan_version` marker. Treat ≥ `1.7` as strict
|
Read the plan header for a `plan_version` marker. Treat ≥ `1.7` as strict
|
||||||
|
|
@ -1023,16 +1050,35 @@ progress file.
|
||||||
"current_step": 0,
|
"current_step": 0,
|
||||||
"status": "{in_progress | completed | failed | stopped | partial}",
|
"status": "{in_progress | completed | failed | stopped | partial}",
|
||||||
"steps": {
|
"steps": {
|
||||||
"1": { "status": "pending", "attempts": 0, "error": null, "completed_at": null, "commit": null }
|
"1": { "status": "pending", "attempts": 0, "fix_rounds": 0, "start_sha": null, "report_path": null, "red_proof": null, "test_lock": null, "error": null, "completed_at": null, "commit": null }
|
||||||
},
|
},
|
||||||
"iterations_remaining": 25,
|
"iterations_remaining": 25,
|
||||||
"entry_condition_checked": false,
|
"entry_condition_checked": false,
|
||||||
"exit_condition_checked": false,
|
"exit_condition_checked": false,
|
||||||
"plan_verification": null,
|
"plan_verification": null,
|
||||||
|
"end_to_end": null,
|
||||||
|
"decisions": [],
|
||||||
"summary": null
|
"summary": null
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**The step-loop fields (Handover 5, additive-optional).** Written by Phase 6;
|
||||||
|
a progress file without them still validates and resumes:
|
||||||
|
|
||||||
|
| Field | Written | Meaning |
|
||||||
|
|---|---|---|
|
||||||
|
| `steps.{N}.start_sha` | Sub-step 0 | the commit the step started from; every check reads `git diff {start_sha}` |
|
||||||
|
| `steps.{N}.report_path` | Sub-step 0 | the step's report file (helper rounds + the done line) |
|
||||||
|
| `steps.{N}.red_proof` | Sub-step B | `"proven" \| "rejected" \| "not-run"` — the exit of `red-proof.mjs` |
|
||||||
|
| `steps.{N}.test_lock` | Sub-steps B, G | `"locked" \| "released" \| "not-locked"` (K8) |
|
||||||
|
| `steps.{N}.fix_rounds` | Sub-step F | 0–5: 0 = passed its first control, 1–3 = fix rounds, 4 = sent up to the plan, 5 = sent up to the operator |
|
||||||
|
| `steps.{N}.attempts` | Sub-steps C, F | helpers launched for the step: 1 + the fix rounds that launched one |
|
||||||
|
| `end_to_end` | Phase 7 | `"verified" \| "not-verified"` (P9 (6)) |
|
||||||
|
| `decisions` | any phase | `[{step, decision, cost_if_wrong}]` — every call the executor made on its own (Phase 5.5) |
|
||||||
|
|
||||||
|
`progress-validator.mjs` rejects `fix_rounds` outside 0–5 (there is no sixth
|
||||||
|
round) and a `decisions` entry without its `cost_if_wrong`.
|
||||||
|
|
||||||
`plan_version` is the value Phase 2 parsed from the plan header (required by
|
`plan_version` is the value Phase 2 parsed from the plan header (required by
|
||||||
`progress-validator.mjs`). The status words here are the validator's:
|
`progress-validator.mjs`). The status words here are the validator's:
|
||||||
`PROGRESS_VALID_STATUSES` for the run, `PROGRESS_STEP_STATUSES` for a step.
|
`PROGRESS_VALID_STATUSES` for the run, `PROGRESS_STEP_STATUSES` for a step.
|
||||||
|
|
@ -1047,8 +1093,8 @@ condition instead) and on early stop. Unknown keys are tolerated by
|
||||||
recovery/retry budget against the global hard cap (unit = **iterations**, per the
|
recovery/retry budget against the global hard cap (unit = **iterations**, per the
|
||||||
brief — token/cost budgeting is out of scope). Seeded to
|
brief — token/cost budgeting is out of scope). Seeded to
|
||||||
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default 25) at start. **Single writer:** the
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default 25) at start. **Single writer:** the
|
||||||
executor decrements it by 1 on **every recovery/retry iteration** — each step
|
executor decrements it by 1 on **every recovery/retry iteration** — each fix
|
||||||
attempt beyond the first, and each Phase 7.6 recovery dispatch. **Backfill-on-absent:**
|
round of a step (Sub-step F), and each Phase 7.6 recovery dispatch. **Backfill-on-absent:**
|
||||||
when resuming a legacy `progress.json` that lacks the field, seed it to the cap
|
when resuming a legacy `progress.json` that lacks the field, seed it to the cap
|
||||||
rather than hard-failing (forward/backward-compatible; shape-validated by
|
rather than hard-failing (forward/backward-compatible; shape-validated by
|
||||||
`progress-validator.mjs` — non-negative integer, never required-top). The field is
|
`progress-validator.mjs` — non-negative integer, never required-top). The field is
|
||||||
|
|
@ -1160,6 +1206,47 @@ If `has_execution_strategy = true`, show a preview of multi-session orchestratio
|
||||||
|
|
||||||
Stop after the dry-run report. Do not execute anything.
|
Stop after the dry-run report. Do not execute anything.
|
||||||
|
|
||||||
|
## Phase 5.5 — Autonomy: what stops a run, and the decision log
|
||||||
|
|
||||||
|
A run is long and mostly unattended. It does not stop to ask: what it decides
|
||||||
|
on its own it writes down, and it stops only for the four reasons below.
|
||||||
|
|
||||||
|
### Closed stop list
|
||||||
|
|
||||||
|
The list is **closed**: a run stops on these four classes and on nothing else.
|
||||||
|
Every stop names its class in the final report (Phase 8 `### Stops`).
|
||||||
|
|
||||||
|
1. **Something that cannot be undone.** A step or command whose effect cannot
|
||||||
|
be reverted from git: deleting data outside the repo's tracked files, a
|
||||||
|
migration against a live store, anything the plan marks irreversible.
|
||||||
|
2. **Something security-sensitive.** A command the executor denylist blocks
|
||||||
|
(Phase 2.4, Sub-step D1), a write to a security-sensitive path (Hard
|
||||||
|
rule 16), credentials or secrets in a diff.
|
||||||
|
3. **Something outside the working copy.** Merge to the main branch, push,
|
||||||
|
tag, publish, a write outside the repository (Hard rule 15). The main-merge
|
||||||
|
gate (Phase 8) is this class: it always pauses.
|
||||||
|
4. **A plan so broken that everything after it is guesswork.** The plan fails
|
||||||
|
to parse, a ≥ 1.8 step has no `Test first:`, a step's own `On failure:
|
||||||
|
escalate` fired (the plan's author declared that past this point is
|
||||||
|
guesswork), or a step went up the fix ladder to the plan (Sub-step F).
|
||||||
|
|
||||||
|
Not on the list, and therefore never a stop: a failing test (that is a fix
|
||||||
|
round), an ambiguity the plan leaves open (decide, and log it), a style
|
||||||
|
choice, a missing optional field.
|
||||||
|
|
||||||
|
### Decision log
|
||||||
|
|
||||||
|
Every decision the executor (or a helper, through its report) takes that the
|
||||||
|
plan did not dictate goes into progress `decisions` as one entry:
|
||||||
|
`{"step": N, "decision": "...", "cost_if_wrong": "..."}`. `cost_if_wrong` is
|
||||||
|
concrete: what has to be redone, by whom, how big — "one rename commit",
|
||||||
|
"step 4's API shape; steps 5–7 build on it". The final report prints the log
|
||||||
|
(Phase 8 `### Decisions taken`), so the operator reviews the calls instead of
|
||||||
|
being asked them one by one.
|
||||||
|
|
||||||
|
Borrowed: Superpowers subagent-driven-development (MIT) — the stop classes and
|
||||||
|
the log of choices made while unattended.
|
||||||
|
|
||||||
## Phase 6 — Step execution loop
|
## Phase 6 — Step execution loop
|
||||||
|
|
||||||
The core execution phase. Runs for modes: `execute`, `resume`, `step`.
|
The core execution phase. Runs for modes: `execute`, `resume`, `step`.
|
||||||
|
|
@ -1179,6 +1266,33 @@ Update progress: `steps.{N}.status = "in_progress"`, `current_step = N`, `update
|
||||||
--- Step {N}/{total}: {description} ---
|
--- Step {N}/{total}: {description} ---
|
||||||
```
|
```
|
||||||
|
|
||||||
|
One step at a time. Inside one session the steps run in order in one working
|
||||||
|
tree; parallel steps are the multi-session wave path (Phase 2.6), never two
|
||||||
|
helpers in one tree.
|
||||||
|
|
||||||
|
#### Sub-step 0 — Record the start point
|
||||||
|
|
||||||
|
Before anything touches the tree, record where the step begins:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git rev-parse HEAD
|
||||||
|
git status --porcelain
|
||||||
|
```
|
||||||
|
|
||||||
|
- `steps.{N}.start_sha` = the `HEAD` sha. Every later check of this step reads
|
||||||
|
`git diff {start_sha}`: the review package is the diff from this recorded
|
||||||
|
start point, not anything the helper says. On `--resume` of a step that
|
||||||
|
already has a `start_sha`, keep it.
|
||||||
|
- `steps.{N}.report_path` = `{progress-dir}/steps/step-{N}.report.md` (create
|
||||||
|
the directory). Every helper round of the step appends to it, and the done
|
||||||
|
line lands in it (Sub-step G). Hand-over between the executor, the helper
|
||||||
|
and the controller is in files: the plan, this report, the diff.
|
||||||
|
- A non-empty `git status --porcelain` (beyond Voyage's own progress and report
|
||||||
|
files) means work the plan does not know about would mix into this step's
|
||||||
|
diff. Not a stop (Phase 5.5): log it as a decision
|
||||||
|
(`cost_if_wrong`: "the controller judges unrelated changes as part of step N")
|
||||||
|
and name the paths in the report.
|
||||||
|
|
||||||
#### Sub-step A — Scope fence check (session specs only)
|
#### Sub-step A — Scope fence check (session specs only)
|
||||||
|
|
||||||
Before touching any file, verify that every file in the step's `Files:` field is
|
Before touching any file, verify that every file in the step's `Files:` field is
|
||||||
|
|
@ -1192,24 +1306,97 @@ Escalating — this step cannot be executed within this session's scope.
|
||||||
|
|
||||||
Treat this as an automatic `escalate`. Jump to the stop-and-report logic.
|
Treat this as an automatic `escalate`. Jump to the stop-and-report logic.
|
||||||
|
|
||||||
#### Sub-step B — Test first (if present)
|
#### Sub-step B — Test first and proof of red
|
||||||
|
|
||||||
If the step has a `Test first:` field:
|
The step's test is written first, and a command proves it red — never the
|
||||||
1. If test file is marked `(new)`: note it will be created during implementation.
|
model's word that "it failed first". This sub-step blocks: a step whose test
|
||||||
2. If test file exists: run it. Expect failure (RED state).
|
is not proven red at its start point and green at its result does not pass.
|
||||||
3. If test unexpectedly passes: warn but continue — step may already be done.
|
(A plan below 1.8 whose step has no `Test first:` skips it: `red_proof` =
|
||||||
|
`"not-run"`, counted in the final report, never called proven.)
|
||||||
|
|
||||||
Do not block on test-first failures — they are expected.
|
**The test command** is the step's `Verify:` command. It must be ONE
|
||||||
|
test-runner invocation without shell operators (the allowlist of
|
||||||
|
`lib/verification/criteria-runner.mjs`); the tools below refuse anything else
|
||||||
|
with exit 2.
|
||||||
|
|
||||||
#### Sub-step C — Implement changes
|
1. **The helper proves red before it builds** (Sub-step C): it writes the test
|
||||||
|
from `Test first:`, runs the gate
|
||||||
|
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {files}`
|
||||||
|
(the test must FAIL at the start point), locks the test files, and only then
|
||||||
|
implements.
|
||||||
|
2. **The executor proves it both ways after the helper returns:**
|
||||||
|
```bash
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {Test first File paths, comma-separated} --json
|
||||||
|
```
|
||||||
|
The test must FAIL against the start point (`start_sha` unpacked with
|
||||||
|
`git archive`, the step's test files copied in) and must PASS against the
|
||||||
|
result (the working tree). The start point holds tracked files only: add
|
||||||
|
`--link node_modules` (or the repo's untracked dependency directory) when
|
||||||
|
the test needs it.
|
||||||
|
|
||||||
Read the step's `Files:` and `Changes:` fields. Implement exactly as described.
|
| Exit | `steps.{N}.red_proof` | Executor |
|
||||||
|
|------|-----------------------|----------|
|
||||||
|
| 0 | `"proven"` | continue |
|
||||||
|
| 1, `not-red-at-base` | `"rejected"` | the test passes without the change, so it proves nothing: a finding for Sub-step F |
|
||||||
|
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
|
||||||
|
| 2 | `"not-run"` | never a pass and never proven. When the reason is the command (the `Verify:` is not one test-runner invocation) on a plan ≥ 1.8, the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4). Any other reason (a missing test file, an unresolvable start point) is a finding for Sub-step F |
|
||||||
|
|
||||||
**Rules:**
|
The exit code is the verdict, the output text is not. `base.tap` (when the
|
||||||
- Follow `Changes:` exactly — do not improvise, add scope, or optimize
|
runner prints TAP) says what the red was: `fail 1 of 1` on a node test file is
|
||||||
- Use Edit for modifications, Write for new files
|
usually a load error, not a named failing test — say so in the step report.
|
||||||
- If `Reuses:` references existing code, read that code first for context
|
|
||||||
- Only touch files listed in `Files:` — nothing else
|
**The test lock (K8).** After its red gate the helper runs
|
||||||
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs lock --files {test files} --step {N}`;
|
||||||
|
from then on the guard refuses Write/Edit/NotebookEdit on those files, so the
|
||||||
|
code moves to meet the test and never the reverse. The executor reads
|
||||||
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs status` and records
|
||||||
|
`steps.{N}.test_lock = "locked"` when it exits 0 and names the step's files.
|
||||||
|
If `lib/util/test-lock.mjs` is not in this plugin, record `"not-locked"` and
|
||||||
|
write "not locked: test-lock.mjs missing" in the step report — never silent.
|
||||||
|
The floor guard (Sub-step D3) still reads every test change in the diff. The
|
||||||
|
lock is released in Sub-step G, or when the run stops.
|
||||||
|
|
||||||
|
#### Sub-step C — Implement through one fresh helper
|
||||||
|
|
||||||
|
The executor does not write the step's code. **Launch** one fresh helper for
|
||||||
|
the step with the Agent tool:
|
||||||
|
|
||||||
|
Launch `subagent_type: "voyage:step-implementer"` with a short `description`
|
||||||
|
("step {N}: {title}") and a prompt that carries, as files and values, not as a
|
||||||
|
summary: the plan path, the step number, the step's text verbatim (every
|
||||||
|
field, including `Interface in:`, `Interface out:` and `Review focus:` when
|
||||||
|
present), `start_sha`, `report_path`, `mode: build`, and
|
||||||
|
`voyage_root: ${CLAUDE_PLUGIN_ROOT}`.
|
||||||
|
|
||||||
|
- **Never pass the Agent tool's `name` parameter.** A named spawn becomes an
|
||||||
|
`in_process_teammate`: it does the work, but its result never reaches this
|
||||||
|
orchestrator (measured: named 0/5 returned, unnamed 3/3 —
|
||||||
|
`docs/agent-return-channel-defect.md`). The same holds for the controller
|
||||||
|
(Sub-step E).
|
||||||
|
- **Never pass `isolation`.** Measured 2026-10-06 (Claude Code 2.1.291): a
|
||||||
|
worktree-isolated helper branched from the repository's default branch
|
||||||
|
(`02f320f`), not from the executor's `HEAD` (`2c89a6b`), so it did not see
|
||||||
|
the earlier steps. Only the operator's `worktree.baseRef: "head"` setting
|
||||||
|
changes that, and a plugin cannot set it. The helper works in this tree,
|
||||||
|
from the recorded `start_sha`.
|
||||||
|
- **Wait for the helper to finish** before anything else; when it runs in the
|
||||||
|
background, its completion notification is the signal. Never start the next
|
||||||
|
sub-step or step while a helper runs.
|
||||||
|
- `steps.{N}.attempts` = the number of helpers launched for the step (1 here).
|
||||||
|
|
||||||
|
The helper's one-line answer is not evidence. After it returns:
|
||||||
|
1. The report must exist at `report_path` with its `### Ran`, `### Saw` and
|
||||||
|
`### Did NOT check` sections (a missing report is a finding for Sub-step F).
|
||||||
|
2. Register the step's new files so the diff sees them: for each path in
|
||||||
|
`Files:` that `git status --porcelain` shows as `??`, run
|
||||||
|
`git add --intent-to-add {path}` (stages nothing).
|
||||||
|
3. A changed path outside the step's `Files:` is a scope finding (Hard rule 2)
|
||||||
|
for the controller. The executor does not revert it on its own.
|
||||||
|
|
||||||
|
**Rules the helper is held to** (its agent file carries them; the controller
|
||||||
|
checks them in the diff): follow `Changes:` exactly — no improvising, no added
|
||||||
|
scope; touch only the files in `Files:`; read `Reuses:` first; never weaken a
|
||||||
|
test; never write a stub; never commit.
|
||||||
|
|
||||||
#### Sub-step D1 — Command verification
|
#### Sub-step D1 — Command verification
|
||||||
|
|
||||||
|
|
@ -1259,7 +1446,9 @@ status = `blocked` and apply `On failure: escalate`. Skip D2 manifest
|
||||||
verification (nothing to verify for a read-only sandbox test). Commit none —
|
verification (nothing to verify for a read-only sandbox test). Commit none —
|
||||||
jump straight to Phase 7 with a structured "sandbox-blocked" reason.
|
jump straight to Phase 7 with a structured "sandbox-blocked" reason.
|
||||||
|
|
||||||
If **PASS**: proceed to Sub-step D2 (manifest verification).
|
If **PASS**: proceed to Sub-step D2 (manifest verification). If **FAIL**: the
|
||||||
|
failure output is a finding for Sub-step F; still run D2 and D3, so the
|
||||||
|
controller and the next fix round see every failure at once.
|
||||||
|
|
||||||
#### Sub-step D2 — Manifest verification
|
#### Sub-step D2 — Manifest verification
|
||||||
|
|
||||||
|
|
@ -1273,10 +1462,10 @@ holds, regardless of Verify exit code.
|
||||||
file exists in the repo. Count how many exist.
|
file exists in the repo. Count how many exist.
|
||||||
2. **min_file_count satisfied:** count from (1) must be ≥ `min_file_count`.
|
2. **min_file_count satisfied:** count from (1) must be ≥ `min_file_count`.
|
||||||
3. **Forbidden paths untouched:** for each `forbidden_paths` entry, run
|
3. **Forbidden paths untouched:** for each `forbidden_paths` entry, run
|
||||||
`git diff --name-only HEAD~{attempts} HEAD -- {path}` (since this step
|
`git diff --name-only {start_sha} -- {path}` (since this step
|
||||||
began). Any modified forbidden path = fail.
|
began). Any modified forbidden path = fail.
|
||||||
4. **Bash syntax check:** for each entry in `bash_syntax_check` AND any
|
4. **Bash syntax check:** for each entry in `bash_syntax_check` AND any
|
||||||
`.sh` file that appears in `git diff --name-only HEAD~1 HEAD` (safety
|
`.sh` file that appears in `git diff --name-only {start_sha}` (safety
|
||||||
net for unlisted scripts), run `bash -n {path}`. Non-zero exit = fail.
|
net for unlisted scripts), run `bash -n {path}`. Non-zero exit = fail.
|
||||||
5. **must_contain patterns:** for each `{path, pattern}` pair, run
|
5. **must_contain patterns:** for each `{path, pattern}` pair, run
|
||||||
`grep -E "{pattern}" {path}`. No match = fail.
|
`grep -E "{pattern}" {path}`. No match = fail.
|
||||||
|
|
@ -1288,45 +1477,144 @@ Manifest verification FAIL for step {N}:
|
||||||
- {which check failed with detail}
|
- {which check failed with detail}
|
||||||
```
|
```
|
||||||
|
|
||||||
Apply the step's `On failure:` clause (same as Sub-step D1 failure). Increment
|
A manifest failure is a finding for Sub-step F, exactly like a command
|
||||||
attempts. Manifest failures count against the retry cap equally with command
|
failure: it counts against the same fix rounds.
|
||||||
failures.
|
|
||||||
|
|
||||||
If all manifest checks pass: proceed to Sub-step F (checkpoint).
|
Either way, proceed to Sub-step D3 (floor guard).
|
||||||
|
|
||||||
Record per-step manifest audit result in progress file:
|
Record per-step manifest audit result in progress file:
|
||||||
`steps.{N}.manifest_audit = "pass" | "fail"`,
|
`steps.{N}.manifest_audit = "pass" | "fail"`,
|
||||||
`steps.{N}.manifest_drift = [{check: reason, ...}]` on fail.
|
`steps.{N}.manifest_drift = [{check: reason, ...}]` on fail.
|
||||||
|
|
||||||
#### Sub-step E — On failure handling
|
#### Sub-step D3 — Floor guard
|
||||||
|
|
||||||
If **FAIL**, read the `On failure:` clause. Apply the retry cap: **maximum 2 retries**
|
Read the step's diff for weakened tests — by a command:
|
||||||
(3 total attempts). Track attempts in `steps.{N}.attempts`.
|
|
||||||
|
|
||||||
**`On failure: revert`**
|
```bash
|
||||||
- If attempts < 3: analyze the failure, re-implement with adjustments, re-verify.
|
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/test-integrity.mjs --base {start_sha} --json
|
||||||
```
|
```
|
||||||
Attempt {A}/3 failed. Retrying...
|
|
||||||
|
It reports a removed assert, a deleted or skipped test, a muted check
|
||||||
|
(`|| true`, continue-on-error, an inline lint/type suppression), a stub body
|
||||||
|
("not implemented") and a threshold loosened in the permissive direction.
|
||||||
|
|
||||||
|
| Exit | Meaning | Executor |
|
||||||
|
|------|---------|----------|
|
||||||
|
| 0 | clean | continue |
|
||||||
|
| 1 | findings | each finding goes to the controller (Sub-step E), which rules it `justified` only by quoting the plan line that asks for the change; every other finding is a `violation` |
|
||||||
|
| 2 | could not run | exit 2 is never green: "floor guard not run: {reason}" is a finding for Sub-step F, and the step cannot pass this round |
|
||||||
|
|
||||||
|
A finding is never dropped silently: the step report and the final report list
|
||||||
|
every one with its ruling.
|
||||||
|
|
||||||
|
#### Sub-step E — Control
|
||||||
|
|
||||||
|
**One controller, two verdicts.** After the mechanical checks (B, D1, D2, D3)
|
||||||
|
— whatever they said — launch one controller with the Agent tool:
|
||||||
|
`subagent_type: "general-purpose"`, `model: "opus"`, no `name` (Sub-step C),
|
||||||
|
no `isolation`. It is a general helper with the prompt below, not a second
|
||||||
|
Voyage agent. It reads the diff, not the helper's summary: it is never given
|
||||||
|
the step report, and it judges what changed, not what the helper says changed.
|
||||||
|
|
||||||
|
The prompt (fill in every `{…}`; pass the mechanical results verbatim as JSON):
|
||||||
|
|
||||||
|
```text
|
||||||
|
You are the controller for step {N} of the plan {plan path}. You judge; you
|
||||||
|
change nothing: no Write, no Edit, no git command that writes.
|
||||||
|
|
||||||
|
Read the step's diff yourself: git diff {start_sha} --stat, then
|
||||||
|
git diff {start_sha}. The diff is the evidence. You are not given the
|
||||||
|
helper's report and must not look for it.
|
||||||
|
|
||||||
|
The step, verbatim:
|
||||||
|
{every field of the step}
|
||||||
|
|
||||||
|
Review focus from the plan (look hardest here): {Review focus, or "none"}
|
||||||
|
|
||||||
|
Mechanical results: proof of red {red-proof JSON}; Verify exit {code};
|
||||||
|
manifest {pass | fail + drift}; floor guard {test-integrity JSON}.
|
||||||
|
|
||||||
|
Give two verdicts:
|
||||||
|
- "spec": does the diff do what the step's Changes, Files and Interface say -
|
||||||
|
no more, no less? Name any changed path outside Files.
|
||||||
|
- "quality": does the code hold - correctness, error handling, edge cases the
|
||||||
|
step names, no stub, no weakened test, the repo's own patterns?
|
||||||
|
Rule every floor-guard finding: "justified: <the plan line, quoted>" or
|
||||||
|
"violation". List what you did not judge.
|
||||||
|
|
||||||
|
Answer with exactly one JSON block:
|
||||||
|
{"spec": "pass|fail", "quality": "pass|fail",
|
||||||
|
"findings": [{"verdict": "spec|quality", "file": "...", "line": 0, "problem": "...", "fix": "..."}],
|
||||||
|
"floor_guard": [{"kind": "...", "file": "...", "ruling": "justified: ... | violation"}],
|
||||||
|
"not_judged": ["..."]}
|
||||||
```
|
```
|
||||||
- If attempts == 3: revert this step's changes:
|
|
||||||
|
Wait for the controller to finish. Parse its JSON block; an answer without a
|
||||||
|
parseable block is a failed control (a finding for Sub-step F), never a pass.
|
||||||
|
Append both verdicts, the findings and `not_judged` to the step report.
|
||||||
|
|
||||||
|
**The round passes** only when ALL hold: `red_proof` is `"proven"` (or, below
|
||||||
|
plan_version 1.8, `"not-run"` because the step has no `Test first:`), Verify
|
||||||
|
PASS, manifest pass, every floor-guard finding ruled `justified` (and the guard
|
||||||
|
ran), `"spec": "pass"` and `"quality": "pass"`. Then go to Sub-step G.
|
||||||
|
Otherwise go to Sub-step F with every finding.
|
||||||
|
|
||||||
|
**Re-control after a fix** checks only what was fixed. A fresh controller gets
|
||||||
|
the previous round's findings and the round's own change — before the fix
|
||||||
|
round the executor saves `git diff {start_sha}` to
|
||||||
|
`{report dir}/step-{N}-round-{R}.diff`, and afterwards hands over
|
||||||
|
`diff -u` of that file against the new `git diff {start_sha}` — and answers,
|
||||||
|
per previous finding, fixed or not, plus whether the round changed anything
|
||||||
|
the findings did not name (that is a new finding). It does not re-judge the
|
||||||
|
rest of the step.
|
||||||
|
|
||||||
|
#### Sub-step F — Fix rounds
|
||||||
|
|
||||||
|
A failed round never loops sideways. **The fix rounds are capped at maximum 5 fix rounds per step, and they escalate upward** — to a fresh helper, then to
|
||||||
|
the plan, then to the operator. The executor does not improvise a new design,
|
||||||
|
and there is no ladder of cheaper models: every helper and every controller
|
||||||
|
runs `opus` (Voyage runs no helper below Opus 5.5 High).
|
||||||
|
|
||||||
|
| `fix_rounds` | Goes to | What happens |
|
||||||
|
|---|---|---|
|
||||||
|
| 1, 2, 3 | a fresh helper | Launch `voyage:step-implementer` again (no `name`, no `isolation`), `mode: fix`, with the findings (file, line, problem, fix) and the mechanical failures. It fixes only those; the tests stay locked. Then B (both ways), D1, D2, D3 run again and a fresh controller re-controls only what was fixed (Sub-step E). `attempts` +1; `iterations_remaining` −1 |
|
||||||
|
| 4 | the plan | **After three failed fix rounds the question is the architecture, not a fourth fix.** The step goes back to planning (below). Stop class 4 |
|
||||||
|
| 5 | the operator | A step already sent to the plan (`fix_rounds` 4) that fails its control again after re-planning goes to the operator. The executor never builds this step again: on a later `--resume` it only runs the checks (B, D1, D2, D3, E) on whatever the operator changed. Stop class 4 |
|
||||||
|
|
||||||
|
**Back to planning** (`fix_rounds = 4`): save `git diff {start_sha}` to
|
||||||
|
`{report dir}/step-{N}-upward.diff` and write `{report dir}/step-{N}-upward.md`:
|
||||||
|
the step verbatim, every round's findings and verdicts, and the question for
|
||||||
|
the planner — which assumption in the step (its Changes, its Interface, its
|
||||||
|
test) makes it fail. Then stop: step `failed`, run `stopped`, and tell the
|
||||||
|
operator to re-plan the step (`/trekplan` revises it) and run
|
||||||
|
`/trekexecute --resume`, which launches a fresh `build` helper for the revised
|
||||||
|
step and keeps `fix_rounds` at 4. A step whose proof of red cannot run on a
|
||||||
|
≥ 1.8 plan because its `Verify:` is not one test-runner invocation goes here
|
||||||
|
directly (Sub-step B).
|
||||||
|
|
||||||
|
**The plan's `On failure:` clause** decides what happens around the ladder:
|
||||||
|
|
||||||
|
**`On failure: revert`** and **`On failure: retry`**
|
||||||
|
- Fix rounds 1–3 run as above; for `retry`, the clause's alternative approach
|
||||||
|
is handed to the round-1 fix helper.
|
||||||
|
- When the step goes up (round 4 or 5), first save the upward files, then
|
||||||
|
revert this step's changes (nothing of the step is committed yet):
|
||||||
```bash
|
```bash
|
||||||
git checkout -- {files from Files: field}
|
git checkout -- {files from Files: field}
|
||||||
```
|
```
|
||||||
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.
|
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.
|
||||||
|
|
||||||
**`On failure: retry`**
|
|
||||||
- If attempts < 3: use the alternative approach described in the On failure clause.
|
|
||||||
- If attempts == 3: revert and stop. Jump to Phase 7.
|
|
||||||
|
|
||||||
**`On failure: skip`**
|
**`On failure: skip`**
|
||||||
- Mark step as skipped regardless of attempt count. Continue to next step.
|
- Mark step as skipped at its first failed round, without fix rounds. Continue
|
||||||
|
to next step.
|
||||||
```
|
```
|
||||||
Step {N}: SKIPPED (non-critical per plan)
|
Step {N}: SKIPPED (non-critical per plan)
|
||||||
```
|
```
|
||||||
Update `steps.{N}.status = "skipped"`.
|
Update `steps.{N}.status = "skipped"`.
|
||||||
|
|
||||||
**`On failure: escalate`**
|
**`On failure: escalate`**
|
||||||
- Stop immediately regardless of attempt count.
|
- Stop at the first failed round, without fix rounds (stop class 4: the plan's
|
||||||
|
author declared that past this point is guesswork).
|
||||||
```
|
```
|
||||||
Step {N}: ESCALATED — requires human judgment
|
Step {N}: ESCALATED — requires human judgment
|
||||||
```
|
```
|
||||||
|
|
@ -1338,9 +1626,33 @@ If **FAIL**, read the `On failure:` clause. Apply the retry cap: **maximum 2 ret
|
||||||
```
|
```
|
||||||
Jump to Phase 7.
|
Jump to Phase 7.
|
||||||
|
|
||||||
#### Sub-step F — Checkpoint
|
The same wip commit of completed work runs whenever a step goes up (round 4
|
||||||
|
or 5). Release the step's test lock before stopping
|
||||||
|
(`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock`, `test_lock` =
|
||||||
|
`"released"`), so the operator and the planner can edit the test.
|
||||||
|
|
||||||
Run the `Checkpoint:` git commit command exactly as written in the plan.
|
#### Sub-step G — Done line, unlock, checkpoint
|
||||||
|
|
||||||
|
1. **The done line.** A step is done only when a script says so, and the script
|
||||||
|
writes its line only when the test command exits 0:
|
||||||
|
```bash
|
||||||
|
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --step {N} --cmd "{Verify command}" --report {report_path}
|
||||||
|
```
|
||||||
|
Exit 0 = `DONE step {N}: …` appended to the step report. Exit 1 = the test
|
||||||
|
command failed after all: back to Sub-step F. Exit 2 = it could not run:
|
||||||
|
never green — on a ≥ 1.8 plan that is the plan's `Verify:` (up to the plan,
|
||||||
|
as in Sub-step B); below 1.8, a step whose `Verify:` is not a test-runner
|
||||||
|
command records "no done line (Verify is not a test-runner command)" in its
|
||||||
|
report and rests on Sub-step D1's exit code. Confirm with
|
||||||
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path}`
|
||||||
|
(exit 0 = present). Neither the helper nor the executor ever types a DONE
|
||||||
|
line.
|
||||||
|
2. **Release the test lock**:
|
||||||
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock` → `test_lock` =
|
||||||
|
`"released"` (a step that was `"not-locked"` stays so).
|
||||||
|
3. **Checkpoint.** Stage the step's `Files:` — and only those, never
|
||||||
|
`git add -A` — then run the `Checkpoint:` git commit command exactly as
|
||||||
|
written in the plan.
|
||||||
|
|
||||||
If the commit fails (nothing to commit, etc.): warn but do NOT fail the step.
|
If the commit fails (nothing to commit, etc.): warn but do NOT fail the step.
|
||||||
The step's verification already passed — the commit is bookkeeping.
|
The step's verification already passed — the commit is bookkeeping.
|
||||||
|
|
@ -1474,6 +1786,23 @@ now enforced on the single-session path and not only on the multi-session one.
|
||||||
Continue to Phase 7.5 either way: the manifest audit is independent, and a
|
Continue to Phase 7.5 either way: the manifest audit is independent, and a
|
||||||
verification failure must not hide a manifest drift.
|
verification failure must not hide a manifest drift.
|
||||||
|
|
||||||
|
### End to end — "run the thing" (P9 (6))
|
||||||
|
|
||||||
|
Tests passing is not the product working. From the runner's JSON, set
|
||||||
|
`end_to_end`:
|
||||||
|
|
||||||
|
- `"verified"` — at least one criterion that **passed** says in its text that
|
||||||
|
it runs the thing end to end (its text contains `end to end`, `end-to-end`
|
||||||
|
or `run the thing`, any case);
|
||||||
|
- `"not-verified"` — otherwise, including a session spec and a plan whose
|
||||||
|
verification did not pass.
|
||||||
|
|
||||||
|
`"not-verified"` is never green: the report says **not verified end to end**
|
||||||
|
in plain words, and the completion gate (Phase 7.5, clause 4) turns
|
||||||
|
`completed` into `partial`. The fix is in the plan — a criterion that runs the
|
||||||
|
built thing — not in the run. (This reads the existing `## Verification`
|
||||||
|
criteria; it adds no plan field.)
|
||||||
|
|
||||||
## Phase 7.5 — Manifest audit (independent)
|
## Phase 7.5 — Manifest audit (independent)
|
||||||
|
|
||||||
**Runs for all modes except dry-run.** This is the last-line-of-defense
|
**Runs for all modes except dry-run.** This is the last-line-of-defense
|
||||||
|
|
@ -1505,6 +1834,13 @@ produces an independent truth based on objective state, not self-narrative.
|
||||||
6. **Forbidden-path sweep:** for each `scope_forbidden` entry, check
|
6. **Forbidden-path sweep:** for each `scope_forbidden` entry, check
|
||||||
`git diff --name-only {start_sha} HEAD -- {path}` is empty.
|
`git diff --name-only {start_sha} HEAD -- {path}` is empty.
|
||||||
|
|
||||||
|
7. **Done-line sweep:** for every step with `status=completed` that carries a
|
||||||
|
`report_path`, run
|
||||||
|
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path}`.
|
||||||
|
Exit 1 on a plan ≥ 1.8 = drift (`{check: "done-line", expected: "DONE step N", actual: "absent"}`);
|
||||||
|
below 1.8 it is drift only when the step's report does not say why it has
|
||||||
|
none (Sub-step G).
|
||||||
|
|
||||||
Compare the audit result against the executor's `progress.status`:
|
Compare the audit result against the executor's `progress.status`:
|
||||||
|
|
||||||
- **Audit pass + progress=completed:** status stays `completed`.
|
- **Audit pass + progress=completed:** status stays `completed`.
|
||||||
|
|
@ -1548,6 +1884,9 @@ Record in progress file:
|
||||||
> result is OVERRIDDEN `completed → partial`, exactly as the audit-override
|
> result is OVERRIDDEN `completed → partial`, exactly as the audit-override
|
||||||
> already does (`:1221`). So the counter's liveness is auditable at the gate,
|
> already does (`:1221`). So the counter's liveness is auditable at the gate,
|
||||||
> not merely asserted in prose.
|
> not merely asserted in prose.
|
||||||
|
> 4. **End to end** — for a trekplan, Phase 7's `end_to_end == "verified"`.
|
||||||
|
> `"not-verified"` OVERRIDES `completed → partial` with the reason
|
||||||
|
> "not verified end to end": tests that pass are not the thing running.
|
||||||
>
|
>
|
||||||
> Termination authority therefore rests on the deterministic audit + an
|
> Termination authority therefore rests on the deterministic audit + an
|
||||||
> objective stop-signal, never on the model's narrative. Native `/goal` (when
|
> objective stop-signal, never on the model's narrative. Native `/goal` (when
|
||||||
|
|
@ -1611,16 +1950,19 @@ Do NOT dispatch a third recovery. Report to the user.
|
||||||
Every recovery/retry loop in this executor has a documented numeric bound.
|
Every recovery/retry loop in this executor has a documented numeric bound.
|
||||||
These are **three distinct axes** — not one nested counter:
|
These are **three distinct axes** — not one nested counter:
|
||||||
|
|
||||||
- **Per-step retry cap** — `attempts ≤ 3` (initial + **maximum 2 retries**;
|
- **Per-step fix-round cap** — `fix_rounds ≤ 5`: maximum 5 fix rounds per step, escalating upward
|
||||||
Hard Rule 5 / Sub-step E). Bounds the retries of a *single* step.
|
(rounds 1–3 fix with a fresh helper, 4 goes to the plan, 5 to the operator;
|
||||||
|
Hard Rule 5 / Sub-step F). `attempts` counts the helpers a step launched.
|
||||||
|
Bounds the rounds of a *single* step.
|
||||||
- **Per-session recovery cap** — `recovery_depth < 2` (Phase 7.6 hard cap).
|
- **Per-session recovery cap** — `recovery_depth < 2` (Phase 7.6 hard cap).
|
||||||
Bounds how deep recovery *dispatch* nests.
|
Bounds how deep recovery *dispatch* nests.
|
||||||
- **Global recovery/retry budget** — a new env-overridable constant
|
- **Global recovery/retry budget** — a new env-overridable constant
|
||||||
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default **25**) — the aggregate ceiling
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default **25**) — the aggregate ceiling
|
||||||
on the **total count of recovery + retry iterations across the whole
|
on the **total count of recovery + retry iterations across the whole
|
||||||
execution**. This is the case the two caps above do NOT bound: many steps each
|
execution**. This is the case the two caps above do NOT bound: many steps each
|
||||||
retrying up to 3× can sum unbounded. It primarily backstops the **retry
|
using up to 3 fix rounds can sum unbounded. It primarily backstops the **fix-round
|
||||||
aggregate** (recovery dispatch is already hard-capped at 2). Surfaced and
|
aggregate** (recovery dispatch is already hard-capped at 2). Reaching 0 stops
|
||||||
|
the run (stop class 4: a plan that needs that many fixes is guesswork). Surfaced and
|
||||||
enforced via the `iterations_remaining` counter (Phase 3 schema; decremented
|
enforced via the `iterations_remaining` counter (Phase 3 schema; decremented
|
||||||
on every recovery/retry iteration; reconciled at the Phase 7.5 completion
|
on every recovery/retry iteration; reconciled at the Phase 7.5 completion
|
||||||
gate).
|
gate).
|
||||||
|
|
@ -1787,15 +2129,37 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
||||||
**Type:** {plan | session-spec}
|
**Type:** {plan | session-spec}
|
||||||
**Mode:** {execute | resume | step N}
|
**Mode:** {execute | resume | step N}
|
||||||
**Result:** {COMPLETED | FAILED at step N | STOPPED (escalation) | PARTIAL (N/total passed)}
|
**Result:** {COMPLETED | FAILED at step N | STOPPED (escalation) | PARTIAL (N/total passed)}
|
||||||
|
**End to end:** {verified | not verified end to end}
|
||||||
|
|
||||||
### Step Results
|
### Step Results
|
||||||
|
|
||||||
| Step | Description | Result | Attempts | Commit | Manifest |
|
| Step | Description | Result | Red proof | Fix rounds | Test lock | Done line | Attempts | Commit | Manifest |
|
||||||
|------|-------------|--------|----------|--------|----------|
|
|------|-------------|--------|-----------|------------|-----------|-----------|----------|--------|----------|
|
||||||
| 0 | Sandbox pre-flight | PASS | 1 | — | n/a |
|
| 0 | Sandbox pre-flight | PASS | n/a | 0 | n/a | n/a | 1 | — | n/a |
|
||||||
| 1 | {desc} | PASS | 1 | abc1234 | pass |
|
| 1 | {desc} | PASS | proven | 0 | released | yes | 1 | abc1234 | pass |
|
||||||
| 2 | {desc} | FAIL | 3 | — | — |
|
| 2 | {desc} | FAIL | rejected | 4 (sent to the plan) | released | no | 4 | — | — |
|
||||||
| 3 | {desc} | — | 0 | — | — |
|
| 3 | {desc} | — | — | 0 | — | — | 0 | — | — |
|
||||||
|
|
||||||
|
Steps without proof of red: {N} ({step numbers}; below plan_version 1.8 only)
|
||||||
|
|
||||||
|
### Controller verdicts
|
||||||
|
|
||||||
|
Per step and round: spec {pass | fail}, quality {pass | fail}, findings (file:line — problem),
|
||||||
|
floor-guard rulings, and what the controller did not judge. Report paths: {report_path per step}.
|
||||||
|
|
||||||
|
### Stops
|
||||||
|
|
||||||
|
{none — the run ended without a stop | the stop that ended the run, with its class from the
|
||||||
|
closed stop list (Phase 5.5): 1 cannot be undone · 2 security-sensitive · 3 outside the working
|
||||||
|
copy · 4 plan so broken that everything after it is guesswork, and the file or command that shows it}
|
||||||
|
|
||||||
|
### Decisions taken
|
||||||
|
|
||||||
|
| Step | Decision | Cost if wrong |
|
||||||
|
|------|----------|---------------|
|
||||||
|
| {N} | {what the executor or a helper decided that the plan did not dictate} | {what must be redone, by whom, how big} |
|
||||||
|
|
||||||
|
{none, when the plan dictated every call}
|
||||||
|
|
||||||
### Plan Verification (Phase 7, trekplans only)
|
### Plan Verification (Phase 7, trekplans only)
|
||||||
|
|
||||||
|
|
@ -1819,8 +2183,12 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
||||||
- Not reached: {N}
|
- Not reached: {N}
|
||||||
- Blocked (sandbox): {N}
|
- Blocked (sandbox): {N}
|
||||||
|
|
||||||
{if all passed + exit condition passed + plan verification passed}:
|
{if all passed + exit condition passed + plan verification passed + end to end verified}:
|
||||||
All steps completed. Exit condition: PASS. Plan verification: PASS.
|
All steps completed. Exit condition: PASS. Plan verification: PASS. End to end: verified.
|
||||||
|
|
||||||
|
{if all passed but end_to_end = not-verified}:
|
||||||
|
All steps passed their checks, but the plan has no passing run-the-thing criterion:
|
||||||
|
not verified end to end. Result: PARTIAL.
|
||||||
|
|
||||||
{if failed/stopped}:
|
{if failed/stopped}:
|
||||||
### Failure Details
|
### Failure Details
|
||||||
|
|
@ -1842,7 +2210,8 @@ To continue (multi-session waves): re-run the same /trekexecute command (not --r
|
||||||
- `completed` — all steps passed AND Phase 7.5 manifest audit passed AND
|
- `completed` — all steps passed AND Phase 7.5 manifest audit passed AND
|
||||||
(for trekplans) Phase 7 plan verification passed
|
(for trekplans) Phase 7 plan verification passed
|
||||||
- `partial` — steps passed per executor but Phase 7.5 found drift, OR
|
- `partial` — steps passed per executor but Phase 7.5 found drift, OR
|
||||||
Phase 7.6 recovery incomplete
|
Phase 7.6 recovery incomplete, OR the run is not verified end to end
|
||||||
|
(Phase 7, gate clause 4)
|
||||||
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
|
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
|
||||||
- `failed` — a step failed and On failure was revert/retry (retry cap hit)
|
- `failed` — a step failed and On failure was revert/retry (retry cap hit)
|
||||||
- `stopped` — On failure: escalate triggered
|
- `stopped` — On failure: escalate triggered
|
||||||
|
|
@ -1871,7 +2240,11 @@ To continue (multi-session waves): re-run the same /trekexecute command (not --r
|
||||||
"recovery_depth": 0,
|
"recovery_depth": 0,
|
||||||
"iterations_remaining": 25,
|
"iterations_remaining": 25,
|
||||||
"legacy_plan": false,
|
"legacy_plan": false,
|
||||||
"progress_file": "{path}"
|
"progress_file": "{path}",
|
||||||
|
"end_to_end": "{verified | not-verified | n/a}",
|
||||||
|
"steps_without_red_proof": 0,
|
||||||
|
"stop_class": "{none | 1 | 2 | 3 | 4}",
|
||||||
|
"decisions": 0
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
@ -1968,9 +2341,10 @@ strategy (gates auto-escalation, parallel-wave choice — see High-effort
|
||||||
behavior below). The resolver does NOT control the orchestrator's own
|
behavior below). The resolver does NOT control the orchestrator's own
|
||||||
model — that is fixed at invocation time (command frontmatter omits
|
model — that is fixed at invocation time (command frontmatter omits
|
||||||
`model:`, so it follows the session model) and cannot be switched mid-turn.
|
`model:`, so it follows the session model) and cannot be switched mid-turn.
|
||||||
`/trekexecute` spawns no sub-agent swarm (Hard Rule 10), so
|
`/trekexecute` spawns no swarm: one step helper at a time and one controller
|
||||||
`phase_signal_result.model` has no spawn site here; it is returned for
|
(Hard Rule 10), and both run `opus` — the step agent's pin and the
|
||||||
cross-command uniformity and stats.
|
controller's `model: "opus"` — so `phase_signal_result.model` still has no
|
||||||
|
spawn site here; it is returned for cross-command uniformity and stats.
|
||||||
|
|
||||||
For `/trekexecute` specifically: `effort == 'low'` activates `--gates open`
|
For `/trekexecute` specifically: `effort == 'low'` activates `--gates open`
|
||||||
+ sequential-only execution (no worktree-isolated parallel waves — runs
|
+ sequential-only execution (no worktree-isolated parallel waves — runs
|
||||||
|
|
@ -2022,7 +2396,9 @@ code-path.
|
||||||
|
|
||||||
1. **No AskUserQuestion for execution decisions.** All execution decisions come
|
1. **No AskUserQuestion for execution decisions.** All execution decisions come
|
||||||
from the plan's On failure clauses. If the plan says escalate, stop and
|
from the plan's On failure clauses. If the plan says escalate, stop and
|
||||||
report — never ask.
|
report — never ask. What the plan leaves open, the executor decides and
|
||||||
|
writes into the decision log with its `cost_if_wrong`; it stops only on the
|
||||||
|
closed stop list (Phase 5.5).
|
||||||
|
|
||||||
2. **No scope creep.** Only touch files listed in the step's `Files:` field.
|
2. **No scope creep.** Only touch files listed in the step's `Files:` field.
|
||||||
If a file outside the list seems to need changing, record it as a finding
|
If a file outside the list seems to need changing, record it as a finding
|
||||||
|
|
@ -2034,7 +2410,10 @@ code-path.
|
||||||
4. **Fresh verification.** Re-run the Verify command from scratch every time.
|
4. **Fresh verification.** Re-run the Verify command from scratch every time.
|
||||||
Never trust cached or prior results.
|
Never trust cached or prior results.
|
||||||
|
|
||||||
5. **Retry cap = 3 attempts.** Initial + 2 retries, then stop. Never loop forever.
|
5. **Fix rounds go up — maximum 5 fix rounds per step, escalating upward.**
|
||||||
|
Rounds 1–3 are fixes by a fresh helper, round 4 sends the step back to
|
||||||
|
planning, round 5 to the operator (Sub-step F). Never a fourth guess at the
|
||||||
|
same level, never a cheaper model, never a loop.
|
||||||
|
|
||||||
6. **Never corrupt completed work.** Only revert files from the failing step.
|
6. **Never corrupt completed work.** Only revert files from the failing step.
|
||||||
Never touch files from earlier passed steps.
|
Never touch files from earlier passed steps.
|
||||||
|
|
@ -2047,11 +2426,15 @@ code-path.
|
||||||
|
|
||||||
9. **Progress file is ground truth.** Resume uses the progress file, not git log.
|
9. **Progress file is ground truth.** Resume uses the progress file, not git log.
|
||||||
|
|
||||||
10. **No sub-agents.** The executor reads and implements directly.
|
10. **One fresh helper per plan step, one controller, nothing more.** The
|
||||||
No Agent tool, no TeamCreate, no delegation. Enforced (not just
|
executor writes no product code: each step is built by a fresh
|
||||||
documented) via `disallowed-tools: Agent, TeamCreate` in the frontmatter,
|
`voyage:step-implementer` (Sub-step C) and judged by one controller, a
|
||||||
which removes both tools from the pool — `allowed-tools` alone does not,
|
general-purpose helper that reads the diff (Sub-step E).
|
||||||
since omission leaves a tool callable (CC 2.1.152).
|
Helpers never start helpers: the step agent has no Agent tool, and the
|
||||||
|
controller is told it changes nothing. Never the Agent tool's `name` (a named spawn's
|
||||||
|
result never returns), never `isolation` (measured: wrong base). No
|
||||||
|
TeamCreate — kept out via `disallowed-tools: TeamCreate`, since omission
|
||||||
|
from `allowed-tools` alone leaves a tool callable (CC 2.1.152).
|
||||||
|
|
||||||
11. **Worktree isolation is mandatory for multi-session execution.** Every
|
11. **Worktree isolation is mandatory for multi-session execution.** Every
|
||||||
session order names its own git worktree. Never run two or more sessions
|
session order names its own git worktree. Never run two or more sessions
|
||||||
|
|
@ -2123,14 +2506,49 @@ code-path.
|
||||||
failure (worktree removed, branch deleted, work lost) into a recoverable
|
failure (worktree removed, branch deleted, work lost) into a recoverable
|
||||||
one (push succeeded, branch preserved on remote). Source: research/02 R3.
|
one (push succeeded, branch preserved on remote). Source: research/02 R3.
|
||||||
|
|
||||||
20. **Three distinct iteration axes.** Per-step `attempts` (≤3), per-session
|
20. **Three distinct iteration axes.** Per-step `fix_rounds` (≤5, upward),
|
||||||
`recovery_depth` (<2), and the global `TREKEXECUTE_MAX_RECOVERY_ITERATIONS`
|
per-session `recovery_depth` (<2), and the global
|
||||||
budget (default 25 — aggregate recovery+retry backstop) are independent
|
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` budget (default 25 — aggregate
|
||||||
bounds, not one nested counter. Every recovery/retry loop is bounded by at
|
fix-round + recovery backstop) are independent bounds, not one nested
|
||||||
least one of them. See "Iteration caps + global recovery/retry budget".
|
counter. Every recovery/retry loop is bounded by at least one of them. See
|
||||||
|
"Iteration caps + global recovery/retry budget".
|
||||||
|
|
||||||
21. **Never start Claude Code.** No `claude` CLI invocation, no Agent SDK, no
|
21. **Never start Claude Code.** No `claude` CLI invocation, no Agent SDK, no
|
||||||
background process that runs a session — not for waves, not for recovery,
|
background process that runs a session — not for waves, not for recovery,
|
||||||
not for a test. Work that needs another session becomes a session order
|
not for a test. Work that needs another session becomes a session order
|
||||||
plus a starter line the user gives a session they open themselves
|
plus a starter line the user gives a session they open themselves
|
||||||
(Phase 2.6, Phase 7.6).
|
(Phase 2.6, Phase 7.6).
|
||||||
|
|
||||||
|
22. **Proof, not word.** Red before green, an unweakened test and "done" are
|
||||||
|
each a command's exit code — `red-proof.mjs`, `test-integrity.mjs`,
|
||||||
|
`done-line.mjs` — never the helper's or the executor's statement. Exit 2
|
||||||
|
from any of them ("could not run") is never green.
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
Every borrowing in this executor is named here, never silent. All sources are
|
||||||
|
MIT-licensed open source except Kiro (a closed product: only its published
|
||||||
|
documentation was read) and Cherny (public talks and posts). Read 2026-10-06;
|
||||||
|
pins and quotes in the local practice analysis.
|
||||||
|
|
||||||
|
- **Superpowers** — adapted from obra/superpowers@8ca22db (MIT, © 2025 Jesse
|
||||||
|
Vincent): `subagent-driven-development` — a fresh helper per task, hand-over
|
||||||
|
in files, a review package from a recorded start point, a reviewer between
|
||||||
|
tasks, the stop classes and the decision log of an unattended run (P8, P4);
|
||||||
|
`executing-plans/scripts/task-done` — the completion line appended only when
|
||||||
|
the test command exits 0, here `done-line.mjs` (P9 (4));
|
||||||
|
`systematic-debugging` — after three failed fixes, question the
|
||||||
|
architecture (P9 (7)). Not borrowed: its ladder of cheaper models.
|
||||||
|
- **ai-native-sdlc** — bashebr/ai-native-sdlc@11fb8d7 (MIT): the helper's
|
||||||
|
report contract, ran / saw / did NOT check (P8).
|
||||||
|
- **Kiro** (documentation only): the task runner takes the diff, not the
|
||||||
|
agent's account of it, as the reviewer's input (P8). Its background daemon is
|
||||||
|
not borrowed; only the step loop.
|
||||||
|
- **GitHub Spec Kit** (MIT), github/spec-kit@2e941fe: test-first as a rule,
|
||||||
|
mechanized here as `red-proof.mjs` (P9 (1)).
|
||||||
|
- **agent-skills** — addyosmani/agent-skills@9d0c60d (MIT): the floor guard of
|
||||||
|
constraint-driven development, as `test-integrity.mjs` (P9 (2)).
|
||||||
|
- **Cherny** (public primary sources): "run the thing" — a plan is verified
|
||||||
|
end to end only by a criterion that runs what was built (P9 (6)).
|
||||||
|
- **Claude Code** itself: subagents for the helper and the controller; the
|
||||||
|
`PreToolUse` guard behind the test lock (K8).
|
||||||
|
|
|
||||||
|
|
@ -270,7 +270,7 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
||||||
|
|
||||||
## Handover 5 — `progress.json` (resume contract)
|
## Handover 5 — `progress.json` (resume contract)
|
||||||
|
|
||||||
**Producer:** `/trekexecute` per-step (after Verify + Manifest audit + Checkpoint).
|
**Producer:** `/trekexecute` per-step (start point recorded before the step; then proof of red + Verify + Manifest audit + floor guard + control + done line + Checkpoint).
|
||||||
|
|
||||||
**Consumer:** `/trekexecute --resume` (re-entry) + `pre-compact-flush` hook (drift detection before context compaction).
|
**Consumer:** `/trekexecute --resume` (re-entry) + `pre-compact-flush` hook (drift detection before context compaction).
|
||||||
|
|
||||||
|
|
@ -296,6 +296,8 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
||||||
| `session_start_sha` | string | optional | git sha at execute start |
|
| `session_start_sha` | string | optional | git sha at execute start |
|
||||||
| `session_end_sha` | string | optional | git sha at execute end |
|
| `session_end_sha` | string | optional | git sha at execute end |
|
||||||
| `steps` | object | yes | Map of step number → step record |
|
| `steps` | object | yes | Map of step number → step record |
|
||||||
|
| `end_to_end` | string | optional | `verified \| not-verified` — a passing run-the-thing criterion in the plan's `## Verification`; `not-verified` turns `completed` into `partial` (wave 1, P9 (6)) |
|
||||||
|
| `decisions` | array | optional | `[{step, decision, cost_if_wrong}]` — every call the executor made that the plan did not dictate; both strings non-empty (wave 1, P4) |
|
||||||
|
|
||||||
**Per-step record:**
|
**Per-step record:**
|
||||||
|
|
||||||
|
|
@ -308,6 +310,15 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
||||||
| `commit` | string \| null | git sha after Checkpoint |
|
| `commit` | string \| null | git sha after Checkpoint |
|
||||||
| `manifest_audit` | string | `pass \| fail \| pass-with-note \| n/a` |
|
| `manifest_audit` | string | `pass \| fail \| pass-with-note \| n/a` |
|
||||||
| `note` | string | optional human-readable annotation |
|
| `note` | string | optional human-readable annotation |
|
||||||
|
| `start_sha` | string \| null | optional (wave 1) — git sha the step started from; every check of the step reads `git diff {start_sha}` |
|
||||||
|
| `report_path` | string \| null | optional (wave 1) — the step's report file: the helper's ran / saw / did-NOT-check rounds and the script-written `DONE step N` line |
|
||||||
|
| `red_proof` | string \| null | optional (wave 1) — `proven \| rejected \| not-run`, the exit of `lib/verification/red-proof.mjs` |
|
||||||
|
| `test_lock` | string \| null | optional (wave 1) — `locked \| released \| not-locked` (K8, `lib/util/test-lock.mjs`) |
|
||||||
|
| `fix_rounds` | number | optional (wave 1) — 0..5: 0 = passed its first control, 1–3 = fix rounds, 4 = sent up to the plan, 5 = sent up to the operator. 6 is invalid |
|
||||||
|
|
||||||
|
The wave-1 fields are **additive-optional on read**: a `progress.json` written
|
||||||
|
before them validates and resumes unchanged (no `schema_version` bump). `null`
|
||||||
|
means "not set yet". When present, `progress-validator.mjs` shape-checks them.
|
||||||
|
|
||||||
**Validation strategy:** `progress-validator.mjs` runs at:
|
**Validation strategy:** `progress-validator.mjs` runs at:
|
||||||
1. `/trekexecute --validate` (alongside plan-validator)
|
1. `/trekexecute --validate` (alongside plan-validator)
|
||||||
|
|
@ -325,6 +336,9 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
||||||
- `PROGRESS_STEP_RANGE` → `current_step` outside `[0, total_steps]`; resume halts
|
- `PROGRESS_STEP_RANGE` → `current_step` outside `[0, total_steps]`; resume halts
|
||||||
- `PROGRESS_ALREADY_DONE` → `status === completed`; nothing to resume
|
- `PROGRESS_ALREADY_DONE` → `status === completed`; nothing to resume
|
||||||
- `PROGRESS_STEP_COUNT_MISMATCH` → warning; not a blocker
|
- `PROGRESS_STEP_COUNT_MISMATCH` → warning; not a blocker
|
||||||
|
- `PROGRESS_FIX_ROUNDS_INVALID` → `fix_rounds` not an integer 0..5; resume halts
|
||||||
|
- `PROGRESS_START_SHA_INVALID` / `PROGRESS_REPORT_PATH_INVALID` / `PROGRESS_RED_PROOF_INVALID` / `PROGRESS_TEST_LOCK_INVALID` → a wave-1 step field of the wrong shape; resume halts
|
||||||
|
- `PROGRESS_DECISION_INVALID` → `decisions` is not an array of `{decision, cost_if_wrong}`; resume halts
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -73,10 +73,11 @@ native `effort:`.
|
||||||
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
||||||
selected agents, overriding the session level:
|
selected agents, overriding the session level:
|
||||||
|
|
||||||
- **Eight agents pin `high`:** the retrieval agents `task-finder`,
|
- **Nine agents pin `high`:** the retrieval agents `task-finder`,
|
||||||
`git-historian`, `dependency-tracer`, `architecture-mapper`, and the
|
`git-historian`, `dependency-tracer`, `architecture-mapper`, the
|
||||||
adversarial-reasoning agents `plan-critic`, `risk-assessor`,
|
adversarial-reasoning agents `plan-critic`, `risk-assessor`,
|
||||||
`contrarian-researcher`, `review-coordinator`. No agent pins below `high`
|
`contrarian-researcher`, `review-coordinator`, and `/trekexecute`'s step
|
||||||
|
helper `step-implementer`. No agent pins below `high`
|
||||||
(operator rule, 2026-10-05: standard is Opus 5.5 at effort High).
|
(operator rule, 2026-10-05: standard is Opus 5.5 at effort High).
|
||||||
- **All other agents:** unset → inherits the session's effort level (a session
|
- **All other agents:** unset → inherits the session's effort level (a session
|
||||||
that nothing sets runs at its model's default — `medium` on Opus 5.5).
|
that nothing sets runs at its model's default — `medium` on Opus 5.5).
|
||||||
|
|
|
||||||
|
|
@ -6,6 +6,9 @@
|
||||||
// PROGRESS_REQUIRED_TOP, so legacy progress.json without it still validates. The field's
|
// PROGRESS_REQUIRED_TOP, so legacy progress.json without it still validates. The field's
|
||||||
// liveness (never-decremented) is enforced deterministically by the Phase 7.5 completion-gate
|
// liveness (never-decremented) is enforced deterministically by the Phase 7.5 completion-gate
|
||||||
// cross-check in trekexecute, not by this per-validation shape check.
|
// cross-check in trekexecute, not by this per-validation shape check.
|
||||||
|
// The same holds for the wave-1 step-loop fields (per step: start_sha, report_path,
|
||||||
|
// red_proof, test_lock, fix_rounds 0..5; per run: decisions): shape-checked when
|
||||||
|
// present, never required.
|
||||||
|
|
||||||
import { readFileSync, existsSync } from 'node:fs';
|
import { readFileSync, existsSync } from 'node:fs';
|
||||||
import { issue, fail } from '../util/result.mjs';
|
import { issue, fail } from '../util/result.mjs';
|
||||||
|
|
@ -17,6 +20,45 @@ export const PROGRESS_VALID_STATUSES = ['pending', 'in_progress', 'completed', '
|
||||||
// know (lib/util/session-orders.mjs ENDED_BAD).
|
// know (lib/util/session-orders.mjs ENDED_BAD).
|
||||||
export const PROGRESS_STEP_STATUSES = ['completed', 'in_progress', 'failed', 'pending', 'deferred', 'skipped', 'blocked'];
|
export const PROGRESS_STEP_STATUSES = ['completed', 'in_progress', 'failed', 'pending', 'deferred', 'skipped', 'blocked'];
|
||||||
|
|
||||||
|
// Wave 1 (P8/P9) per-step fields, all optional on read so an older progress.json
|
||||||
|
// still resumes; `null` means "not set yet". fix_rounds is the step's place on
|
||||||
|
// the upward ladder: 0 = passed its first control, 1-3 = fix rounds, 4 = sent up
|
||||||
|
// to the plan, 5 = sent up to the operator. There is no sixth round.
|
||||||
|
export const PROGRESS_MAX_FIX_ROUNDS = 5;
|
||||||
|
export const PROGRESS_RED_PROOF_VALUES = ['proven', 'rejected', 'not-run'];
|
||||||
|
export const PROGRESS_TEST_LOCK_VALUES = ['locked', 'released', 'not-locked'];
|
||||||
|
|
||||||
|
function checkStepFields(k, s, errors) {
|
||||||
|
const set = (v) => v !== undefined && v !== null;
|
||||||
|
if (s.fix_rounds !== undefined &&
|
||||||
|
!(Number.isInteger(s.fix_rounds) && s.fix_rounds >= 0 && s.fix_rounds <= PROGRESS_MAX_FIX_ROUNDS)) {
|
||||||
|
errors.push(issue('PROGRESS_FIX_ROUNDS_INVALID',
|
||||||
|
`steps["${k}"].fix_rounds=${JSON.stringify(s.fix_rounds)} must be an integer 0..${PROGRESS_MAX_FIX_ROUNDS}`));
|
||||||
|
}
|
||||||
|
if (set(s.start_sha) && !(typeof s.start_sha === 'string' && /^[0-9a-f]{7,64}$/.test(s.start_sha))) {
|
||||||
|
errors.push(issue('PROGRESS_START_SHA_INVALID', `steps["${k}"].start_sha must be a git sha, got ${JSON.stringify(s.start_sha)}`));
|
||||||
|
}
|
||||||
|
if (set(s.report_path) && !(typeof s.report_path === 'string' && s.report_path.trim() !== '')) {
|
||||||
|
errors.push(issue('PROGRESS_REPORT_PATH_INVALID', `steps["${k}"].report_path must be a non-empty path`));
|
||||||
|
}
|
||||||
|
if (set(s.red_proof) && !PROGRESS_RED_PROOF_VALUES.includes(s.red_proof)) {
|
||||||
|
errors.push(issue('PROGRESS_RED_PROOF_INVALID', `steps["${k}"].red_proof "${s.red_proof}" not in [${PROGRESS_RED_PROOF_VALUES.join(', ')}]`));
|
||||||
|
}
|
||||||
|
if (set(s.test_lock) && !PROGRESS_TEST_LOCK_VALUES.includes(s.test_lock)) {
|
||||||
|
errors.push(issue('PROGRESS_TEST_LOCK_INVALID', `steps["${k}"].test_lock "${s.test_lock}" not in [${PROGRESS_TEST_LOCK_VALUES.join(', ')}]`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The run-level decision log (P4): every decision the executor took on its own,
|
||||||
|
// with what it costs if it is wrong. Optional; when present, every entry carries both.
|
||||||
|
function checkDecisions(decisions, errors) {
|
||||||
|
if (decisions === undefined) return;
|
||||||
|
const ok = Array.isArray(decisions) && decisions.every((d) => d && typeof d === 'object' &&
|
||||||
|
typeof d.decision === 'string' && d.decision.trim() !== '' &&
|
||||||
|
typeof d.cost_if_wrong === 'string' && d.cost_if_wrong.trim() !== '');
|
||||||
|
if (!ok) errors.push(issue('PROGRESS_DECISION_INVALID', 'decisions must be an array of {decision, cost_if_wrong} with both non-empty'));
|
||||||
|
}
|
||||||
|
|
||||||
export function validateProgressContent(jsonText, opts = {}) {
|
export function validateProgressContent(jsonText, opts = {}) {
|
||||||
let parsed;
|
let parsed;
|
||||||
try { parsed = JSON.parse(jsonText); }
|
try { parsed = JSON.parse(jsonText); }
|
||||||
|
|
@ -82,9 +124,12 @@ export function validateProgressObject(parsed, opts = {}) {
|
||||||
if (s.status !== undefined && !PROGRESS_STEP_STATUSES.includes(s.status)) {
|
if (s.status !== undefined && !PROGRESS_STEP_STATUSES.includes(s.status)) {
|
||||||
warnings.push(issue('PROGRESS_STEP_BAD_STATUS', `steps["${k}"].status "${s.status}" unrecognized`));
|
warnings.push(issue('PROGRESS_STEP_BAD_STATUS', `steps["${k}"].status "${s.status}" unrecognized`));
|
||||||
}
|
}
|
||||||
|
checkStepFields(k, s, errors);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
checkDecisions(parsed.decisions, errors);
|
||||||
|
|
||||||
return { valid: errors.length === 0, errors, warnings, parsed };
|
return { valid: errors.length === 0, errors, warnings, parsed };
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
96
lib/verification/done-line.mjs
Normal file
96
lib/verification/done-line.mjs
Normal file
|
|
@ -0,0 +1,96 @@
|
||||||
|
// lib/verification/done-line.mjs
|
||||||
|
// The done line (P9 (4)). A step is "done" only when this script has written
|
||||||
|
// its line, and it writes the line only when the test command exits 0:
|
||||||
|
//
|
||||||
|
// DONE step <N>: `<command>` exit 0 at <HEAD sha> <ISO time>
|
||||||
|
//
|
||||||
|
// appended to the step's report file. Exit 0 = written, 1 = the command failed
|
||||||
|
// (nothing written), 2 = could not run (nothing written; never green): no
|
||||||
|
// --step, a command outside the test-runner allowlist, a command that cannot
|
||||||
|
// start, a working tree that is not a git repo.
|
||||||
|
//
|
||||||
|
// `--check --step <N> --report <file>`: exit 0 when the report carries a
|
||||||
|
// well-formed line for step N, 1 when it does not. A hand-typed "DONE" is not
|
||||||
|
// the script's line.
|
||||||
|
//
|
||||||
|
// Adapted from obra/superpowers@8ca22db (MIT, (c) 2025 Jesse Vincent),
|
||||||
|
// skills/executing-plans/scripts/task-done: the completion line is appended
|
||||||
|
// only when the test command exits 0.
|
||||||
|
|
||||||
|
import { appendFileSync, existsSync, readFileSync, realpathSync } from 'node:fs';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { resolve } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||||
|
import { commandEnv, DEFAULT_TIMEOUT_MS, repoRoot } from './red-proof.mjs';
|
||||||
|
|
||||||
|
export function doneLineRe(step) {
|
||||||
|
return new RegExp(`^DONE step ${step}: \`.+\` exit 0 at [0-9a-f]{40} \\d{4}-\\d{2}-\\d{2}T\\S+$`, 'm');
|
||||||
|
}
|
||||||
|
|
||||||
|
export function hasDoneLine(reportPath, step) {
|
||||||
|
if (!existsSync(reportPath)) return false;
|
||||||
|
return doneLineRe(step).test(readFileSync(reportPath, 'utf8'));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** @returns {{status: 'written'|'failed'|'not-run', reason: string, line?: string, exit?: number|null}} */
|
||||||
|
export function writeDoneLine({ cwd, step, cmd, report, timeoutMs = DEFAULT_TIMEOUT_MS }) {
|
||||||
|
if (!/^\d+$/.test(String(step ?? ''))) return { status: 'not-run', reason: '--step must be a step number' };
|
||||||
|
if (!report) return { status: 'not-run', reason: '--report is required' };
|
||||||
|
// The plan's Verify command is relative to the repo root, as in red-proof.mjs.
|
||||||
|
const dir = repoRoot(resolve(cwd ?? process.cwd()));
|
||||||
|
if (!dir) return { status: 'not-run', reason: `${cwd ?? process.cwd()} is not inside a git repository` };
|
||||||
|
const allowed = allowedCommand(cmd, { cwd: dir });
|
||||||
|
if (!allowed.allowed) return { status: 'not-run', reason: `command not run: ${allowed.reason}` };
|
||||||
|
const screen = screenCommand(cmd);
|
||||||
|
if (!screen.allowed) return { status: 'not-run', reason: `command blocked: ${screen.rule}` };
|
||||||
|
const head = spawnSync('git', ['rev-parse', '--verify', '--quiet', 'HEAD'], { cwd: dir, encoding: 'utf8' });
|
||||||
|
if (head.status !== 0) return { status: 'not-run', reason: `no git HEAD in ${dir}` };
|
||||||
|
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd: dir, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||||
|
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
|
||||||
|
return { status: 'not-run', reason: `the command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`, exit: r.status };
|
||||||
|
}
|
||||||
|
if (r.status !== 0) return { status: 'failed', reason: `the test command exited ${r.status}`, exit: r.status };
|
||||||
|
const line = `DONE step ${step}: \`${cmd}\` exit 0 at ${head.stdout.trim()} ${new Date().toISOString()}`;
|
||||||
|
appendFileSync(report, `${line}\n`);
|
||||||
|
return { status: 'written', reason: 'the test command exited 0', line, exit: 0 };
|
||||||
|
}
|
||||||
|
|
||||||
|
const USAGE =
|
||||||
|
'usage: done-line.mjs --step <N> --cmd "<test command>" --report <file> [--cwd <dir>] [--json]\n' +
|
||||||
|
' done-line.mjs --check --step <N> --report <file>';
|
||||||
|
|
||||||
|
export function main(argv) {
|
||||||
|
const o = {};
|
||||||
|
let check = false;
|
||||||
|
let json = false;
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
if (a === '--step' && argv[i + 1]) o.step = argv[++i];
|
||||||
|
else if (a === '--cmd' && argv[i + 1]) o.cmd = argv[++i];
|
||||||
|
else if (a === '--report' && argv[i + 1]) o.report = resolve(argv[++i]);
|
||||||
|
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||||
|
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||||
|
else if (a === '--check') check = true;
|
||||||
|
else if (a === '--json') json = true;
|
||||||
|
else { process.stderr.write(`done-line: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||||
|
}
|
||||||
|
if (check) {
|
||||||
|
if (!/^\d+$/.test(String(o.step ?? '')) || !o.report) { process.stderr.write(`done-line: --check needs --step and --report\n${USAGE}\n`); return 2; }
|
||||||
|
const ok = hasDoneLine(o.report, o.step);
|
||||||
|
process.stdout.write(json ? `${JSON.stringify({ step: Number(o.step), done: ok })}\n` : `step ${o.step}: ${ok ? 'DONE line present' : 'no DONE line - not done'}\n`);
|
||||||
|
return ok ? 0 : 1;
|
||||||
|
}
|
||||||
|
let r;
|
||||||
|
try { r = writeDoneLine(o); }
|
||||||
|
catch (e) { r = { status: 'not-run', reason: `done-line crashed: ${e.message}` }; }
|
||||||
|
process.stdout.write(json ? `${JSON.stringify(r)}\n` : `done line: ${r.status.toUpperCase()} - ${r.line ?? r.reason}\n`);
|
||||||
|
return r.status === 'written' ? 0 : r.status === 'failed' ? 1 : 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isMain() {
|
||||||
|
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||||
|
catch { return false; }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||||
198
lib/verification/red-proof.mjs
Normal file
198
lib/verification/red-proof.mjs
Normal file
|
|
@ -0,0 +1,198 @@
|
||||||
|
// lib/verification/red-proof.mjs
|
||||||
|
// Proof of red (P9 (1)). A step's test is run by a command twice:
|
||||||
|
//
|
||||||
|
// 1. against the START POINT (the step's recorded start_sha, with the step's
|
||||||
|
// test files copied in): it must FAIL - otherwise the test proves nothing
|
||||||
|
// about the change;
|
||||||
|
// 2. against the RESULT (the working tree as the step left it): it must PASS.
|
||||||
|
//
|
||||||
|
// `--red-only` runs step 1 alone: the gate the step helper passes BEFORE it
|
||||||
|
// builds (and locks the test). The executor's own check afterwards is the full
|
||||||
|
// two-sided run.
|
||||||
|
//
|
||||||
|
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),
|
||||||
|
// 2 = could not run (`not-run`: no test files, an unknown start point, a command
|
||||||
|
// outside the test-runner allowlist, a command that cannot start). 2 is never a
|
||||||
|
// pass and never "proven".
|
||||||
|
//
|
||||||
|
// The start point is materialized with `git archive <start_sha>` into a temp
|
||||||
|
// directory, never by checking anything out in the working tree: the step's
|
||||||
|
// work is not touched, and no worktree is registered in the repo. Tracked files
|
||||||
|
// only - pass `--link node_modules` (comma list) to symlink an untracked
|
||||||
|
// dependency directory from the working tree into the start-point copy.
|
||||||
|
//
|
||||||
|
// The command must be ONE test-runner invocation without shell operators
|
||||||
|
// (criteria-runner.mjs `allowedCommand`), and it is screened through the
|
||||||
|
// plugin's own PreToolUse denylist (`screenCommand`): a command spawned from
|
||||||
|
// node never passes through the Bash tool, so the hook cannot fire by itself.
|
||||||
|
//
|
||||||
|
// When the output is TAP (`--test-reporter=tap`, pytest-tap, ...), the counts
|
||||||
|
// `# tests N` / `# fail N` are reported for both runs. They show WHAT the red
|
||||||
|
// was: "fail 1 of 1" on a node test file is usually a load error (an import the
|
||||||
|
// start point lacks), not named failing tests - see
|
||||||
|
// tests/fixtures/red-first/02243c6-always-allow-shim.mjs, re-measured through
|
||||||
|
// this tool in tests/lib/red-proof.test.mjs.
|
||||||
|
//
|
||||||
|
// Borrowed: GitHub Spec Kit (MIT, github/spec-kit@2e941fe) - its test-first
|
||||||
|
// rule is an instruction there; here a command proves it.
|
||||||
|
|
||||||
|
import { cpSync, existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||||
|
|
||||||
|
export const DEFAULT_TIMEOUT_MS = 900_000;
|
||||||
|
const TAIL = 2000;
|
||||||
|
|
||||||
|
function tap(output) {
|
||||||
|
const tests = String(output).match(/^# tests (\d+)$/m);
|
||||||
|
const fail = String(output).match(/^# fail (\d+)$/m);
|
||||||
|
return tests && fail ? { tests: Number(tests[1]), fail: Number(fail[1]) } : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
// The command's exit code must be its own. Under `node --test` a child inherits
|
||||||
|
// NODE_TEST_CONTEXT, and a nested `node --test` then reports to that parent and
|
||||||
|
// exits 0 whatever its tests did (measured 2026-10-06: a red start point read as
|
||||||
|
// green). So the variable never reaches the command.
|
||||||
|
export function commandEnv(env = process.env) {
|
||||||
|
const { NODE_TEST_CONTEXT, ...rest } = env;
|
||||||
|
return rest;
|
||||||
|
}
|
||||||
|
|
||||||
|
function run(cmd, cwd, timeoutMs) {
|
||||||
|
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||||
|
const output = `${r.stdout ?? ''}${r.stderr ?? ''}`;
|
||||||
|
return { exit: r.status, signal: r.signal ?? null, tap: tap(output), tail: output.slice(-TAIL), spawnError: r.error ? r.error.message : null };
|
||||||
|
}
|
||||||
|
|
||||||
|
const notRun = (reason, extra = {}) => ({ status: 'not-run', reason, ...extra });
|
||||||
|
|
||||||
|
// A shell that could not find or execute the command (126/127), a signal (the
|
||||||
|
// timeout), or a spawn error is not a test verdict at all.
|
||||||
|
function unrunnable(r) {
|
||||||
|
if (r.spawnError) return `the command could not start: ${r.spawnError}`;
|
||||||
|
if (r.exit === null) return `the command was killed (${r.signal ?? 'timeout'})`;
|
||||||
|
if (r.exit === 126 || r.exit === 127) return `the shell could not run the command (exit ${r.exit})`;
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], timeoutMs?: number}} o
|
||||||
|
* @returns {{status: 'proven'|'rejected'|'not-run', reason: string, base?: object, result?: object}}
|
||||||
|
*/
|
||||||
|
export function repoRoot(dir) {
|
||||||
|
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: dir, encoding: 'utf8' });
|
||||||
|
return top.status === 0 ? top.stdout.trim() : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function proveRed(o) {
|
||||||
|
const given = resolve(o.cwd ?? process.cwd());
|
||||||
|
const timeoutMs = o.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||||
|
const tests = (o.tests ?? []).filter(Boolean);
|
||||||
|
if (!existsSync(given)) return notRun(`--cwd ${given} does not exist`);
|
||||||
|
// Plan paths and commands are relative to the repo root, and `git archive`
|
||||||
|
// run from a subdirectory packs only that subdirectory (measured
|
||||||
|
// 2026-10-06). So everything below runs at the root of --cwd's repo.
|
||||||
|
const cwd = repoRoot(given);
|
||||||
|
if (!cwd) return notRun(`--cwd ${given} is not inside a git repository`);
|
||||||
|
if (tests.length === 0) return notRun('no test files given: a step without its test cannot prove red');
|
||||||
|
for (const t of tests) {
|
||||||
|
if (isAbsolute(t) || t.split('/').includes('..')) return notRun(`test path must be repo-relative: ${t}`);
|
||||||
|
if (!existsSync(join(cwd, t))) return notRun(`test file not found in the working tree: ${t}`);
|
||||||
|
}
|
||||||
|
const allowed = allowedCommand(o.cmd, { cwd });
|
||||||
|
if (!allowed.allowed) return notRun(`command not run: ${allowed.reason}`);
|
||||||
|
const screen = screenCommand(o.cmd);
|
||||||
|
if (!screen.allowed) return notRun(`command blocked: ${screen.rule}`);
|
||||||
|
const sha = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${o.base}^{commit}`], { cwd, encoding: 'utf8' });
|
||||||
|
if (sha.status !== 0) return notRun(`cannot resolve the start point ${o.base} in ${cwd}`);
|
||||||
|
const baseSha = sha.stdout.trim();
|
||||||
|
|
||||||
|
// The result first: it is the working tree as it stands. `redOnly` is the
|
||||||
|
// gate before building (the step helper's own check): the result is not run.
|
||||||
|
let result;
|
||||||
|
if (!o.redOnly) {
|
||||||
|
result = run(o.cmd, cwd, timeoutMs);
|
||||||
|
const resultBroken = unrunnable(result);
|
||||||
|
if (resultBroken) return notRun(`at the result: ${resultBroken}`, { result });
|
||||||
|
}
|
||||||
|
|
||||||
|
const tmp = mkdtempSync(join(tmpdir(), 'voyage-red-proof-'));
|
||||||
|
try {
|
||||||
|
const tree = join(tmp, 'tree');
|
||||||
|
mkdirSync(tree);
|
||||||
|
const tar = join(tmp, 'start.tar');
|
||||||
|
const a = spawnSync('git', ['archive', '--format=tar', '-o', tar, baseSha], { cwd, encoding: 'utf8' });
|
||||||
|
if (a.status !== 0) return notRun(`git archive ${baseSha} failed: ${String(a.stderr).trim()}`, { result });
|
||||||
|
const x = spawnSync('tar', ['-xf', tar, '-C', tree], { encoding: 'utf8' });
|
||||||
|
if (x.status !== 0) return notRun(`tar could not unpack the start point: ${String(x.stderr).trim()}`, { result });
|
||||||
|
for (const t of tests) {
|
||||||
|
mkdirSync(dirname(join(tree, t)), { recursive: true });
|
||||||
|
cpSync(join(cwd, t), join(tree, t));
|
||||||
|
}
|
||||||
|
for (const l of o.link ?? []) {
|
||||||
|
if (!l || isAbsolute(l) || l.split('/').includes('..')) return notRun(`--link must be repo-relative: ${l}`, { result });
|
||||||
|
if (existsSync(join(cwd, l)) && !existsSync(join(tree, l))) {
|
||||||
|
mkdirSync(dirname(join(tree, l)), { recursive: true });
|
||||||
|
symlinkSync(join(cwd, l), join(tree, l));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
const base = run(o.cmd, tree, timeoutMs);
|
||||||
|
const baseBroken = unrunnable(base);
|
||||||
|
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result });
|
||||||
|
|
||||||
|
const out = { base_sha: baseSha, base, result };
|
||||||
|
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...out };
|
||||||
|
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', base_sha: baseSha, base };
|
||||||
|
if (result.exit !== 0) return { status: 'rejected', reason: 'not-green-at-result', ...out };
|
||||||
|
return { status: 'proven', reason: 'red at the start point, green at the result', ...out };
|
||||||
|
} finally {
|
||||||
|
rmSync(tmp, { recursive: true, force: true });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- CLI ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const USAGE =
|
||||||
|
'usage: red-proof.mjs --base <start_sha> --cmd "<test command>" --tests <a,b> [--red-only] [--cwd <dir>] [--link <dir,...>] [--timeout <ms>] [--json]';
|
||||||
|
|
||||||
|
const list = (v) => String(v).split(',').map((s) => s.trim()).filter(Boolean);
|
||||||
|
|
||||||
|
export function main(argv) {
|
||||||
|
const o = { cwd: process.cwd(), tests: [], link: [] };
|
||||||
|
let json = false;
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
if (a === '--base' && argv[i + 1]) o.base = argv[++i];
|
||||||
|
else if (a === '--cmd' && argv[i + 1]) o.cmd = argv[++i];
|
||||||
|
else if (a === '--tests' && argv[i + 1]) o.tests = list(argv[++i]);
|
||||||
|
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||||
|
else if (a === '--link' && argv[i + 1]) o.link = list(argv[++i]);
|
||||||
|
else if (a === '--red-only') o.redOnly = true;
|
||||||
|
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||||
|
else if (a === '--json') json = true;
|
||||||
|
else { process.stderr.write(`red-proof: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||||
|
}
|
||||||
|
if (!o.base || !o.cmd) { process.stderr.write(`red-proof: --base and --cmd are required\n${USAGE}\n`); return 2; }
|
||||||
|
if (o.timeoutMs !== undefined && !Number.isFinite(o.timeoutMs)) { process.stderr.write(`red-proof: --timeout must be a number\n${USAGE}\n`); return 2; }
|
||||||
|
let r;
|
||||||
|
try { r = proveRed(o); }
|
||||||
|
catch (e) { r = notRun(`red-proof crashed: ${e.message}`); }
|
||||||
|
if (json) process.stdout.write(JSON.stringify(r, null, 2) + '\n');
|
||||||
|
else {
|
||||||
|
const counts = (x) => (x?.tap ? ` (tap: ${x.tap.fail} of ${x.tap.tests} failed)` : '');
|
||||||
|
process.stdout.write(`red proof: ${r.status.toUpperCase()} - ${r.reason}\n`);
|
||||||
|
if (r.base) process.stdout.write(` start point ${r.base_sha ?? o.base}: exit ${r.base.exit}${counts(r.base)}\n`);
|
||||||
|
if (r.result) process.stdout.write(` result: exit ${r.result.exit}${counts(r.result)}\n`);
|
||||||
|
}
|
||||||
|
return r.status === 'proven' ? 0 : r.status === 'rejected' ? 1 : 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isMain() {
|
||||||
|
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||||
|
catch { return false; }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||||
283
lib/verification/test-integrity.mjs
Normal file
283
lib/verification/test-integrity.mjs
Normal file
|
|
@ -0,0 +1,283 @@
|
||||||
|
// lib/verification/test-integrity.mjs
|
||||||
|
// The floor guard (P9 (2)): read a step's diff and report every change that
|
||||||
|
// makes the tests weaker instead of the code better.
|
||||||
|
//
|
||||||
|
// removed-assert a test file loses more assertions than it gains
|
||||||
|
// deleted-test a test file loses more test declarations than it gains
|
||||||
|
// skipped-test a test is skipped, marked todo/xfail, or focused (.only)
|
||||||
|
// muted-check a check is silenced: `|| true`, continue-on-error,
|
||||||
|
// --passWithNoTests, an inline lint/type suppression ...
|
||||||
|
// stub non-test code gains a "not implemented" body
|
||||||
|
// loosened-threshold a numeric bound in a test or a check config moves in the
|
||||||
|
// permissive direction (`ms < 200` -> `ms < 2000`)
|
||||||
|
//
|
||||||
|
// Exit 0 = clean, 1 = findings, 2 = could not run (no input, unreadable input,
|
||||||
|
// an unresolvable --base). 2 is never green: an absent measurement is reported
|
||||||
|
// as absent, not as a pass.
|
||||||
|
//
|
||||||
|
// A finding is not a verdict. /trekexecute hands every finding to its
|
||||||
|
// controller, which either quotes the plan line that asked for the change or
|
||||||
|
// fails the step; nothing is dropped silently.
|
||||||
|
//
|
||||||
|
// Borrowed: the floor guard of addyosmani/agent-skills@9d0c60d (MIT),
|
||||||
|
// constraint-driven development - mechanized here as a diff reader so a
|
||||||
|
// command, not the model's word, says whether a test was weakened.
|
||||||
|
|
||||||
|
import { readFileSync, realpathSync } from 'node:fs';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { resolve } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
// --- what counts as a test / a check config ----------------------------------
|
||||||
|
|
||||||
|
const TEST_PATH_RE = [
|
||||||
|
/(^|\/)(tests?|__tests__|spec|specs)\//,
|
||||||
|
/\.(test|spec)\.[a-z0-9]+$/i,
|
||||||
|
/_(test|spec)\.[a-z0-9]+$/i,
|
||||||
|
/(^|\/)test_[^/]*\.py$/,
|
||||||
|
];
|
||||||
|
|
||||||
|
const CHECK_CONFIG_RE =
|
||||||
|
/(^|\/)(\.github|\.forgejo|\.gitea|\.woodpecker)\/|(^|\/)(\.gitlab-ci\.yml|Makefile|package\.json|pyproject\.toml|setup\.cfg|tox\.ini|\.pre-commit-config\.yaml|tsconfig[^/]*\.json|\.eslintrc[^/]*|eslint\.config\.[a-z]+|(jest|vitest|playwright)\.config\.[a-z]+|\.c8rc[^/]*|\.nycrc[^/]*|codecov\.ya?ml)$/;
|
||||||
|
|
||||||
|
export function isTestPath(p) {
|
||||||
|
return TEST_PATH_RE.some((re) => re.test(p));
|
||||||
|
}
|
||||||
|
|
||||||
|
const isCheckPath = (p) => isTestPath(p) || CHECK_CONFIG_RE.test(p);
|
||||||
|
|
||||||
|
// --- the patterns -------------------------------------------------------------
|
||||||
|
|
||||||
|
const ASSERT_RE =
|
||||||
|
/\bassert\w*\b|\bexpect\s*\(|\.should\b|\bt\.(equal|is|ok|true|false|deepEqual|throws|notThrows|match)\b|\bself\.assert\w+|\brequire\.\w+\(|\bt\.(Error|Fatal)f?\(/g;
|
||||||
|
|
||||||
|
const TEST_DECL_RE =
|
||||||
|
/^\s*(test|it|describe|suite)(\.\w+)?\s*\(|^\s*(async\s+)?def\s+test_\w*|^\s*func\s+Test\w*|^\s*#\[test\]|@Test\b/;
|
||||||
|
|
||||||
|
const SKIP_RE =
|
||||||
|
/\b(test|it|describe|suite)\.(skip|todo|only)\s*\(|\bx(it|describe|test)\s*\(|\bf(it|describe)\s*\(|\{\s*(skip|todo)\s*:\s*(true|['"`])|@pytest\.mark\.(skip|skipif|xfail)\b|\bpytest\.skip\s*\(|@unittest\.skip|\bt\.Skip(Now|f)?\s*\(|#\[ignore\]|@(Disabled|Ignore)\b/;
|
||||||
|
|
||||||
|
// Inline suppressions silence a check wherever they appear.
|
||||||
|
const SUPPRESS_RE =
|
||||||
|
/@ts-ignore|@ts-nocheck|eslint-disable|#\s*type:\s*ignore|#\s*noqa\b|\/\/\s*nolint\b|#\s*pragma:\s*no\s*cover|istanbul\s+ignore|c8\s+ignore/;
|
||||||
|
|
||||||
|
// Exit-code mutes count only where a check runs: tests, scripts under tests/, CI
|
||||||
|
// and check configs. In ordinary code `|| true` is a normal shell idiom.
|
||||||
|
const MUTE_RE =
|
||||||
|
/\|\|\s*true\b|\|\|\s*:\s*($|[;)])|\|\|\s*exit\s+0\b|continue-on-error\s*:\s*true|--passWithNoTests|--pass-with-no-tests|\bset\s+\+e\b|--no-verify\b|allow_failure\s*:\s*true/;
|
||||||
|
|
||||||
|
const STUB_RE =
|
||||||
|
/throw\s+new\s+Error\s*\(\s*['"`]\s*(not\s+implemented|unimplemented|todo)\b|\braise\s+NotImplementedError\b|\b(unimplemented|todo)!\s*\(|panic\s*\(\s*"(not\s+implemented|todo)|(\/\/|#)\s*stub\b|TODO:?\s*implement/i;
|
||||||
|
|
||||||
|
// Number after a "smaller is stricter" bound: bigger = looser.
|
||||||
|
const UPPER_BOUND_RE = /(<=?|toBeLessThan(OrEqual)?|lessThan|\bmax\w*|timeout\w*|tolerance|epsilon|delta|within|retries)\W*$/i;
|
||||||
|
// Number after a "bigger is stricter" bound: smaller = looser.
|
||||||
|
const LOWER_BOUND_RE = /(>=?|toBeGreaterThan(OrEqual)?|greaterThan|\bmin\w*|threshold|coverage|branches|lines|functions|statements)\W*$/i;
|
||||||
|
|
||||||
|
const NUM_RE = /\d+(?:\.\d+)?/g;
|
||||||
|
|
||||||
|
// --- diff parsing ---------------------------------------------------------------
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Parse a unified diff into files with hunks of {type: ' '|'-'|'+', text, line}.
|
||||||
|
* Returns null when non-empty input carries no diff header at all.
|
||||||
|
*/
|
||||||
|
export function parseDiff(text) {
|
||||||
|
const src = String(text ?? '');
|
||||||
|
if (src.trim() === '') return [];
|
||||||
|
if (!/^(diff --git |--- |\+\+\+ )/m.test(src)) return null;
|
||||||
|
const files = [];
|
||||||
|
let file = null;
|
||||||
|
let hunk = null;
|
||||||
|
let oldLine = 0;
|
||||||
|
let newLine = 0;
|
||||||
|
// Lines still owed to the current hunk, from its @@ header. While a hunk is
|
||||||
|
// owed lines, every line is content - even one that reads "--- x" or
|
||||||
|
// "+++ y" (a removed "-- x" / an added "++ y"). Headers only between hunks.
|
||||||
|
let oldLeft = 0;
|
||||||
|
let newLeft = 0;
|
||||||
|
const lines = src.split('\n');
|
||||||
|
for (const l of lines) {
|
||||||
|
if (hunk && (oldLeft > 0 || newLeft > 0)) {
|
||||||
|
if (l.startsWith('\\')) continue; // "\ No newline at end of file"
|
||||||
|
if (l.startsWith('-')) { hunk.push({ type: '-', text: l.slice(1), line: oldLine++ }); oldLeft--; continue; }
|
||||||
|
if (l.startsWith('+')) { hunk.push({ type: '+', text: l.slice(1), line: newLine++ }); newLeft--; continue; }
|
||||||
|
if (l.startsWith(' ') || l === '') {
|
||||||
|
hunk.push({ type: ' ', text: l.slice(1), line: newLine });
|
||||||
|
oldLine++; newLine++; oldLeft--; newLeft--;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
// Anything else ends a malformed hunk early; fall through to headers.
|
||||||
|
oldLeft = 0; newLeft = 0;
|
||||||
|
}
|
||||||
|
if (l.startsWith('diff --git ')) {
|
||||||
|
const m = l.match(/^diff --git a\/(.+) b\/(.+)$/);
|
||||||
|
file = { path: m ? m[2] : l.slice(11), deleted: false, hunks: [] };
|
||||||
|
files.push(file);
|
||||||
|
hunk = null;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (l.startsWith('--- ')) {
|
||||||
|
if (!file || file.hunks.length > 0) { file = { path: '', deleted: false, hunks: [] }; files.push(file); }
|
||||||
|
const oldPath = l.slice(4).replace(/^a\//, '');
|
||||||
|
if (!file.path && oldPath !== '/dev/null') file.path = oldPath;
|
||||||
|
hunk = null;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (l.startsWith('+++ ') && file) {
|
||||||
|
const newPath = l.slice(4).replace(/^b\//, '');
|
||||||
|
if (newPath === '/dev/null') file.deleted = true;
|
||||||
|
else file.path = newPath;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const h = l.match(/^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@/);
|
||||||
|
if (h && file) {
|
||||||
|
oldLine = Number(h[1]);
|
||||||
|
newLine = Number(h[3]);
|
||||||
|
oldLeft = h[2] === undefined ? 1 : Number(h[2]);
|
||||||
|
newLeft = h[4] === undefined ? 1 : Number(h[4]);
|
||||||
|
hunk = [];
|
||||||
|
file.hunks.push(hunk);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return files;
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- detectors ------------------------------------------------------------------
|
||||||
|
|
||||||
|
const count = (lines, re) => lines.reduce((n, l) => n + (l.text.match(new RegExp(re.source, 'g')) || []).length, 0);
|
||||||
|
const norm = (s) => s.trim().replace(/\s+/g, ' ');
|
||||||
|
|
||||||
|
function addedNotRemoved(removed, added, re) {
|
||||||
|
const before = new Set(removed.filter((l) => re.test(l.text)).map((l) => norm(l.text)));
|
||||||
|
return added.filter((l) => re.test(l.text) && !before.has(norm(l.text)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A paired -/+ line whose only change is numbers; returns the looser numbers. */
|
||||||
|
function loosened(oldText, newText) {
|
||||||
|
const shape = (s) => norm(s).replace(NUM_RE, '#');
|
||||||
|
if (shape(oldText) !== shape(newText)) return null;
|
||||||
|
const a = [...norm(oldText).matchAll(NUM_RE)];
|
||||||
|
const b = [...norm(newText).matchAll(NUM_RE)];
|
||||||
|
for (let i = 0; i < a.length; i++) {
|
||||||
|
const before = Number(a[i][0]);
|
||||||
|
const after = Number(b[i][0]);
|
||||||
|
if (before === after) continue;
|
||||||
|
const lead = norm(newText).slice(Math.max(0, b[i].index - 30), b[i].index);
|
||||||
|
if (UPPER_BOUND_RE.test(lead) && after > before) return `${before} -> ${after}`;
|
||||||
|
if (LOWER_BOUND_RE.test(lead) && after < before) return `${before} -> ${after}`;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
function finding(kind, file, l, detail) {
|
||||||
|
return { kind, file, line: l ? l.line : null, text: l ? l.text.trim() : '', ...(detail ? { detail } : {}) };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function analyzeDiff(text) {
|
||||||
|
const files = parseDiff(text);
|
||||||
|
if (files === null) return { status: 'unreadable', findings: [], reason: 'the input is not a unified diff' };
|
||||||
|
const findings = [];
|
||||||
|
for (const f of files) {
|
||||||
|
const all = f.hunks.flat();
|
||||||
|
const removed = all.filter((l) => l.type === '-');
|
||||||
|
const added = all.filter((l) => l.type === '+');
|
||||||
|
const test = isTestPath(f.path);
|
||||||
|
|
||||||
|
if (test) {
|
||||||
|
const lostAsserts = count(removed, ASSERT_RE) - count(added, ASSERT_RE);
|
||||||
|
if (lostAsserts > 0) {
|
||||||
|
findings.push(finding('removed-assert', f.path, removed.find((l) => new RegExp(ASSERT_RE.source).test(l.text)), `${lostAsserts} fewer`));
|
||||||
|
}
|
||||||
|
const lostTests = removed.filter((l) => TEST_DECL_RE.test(l.text)).length - added.filter((l) => TEST_DECL_RE.test(l.text)).length;
|
||||||
|
if (lostTests > 0 || (f.deleted && removed.some((l) => TEST_DECL_RE.test(l.text)))) {
|
||||||
|
findings.push(finding('deleted-test', f.path, removed.find((l) => TEST_DECL_RE.test(l.text)), f.deleted ? 'test file deleted' : `${lostTests} fewer`));
|
||||||
|
}
|
||||||
|
for (const l of addedNotRemoved(removed, added, SKIP_RE)) findings.push(finding('skipped-test', f.path, l));
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const l of addedNotRemoved(removed, added, SUPPRESS_RE)) findings.push(finding('muted-check', f.path, l));
|
||||||
|
if (isCheckPath(f.path)) {
|
||||||
|
for (const l of addedNotRemoved(removed, added, MUTE_RE)) findings.push(finding('muted-check', f.path, l));
|
||||||
|
for (const h of f.hunks) {
|
||||||
|
const minus = h.filter((l) => l.type === '-');
|
||||||
|
const plus = h.filter((l) => l.type === '+');
|
||||||
|
for (const m of minus) {
|
||||||
|
for (const p of plus) {
|
||||||
|
const d = loosened(m.text, p.text);
|
||||||
|
if (d) { findings.push(finding('loosened-threshold', f.path, p, d)); break; }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!test) for (const l of addedNotRemoved(removed, added, STUB_RE)) findings.push(finding('stub', f.path, l));
|
||||||
|
}
|
||||||
|
return { status: findings.length ? 'findings' : 'clean', findings };
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- CLI ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const USAGE = 'usage: test-integrity.mjs (--diff <file|-> | --base <ref> [--head <ref>]) [--cwd <dir>] [--json]';
|
||||||
|
|
||||||
|
function diffFromGit(cwd, base, head) {
|
||||||
|
const ok = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${base}^{commit}`], { cwd, encoding: 'utf8' });
|
||||||
|
if (ok.status !== 0) return { error: `cannot resolve --base ${base} in ${cwd}` };
|
||||||
|
if (head) {
|
||||||
|
const okHead = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${head}^{commit}`], { cwd, encoding: 'utf8' });
|
||||||
|
if (okHead.status !== 0) return { error: `cannot resolve --head ${head} in ${cwd}` };
|
||||||
|
}
|
||||||
|
const args = ['diff', '--no-color', '--no-ext-diff', '--find-renames', base, ...(head ? [head] : [])];
|
||||||
|
const r = spawnSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
|
||||||
|
if (r.status !== 0) return { error: `git diff failed: ${String(r.stderr).trim()}` };
|
||||||
|
return { text: r.stdout };
|
||||||
|
}
|
||||||
|
|
||||||
|
export function main(argv) {
|
||||||
|
let diffPath = null;
|
||||||
|
let base = null;
|
||||||
|
let head = null;
|
||||||
|
let cwd = process.cwd();
|
||||||
|
let json = false;
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
if (a === '--diff' && argv[i + 1] !== undefined) diffPath = argv[++i];
|
||||||
|
else if (a === '--base' && argv[i + 1]) base = argv[++i];
|
||||||
|
else if (a === '--head' && argv[i + 1]) head = argv[++i];
|
||||||
|
else if (a === '--cwd' && argv[i + 1]) cwd = resolve(argv[++i]);
|
||||||
|
else if (a === '--json') json = true;
|
||||||
|
else { process.stderr.write(`test-integrity: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||||
|
}
|
||||||
|
if ((diffPath === null) === (base === null)) {
|
||||||
|
process.stderr.write(`test-integrity: give exactly one of --diff or --base\n${USAGE}\n`);
|
||||||
|
return 2;
|
||||||
|
}
|
||||||
|
let text;
|
||||||
|
if (diffPath !== null) {
|
||||||
|
try { text = readFileSync(diffPath === '-' ? 0 : diffPath, 'utf8'); }
|
||||||
|
catch (e) { return out({ status: 'unreadable', findings: [], reason: `cannot read ${diffPath}: ${e.message}` }, json); }
|
||||||
|
} else {
|
||||||
|
const g = diffFromGit(cwd, base, head);
|
||||||
|
if (g.error) return out({ status: 'unreadable', findings: [], reason: g.error }, json);
|
||||||
|
text = g.text;
|
||||||
|
}
|
||||||
|
return out(analyzeDiff(text), json);
|
||||||
|
}
|
||||||
|
|
||||||
|
function out(r, json) {
|
||||||
|
if (json) process.stdout.write(JSON.stringify(r, null, 2) + '\n');
|
||||||
|
else if (r.status === 'unreadable') process.stdout.write(`floor guard: NOT RUN - ${r.reason} (exit 2, never green)\n`);
|
||||||
|
else if (r.status === 'clean') process.stdout.write('floor guard: clean\n');
|
||||||
|
else {
|
||||||
|
process.stdout.write(`floor guard: ${r.findings.length} finding(s)\n`);
|
||||||
|
for (const f of r.findings) process.stdout.write(` ${f.kind} ${f.file}:${f.line ?? '-'} ${f.text}${f.detail ? ` (${f.detail})` : ''}\n`);
|
||||||
|
}
|
||||||
|
return r.status === 'clean' ? 0 : r.status === 'findings' ? 1 : 2;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isMain() {
|
||||||
|
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||||
|
catch { return false; }
|
||||||
|
}
|
||||||
|
|
||||||
|
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||||
|
|
@ -168,3 +168,61 @@ test('SC5 control: the extractors report the old vocabulary', () => {
|
||||||
const bad = offenders(writtenStatuses(old)).join('\n');
|
const bad = offenders(writtenStatuses(old)).join('\n');
|
||||||
for (const word of ['"in-progress"', '"passed"', '"running"']) assert.ok(bad.includes(word), `control must flag ${word}:\n${bad}`);
|
for (const word of ['"in-progress"', '"passed"', '"running"']) assert.ok(bad.includes(word), `control must flag ${word}:\n${bad}`);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// --- Handover 5, wave 1 (P8/P9): the per-step fields of the step loop ---------
|
||||||
|
// fix_rounds counts the rounds a step went through after its first control:
|
||||||
|
// 0 = passed first time, 1-3 = fix rounds, 4 = sent up to the plan, 5 = sent up
|
||||||
|
// to the operator. There is no sixth round, so 6 is invalid, not a warning.
|
||||||
|
// All new fields are optional on read: a progress.json written before them
|
||||||
|
// still resumes (SC4 above validates the old shape).
|
||||||
|
|
||||||
|
const { validateProgressObject } = progressValidator;
|
||||||
|
|
||||||
|
function withStep(extra) {
|
||||||
|
return {
|
||||||
|
schema_version: '1', plan: 'plan.md', plan_version: '1.8', mode: 'execute', status: 'in_progress',
|
||||||
|
total_steps: 1, current_step: 1,
|
||||||
|
steps: { 1: { status: 'in_progress', attempts: 1, ...extra } },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
const codes = (r) => r.errors.map((e) => e.code);
|
||||||
|
|
||||||
|
test('H5: fix_rounds 0-5 is accepted, 6 is rejected', () => {
|
||||||
|
for (const n of [0, 1, 2, 3, 4, 5]) {
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ fix_rounds: n }))), [], `fix_rounds ${n}`);
|
||||||
|
}
|
||||||
|
for (const bad of [6, -1, 2.5, '3']) {
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ fix_rounds: bad }))), ['PROGRESS_FIX_ROUNDS_INVALID'], `fix_rounds ${JSON.stringify(bad)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('H5: start_sha, report_path, red_proof and test_lock are shape-checked when present', () => {
|
||||||
|
const ok = { start_sha: '2c89a6b580f689070638f3ff83aea82d355f312f', report_path: '.claude/projects/x/steps/step-1.report.md', red_proof: 'proven', test_lock: 'not-locked' };
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep(ok))), []);
|
||||||
|
for (const v of ['proven', 'rejected', 'not-run']) assert.deepEqual(codes(validateProgressObject(withStep({ red_proof: v }))), [], v);
|
||||||
|
for (const v of ['locked', 'released', 'not-locked']) assert.deepEqual(codes(validateProgressObject(withStep({ test_lock: v }))), [], v);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ start_sha: 'HEAD' }))), ['PROGRESS_START_SHA_INVALID']);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ report_path: '' }))), ['PROGRESS_REPORT_PATH_INVALID']);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ red_proof: 'green' }))), ['PROGRESS_RED_PROOF_INVALID']);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ test_lock: 'yes' }))), ['PROGRESS_TEST_LOCK_INVALID']);
|
||||||
|
// null = not set yet (the schema fence's fresh step).
|
||||||
|
assert.deepEqual(codes(validateProgressObject(withStep({ start_sha: null, report_path: null, red_proof: null, test_lock: null }))), []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('H5: the run-level decision log is an array of {decision, cost_if_wrong}', () => {
|
||||||
|
const run = (decisions) => ({ ...withStep({}), decisions });
|
||||||
|
assert.deepEqual(codes(validateProgressObject(run([{ step: 2, decision: 'kept the old name', cost_if_wrong: 'one rename commit' }]))), []);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(run([{ step: 2, decision: 'kept the old name' }]))), ['PROGRESS_DECISION_INVALID']);
|
||||||
|
assert.deepEqual(codes(validateProgressObject(run('none'))), ['PROGRESS_DECISION_INVALID']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('H5: the progress schema fence carries the new per-step fields and the decision log', () => {
|
||||||
|
const schema = schemaObject(read());
|
||||||
|
const step = schema.steps['1'];
|
||||||
|
for (const k of ['start_sha', 'report_path', 'red_proof', 'test_lock', 'fix_rounds']) {
|
||||||
|
assert.ok(k in step, `the schema's step record must carry ${k}`);
|
||||||
|
}
|
||||||
|
assert.equal(step.fix_rounds, 0);
|
||||||
|
assert.ok(Array.isArray(schema.decisions), 'the schema must carry the run-level decisions log');
|
||||||
|
});
|
||||||
|
|
|
||||||
|
|
@ -109,11 +109,16 @@ test('trekexecute — global recovery/retry budget TREKEXECUTE_MAX_RECOVERY_ITER
|
||||||
|
|
||||||
test('trekexecute — every recovery/retry loop bounded + 3-axis cap hierarchy (SC b)', () => {
|
test('trekexecute — every recovery/retry loop bounded + 3-axis cap hierarchy (SC b)', () => {
|
||||||
const text = read();
|
const text = read();
|
||||||
assert.match(text, /maximum 2 retries|Retry cap = 3 attempts/, 'per-step retry cap must be explicit');
|
// Wave 1 (P8): the per-step cap is at most 5 fix rounds, and they go UP (to the
|
||||||
|
// plan, then to the operator) - never sideways into a fourth guess, never down
|
||||||
|
// a ladder of cheaper models. The old "3 attempts" cap is gone.
|
||||||
|
assert.match(text, /maximum 5 fix rounds[^\n]*upward/i, 'per-step cap must be "maximum 5 fix rounds", escalating upward');
|
||||||
|
assert.doesNotMatch(text, /maximum 2 retries|Retry cap = 3 attempts/, 'the old 3-attempt retry cap must be gone');
|
||||||
assert.match(text, /recovery_depth < 2/, 'recovery-dispatch cap must be explicit');
|
assert.match(text, /recovery_depth < 2/, 'recovery-dispatch cap must be explicit');
|
||||||
const capIdx = text.indexOf('Iteration caps');
|
const capIdx = text.indexOf('Iteration caps');
|
||||||
assert.ok(capIdx >= 0, 'cap-hierarchy section ("Iteration caps") must exist');
|
assert.ok(capIdx >= 0, 'cap-hierarchy section ("Iteration caps") must exist');
|
||||||
const section = text.slice(capIdx, capIdx + 1500);
|
const section = text.slice(capIdx, capIdx + 1500);
|
||||||
|
assert.match(section, /fix_rounds/, 'hierarchy must name the per-step fix_rounds axis');
|
||||||
assert.match(section, /attempts/, 'hierarchy must name per-step attempts axis');
|
assert.match(section, /attempts/, 'hierarchy must name per-step attempts axis');
|
||||||
assert.match(section, /recovery_depth/, 'hierarchy must name per-session recovery_depth axis');
|
assert.match(section, /recovery_depth/, 'hierarchy must name per-session recovery_depth axis');
|
||||||
assert.match(section, /TREKEXECUTE_MAX_RECOVERY_ITERATIONS/, 'hierarchy must name the global budget axis');
|
assert.match(section, /TREKEXECUTE_MAX_RECOVERY_ITERATIONS/, 'hierarchy must name the global budget axis');
|
||||||
|
|
|
||||||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -22,7 +22,7 @@
|
||||||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:961-976 | profile, phase_models, profile_source | none |
|
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:961-976 | profile, phase_models, profile_source | none |
|
||||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
|
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
|
||||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:1890-1912 | profile, phase_models, profile_source | none |
|
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2263-2285 | profile, phase_models, profile_source | none |
|
||||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
||||||
|
|
|
||||||
22
tests/fixtures/test-integrity/clean.diff
vendored
Normal file
22
tests/fixtures/test-integrity/clean.diff
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
||||||
|
diff --git a/lib/sum.mjs b/lib/sum.mjs
|
||||||
|
index 1111111..2222222 100644
|
||||||
|
--- a/lib/sum.mjs
|
||||||
|
+++ b/lib/sum.mjs
|
||||||
|
@@ -1,3 +1,4 @@
|
||||||
|
export function sum(a, b) {
|
||||||
|
- return a - b;
|
||||||
|
+ // fixed: addition, not subtraction
|
||||||
|
+ return a + b;
|
||||||
|
}
|
||||||
|
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||||
|
index 3333333..4444444 100644
|
||||||
|
--- a/tests/sum.test.mjs
|
||||||
|
+++ b/tests/sum.test.mjs
|
||||||
|
@@ -3,3 +3,7 @@ import { sum } from '../lib/sum.mjs';
|
||||||
|
test('sum adds', () => {
|
||||||
|
assert.equal(sum(1, 2), 3);
|
||||||
|
});
|
||||||
|
+
|
||||||
|
+test('sum adds negatives', () => {
|
||||||
|
+ assert.equal(sum(-1, -2), -3);
|
||||||
|
+});
|
||||||
11
tests/fixtures/test-integrity/deleted-test.diff
vendored
Normal file
11
tests/fixtures/test-integrity/deleted-test.diff
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||||
|
index 3333333..4444444 100644
|
||||||
|
--- a/tests/sum.test.mjs
|
||||||
|
+++ b/tests/sum.test.mjs
|
||||||
|
@@ -3,6 +3,3 @@ import { sum } from '../lib/sum.mjs';
|
||||||
|
test('sum adds', () => {
|
||||||
|
assert.equal(sum(1, 2), 3);
|
||||||
|
});
|
||||||
|
-test('sum adds negatives', () => {
|
||||||
|
- sum(-1, -2);
|
||||||
|
-});
|
||||||
9
tests/fixtures/test-integrity/loosened-threshold.diff
vendored
Normal file
9
tests/fixtures/test-integrity/loosened-threshold.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
diff --git a/tests/latency.test.mjs b/tests/latency.test.mjs
|
||||||
|
index 7777777..8888888 100644
|
||||||
|
--- a/tests/latency.test.mjs
|
||||||
|
+++ b/tests/latency.test.mjs
|
||||||
|
@@ -4,3 +4,3 @@ test('tail latency', () => {
|
||||||
|
const ms = measure();
|
||||||
|
- assert.ok(ms < 200, `tail latency ${ms}ms`);
|
||||||
|
+ assert.ok(ms < 2000, `tail latency ${ms}ms`);
|
||||||
|
});
|
||||||
8
tests/fixtures/test-integrity/muted-check.diff
vendored
Normal file
8
tests/fixtures/test-integrity/muted-check.diff
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
diff --git a/tests/run.sh b/tests/run.sh
|
||||||
|
index 5555555..6666666 100644
|
||||||
|
--- a/tests/run.sh
|
||||||
|
+++ b/tests/run.sh
|
||||||
|
@@ -1,2 +1,2 @@
|
||||||
|
#!/bin/sh
|
||||||
|
-node --test tests/
|
||||||
|
+node --test tests/ || true
|
||||||
9
tests/fixtures/test-integrity/removed-assert.diff
vendored
Normal file
9
tests/fixtures/test-integrity/removed-assert.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||||
|
index 3333333..4444444 100644
|
||||||
|
--- a/tests/sum.test.mjs
|
||||||
|
+++ b/tests/sum.test.mjs
|
||||||
|
@@ -3,5 +3,4 @@ import { sum } from '../lib/sum.mjs';
|
||||||
|
test('sum adds', () => {
|
||||||
|
assert.equal(sum(1, 2), 3);
|
||||||
|
- assert.equal(sum(2, 2), 4);
|
||||||
|
});
|
||||||
9
tests/fixtures/test-integrity/skipped-test.diff
vendored
Normal file
9
tests/fixtures/test-integrity/skipped-test.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||||
|
index 3333333..4444444 100644
|
||||||
|
--- a/tests/sum.test.mjs
|
||||||
|
+++ b/tests/sum.test.mjs
|
||||||
|
@@ -3,3 +3,3 @@ import { sum } from '../lib/sum.mjs';
|
||||||
|
-test('sum adds', () => {
|
||||||
|
+test.skip('sum adds', () => {
|
||||||
|
assert.equal(sum(1, 2), 3);
|
||||||
|
});
|
||||||
9
tests/fixtures/test-integrity/stub.diff
vendored
Normal file
9
tests/fixtures/test-integrity/stub.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
diff --git a/lib/sum.mjs b/lib/sum.mjs
|
||||||
|
index 1111111..2222222 100644
|
||||||
|
--- a/lib/sum.mjs
|
||||||
|
+++ b/lib/sum.mjs
|
||||||
|
@@ -1,3 +1,3 @@
|
||||||
|
export function sum(a, b) {
|
||||||
|
- return a - b;
|
||||||
|
+ throw new Error('not implemented');
|
||||||
|
}
|
||||||
9
tests/fixtures/test-integrity/tightened-threshold.diff
vendored
Normal file
9
tests/fixtures/test-integrity/tightened-threshold.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
diff --git a/tests/latency.test.mjs b/tests/latency.test.mjs
|
||||||
|
index 7777777..8888888 100644
|
||||||
|
--- a/tests/latency.test.mjs
|
||||||
|
+++ b/tests/latency.test.mjs
|
||||||
|
@@ -4,3 +4,3 @@ test('tail latency', () => {
|
||||||
|
const ms = measure();
|
||||||
|
- assert.ok(ms < 2000, `tail latency ${ms}ms`);
|
||||||
|
+ assert.ok(ms < 200, `tail latency ${ms}ms`);
|
||||||
|
});
|
||||||
2
tests/fixtures/test-integrity/unreadable.diff
vendored
Normal file
2
tests/fixtures/test-integrity/unreadable.diff
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
||||||
|
this is not a diff
|
||||||
|
just prose that a broken pipe left behind
|
||||||
|
|
@ -1667,9 +1667,11 @@ for (const cmd of ['trekresearch', 'trekplan', 'trekreview', 'trekexecute']) {
|
||||||
// The derived set is asserted exactly, so the pin cannot go vacuous if a
|
// The derived set is asserted exactly, so the pin cannot go vacuous if a
|
||||||
// command stops spawning or a new spawning command is added.
|
// command stops spawning or a new spawning command is added.
|
||||||
|
|
||||||
const SPAWNING_COMMANDS = ['trekbrief', 'trekplan', 'trekresearch', 'trekreview'];
|
// Wave 1 (P8): /trekexecute joined the set - one fresh step helper per plan step
|
||||||
|
// plus one controller - so the no-name rule now covers five commands.
|
||||||
|
const SPAWNING_COMMANDS = ['trekbrief', 'trekexecute', 'trekplan', 'trekresearch', 'trekreview'];
|
||||||
|
|
||||||
test('S85: the set of agent-spawning commands is exactly the four that carry the no-name rule', () => {
|
test('S85: the set of agent-spawning commands is exactly the five that carry the no-name rule', () => {
|
||||||
const spawns = listMd('commands')
|
const spawns = listMd('commands')
|
||||||
.filter((f) => /^Launch\b/m.test(read(`commands/${f}`)))
|
.filter((f) => /^Launch\b/m.test(read(`commands/${f}`)))
|
||||||
.map((f) => f.replace(/\.md$/, ''))
|
.map((f) => f.replace(/\.md$/, ''))
|
||||||
|
|
|
||||||
88
tests/lib/done-line.test.mjs
Normal file
88
tests/lib/done-line.test.mjs
Normal file
|
|
@ -0,0 +1,88 @@
|
||||||
|
// tests/lib/done-line.test.mjs
|
||||||
|
// The done line (P9 (4)): a step's "done" is written by a script, and only when
|
||||||
|
// the test command exits 0. Exit 0 = written, 1 = the command failed (nothing
|
||||||
|
// written), 2 = could not run (nothing written, never green).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const CLI = join(ROOT, 'lib', 'verification', 'done-line.mjs');
|
||||||
|
|
||||||
|
function sh(cwd, cmd, args) {
|
||||||
|
const r = spawnSync(cmd, args, { cwd, encoding: 'utf8' });
|
||||||
|
assert.equal(r.status, 0, `${cmd} ${args.join(' ')}: ${r.stderr}`);
|
||||||
|
return r.stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function repo() {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'done-line-'));
|
||||||
|
sh(dir, 'git', ['init', '-q']);
|
||||||
|
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||||
|
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||||
|
mkdirSync(join(dir, 'tests'));
|
||||||
|
writeFileSync(join(dir, 'tests', 'ok.test.mjs'), "import { test } from 'node:test';\ntest('ok', () => {});\n");
|
||||||
|
writeFileSync(join(dir, 'tests', 'bad.test.mjs'), "import { test } from 'node:test';\ntest('bad', () => { throw new Error('red'); });\n");
|
||||||
|
sh(dir, 'git', ['add', 'tests']);
|
||||||
|
sh(dir, 'git', ['commit', '-q', '-m', 'tests']);
|
||||||
|
return dir;
|
||||||
|
}
|
||||||
|
|
||||||
|
const run = (args) => spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8' });
|
||||||
|
|
||||||
|
test('done line: the test command exits 0 -> one DONE line appended, exit 0', () => {
|
||||||
|
const dir = repo();
|
||||||
|
const report = join(dir, 'step-3.report.md');
|
||||||
|
try {
|
||||||
|
writeFileSync(report, '# Step 3 report\n');
|
||||||
|
const r = run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/ok.test.mjs', '--report', report]);
|
||||||
|
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||||
|
const head = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||||
|
const lines = readFileSync(report, 'utf8').split('\n').filter((l) => l.startsWith('DONE '));
|
||||||
|
assert.equal(lines.length, 1);
|
||||||
|
assert.match(lines[0], new RegExp(`^DONE step 3: \`node --test tests/ok\\.test\\.mjs\` exit 0 at ${head} \\d{4}-\\d{2}-\\d{2}T`));
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('done line: the test command fails -> nothing written, exit 1', () => {
|
||||||
|
const dir = repo();
|
||||||
|
const report = join(dir, 'step-3.report.md');
|
||||||
|
try {
|
||||||
|
writeFileSync(report, '# Step 3 report\n');
|
||||||
|
const r = run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/bad.test.mjs', '--report', report]);
|
||||||
|
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||||
|
assert.equal(readFileSync(report, 'utf8'), '# Step 3 report\n');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('done line: a command outside the test-runner allowlist is not run -> exit 2, nothing written', () => {
|
||||||
|
const dir = repo();
|
||||||
|
const report = join(dir, 'step-3.report.md');
|
||||||
|
try {
|
||||||
|
for (const cmd of ['echo done', 'node --test tests/bad.test.mjs || true', 'true']) {
|
||||||
|
const r = run(['--cwd', dir, '--step', '3', '--cmd', cmd, '--report', report]);
|
||||||
|
assert.equal(r.status, 2, `${cmd}: ${r.stdout}${r.stderr}`);
|
||||||
|
}
|
||||||
|
assert.equal(existsSync(report), false);
|
||||||
|
assert.equal(run(['--cwd', dir, '--cmd', 'node --test tests/ok.test.mjs', '--report', report]).status, 2, 'no --step');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('done line: --check finds the line for that step only', () => {
|
||||||
|
const dir = repo();
|
||||||
|
const report = join(dir, 'step-3.report.md');
|
||||||
|
try {
|
||||||
|
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'no report file = not done');
|
||||||
|
writeFileSync(report, '# Step 3 report\nDONE step 3 (typed by hand)\n');
|
||||||
|
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'a hand-typed line is not the script line');
|
||||||
|
run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/ok.test.mjs', '--report', report]);
|
||||||
|
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 0);
|
||||||
|
assert.equal(run(['--check', '--step', '4', '--report', report]).status, 1);
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
199
tests/lib/executor-prose-pins.test.mjs
Normal file
199
tests/lib/executor-prose-pins.test.mjs
Normal file
|
|
@ -0,0 +1,199 @@
|
||||||
|
// tests/lib/executor-prose-pins.test.mjs
|
||||||
|
// Wave 1, process 1 (the executor): prose pins for P8 (fresh helper per step,
|
||||||
|
// one controller with two verdicts, fix rounds that go up), P9 on the executor
|
||||||
|
// side (proof of red, the floor guard, the done line, end to end, back to the
|
||||||
|
// architecture after three failed fixes) and P4 (the closed stop list and the
|
||||||
|
// decision log). The mechanisms themselves are tested as code in
|
||||||
|
// tests/lib/{red-proof,test-integrity,done-line}.test.mjs; these pins hold the
|
||||||
|
// command prose to them. Fix the SOURCE a pin guards, not the pin.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { existsSync, readFileSync } from 'node:fs';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const read = (rel) => readFileSync(join(ROOT, rel), 'utf8');
|
||||||
|
const CMD = 'commands/trekexecute.md';
|
||||||
|
const AGENT = 'agents/step-implementer.md';
|
||||||
|
|
||||||
|
function frontmatter(text) {
|
||||||
|
const m = text.match(/^---\n([\s\S]*?)\n---/);
|
||||||
|
assert.ok(m, 'frontmatter missing');
|
||||||
|
return m[1];
|
||||||
|
}
|
||||||
|
const fmValue = (fm, key) => (fm.match(new RegExp(`^${key}:\\s*(.+)$`, 'm')) || [])[1];
|
||||||
|
|
||||||
|
/** The text between `## {start}` (or `#### {start}`) and the next heading of the same or higher level. */
|
||||||
|
function section(text, heading) {
|
||||||
|
const at = text.indexOf(heading);
|
||||||
|
assert.ok(at >= 0, `${CMD} must have "${heading}"`);
|
||||||
|
const level = heading.match(/^#+/)[0].length;
|
||||||
|
const rest = text.slice(at + heading.length);
|
||||||
|
const next = rest.search(new RegExp(`\\n#{1,${level}} `));
|
||||||
|
return heading + (next >= 0 ? rest.slice(0, next) : rest);
|
||||||
|
}
|
||||||
|
|
||||||
|
// --- P8: the Agent tool is back, for one purpose -------------------------------
|
||||||
|
|
||||||
|
test('P8: /trekexecute may spawn - Agent is allowed and no longer disallowed', () => {
|
||||||
|
const fm = frontmatter(read(CMD));
|
||||||
|
assert.match(fmValue(fm, 'allowed-tools'), /\bAgent\b/, 'allowed-tools must list Agent');
|
||||||
|
const disallowed = fmValue(fm, 'disallowed-tools') || '';
|
||||||
|
assert.doesNotMatch(disallowed, /\bAgent\b/, 'disallowed-tools must no longer remove Agent');
|
||||||
|
assert.match(disallowed, /\bTeamCreate\b/, 'TeamCreate stays removed - helpers, never a team');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: Hard rule 10 is rewritten - one fresh helper per plan step, and helpers never start helpers', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.doesNotMatch(text, /\*\*No sub-agents\.\*\*/, 'the old "No sub-agents" rule must be gone');
|
||||||
|
const rule = text.slice(text.indexOf('10. **'), text.indexOf('11. **'));
|
||||||
|
assert.match(rule, /fresh helper per plan step/i);
|
||||||
|
assert.match(rule, /voyage:step-implementer/);
|
||||||
|
assert.match(rule, /helpers never start helpers/i);
|
||||||
|
assert.match(rule, /controller/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: the step agent exists, is opus at effort >= high, and cannot spawn', () => {
|
||||||
|
assert.ok(existsSync(join(ROOT, AGENT)), `${AGENT} must exist`);
|
||||||
|
const fm = frontmatter(read(AGENT));
|
||||||
|
assert.equal(fmValue(fm, 'name'), 'step-implementer');
|
||||||
|
assert.equal(fmValue(fm, 'model'), 'opus');
|
||||||
|
assert.ok(['high', 'xhigh', 'max'].includes(fmValue(fm, 'effort')), 'effort must be >= high');
|
||||||
|
assert.doesNotMatch(fmValue(fm, 'tools') || '', /\bAgent\b/, 'a helper must not hold the Agent tool');
|
||||||
|
// Measured 2026-10-06 (CC 2.1.291): a worktree-isolated helper branched from
|
||||||
|
// the default branch (02f320f), not from the executor's HEAD (2c89a6b), and so
|
||||||
|
// did not see the earlier steps. A plugin cannot set worktree.baseRef. So no
|
||||||
|
// isolation: the helper works in the executor's tree, from a recorded start_sha.
|
||||||
|
assert.doesNotMatch(fm, /^isolation:/m, 'no isolation: worktree on the step agent (measured: wrong base)');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: the helper report contract is ran / saw / did NOT check', () => {
|
||||||
|
const body = read(AGENT);
|
||||||
|
for (const h of ['### Ran', '### Saw', '### Did NOT check']) assert.ok(body.includes(h), `${AGENT} report must have "${h}"`);
|
||||||
|
assert.ok(read(CMD).includes('Did NOT check'), 'the command must name the report contract too');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: hand-over is in files, from a recorded start point', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /start_sha/);
|
||||||
|
assert.match(text, /report_path/);
|
||||||
|
assert.match(text, /git diff \{start_sha\}/, 'the review package is the diff from the recorded start point');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: one controller, two verdicts, reads the diff not the summary, re-checks only what was fixed', () => {
|
||||||
|
const ctl = section(read(CMD), '#### Sub-step E — Control');
|
||||||
|
assert.match(ctl, /one controller/i);
|
||||||
|
assert.match(ctl, /two verdicts/i);
|
||||||
|
assert.match(ctl, /"spec"/);
|
||||||
|
assert.match(ctl, /"quality"/);
|
||||||
|
assert.match(ctl, /reads the diff, not the helper's (summary|report)/i);
|
||||||
|
assert.match(ctl, /only what (was|the round) fixed/i);
|
||||||
|
assert.match(ctl, /general-purpose/, 'the controller is a general helper with a prompt from the command, not agent no. 2');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: fix rounds go up - maximum 5, plan then operator, never a cheaper-model ladder', () => {
|
||||||
|
const fix = section(read(CMD), '#### Sub-step F — Fix rounds');
|
||||||
|
assert.match(fix, /maximum 5 fix rounds[^\n]*upward/i);
|
||||||
|
assert.match(fix, /back to (the )?planning/i);
|
||||||
|
assert.match(fix, /operator/i);
|
||||||
|
assert.match(fix, /does not improvise/i);
|
||||||
|
assert.match(fix, /no ladder of cheaper models/i);
|
||||||
|
// P9 (7): after three failed fixes the question is the architecture, not a fourth fix.
|
||||||
|
assert.match(fix, /three failed fix(es| rounds)[^\n]*architecture/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P8: the no-name rule covers the step helper and the controller', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.ok(text.includes('in_process_teammate'));
|
||||||
|
assert.match(text, /never pass .*`name`/i);
|
||||||
|
assert.match(text, /^Launch\b/m, 'the spawn site starts with "Launch" (the S85 detector reads it)');
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- P9 (executor side) ---------------------------------------------------------
|
||||||
|
|
||||||
|
test('P9 (1): proof of red is a command and it blocks - the old "do not block" sentence is gone', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.doesNotMatch(text, /Do not block on test-first failures/);
|
||||||
|
assert.match(text, /lib\/verification\/red-proof\.mjs/);
|
||||||
|
const red = section(text, '#### Sub-step B — Test first and proof of red');
|
||||||
|
assert.match(red, /must FAIL/);
|
||||||
|
assert.match(red, /must PASS/);
|
||||||
|
assert.match(red, /not-run/);
|
||||||
|
assert.match(red, /never (a pass|green|proven)/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('K4: Test first is required from plan_version 1.8; Interface in/out and Review focus are read, not required', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /plan_version ≥ 1\.8[^\n]*`Test first:`|`Test first:`[^\n]*plan_version ≥ 1\.8/);
|
||||||
|
for (const f of ['Interface in:', 'Interface out:', 'Review focus:']) assert.ok(text.includes(f), `${CMD} must read ${f}`);
|
||||||
|
assert.match(text, /tolerat(e|es) (their|its) absence|never required/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('K8: the test lock is used through its CLI, and a missing CLI is said out loud', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /lib\/util\/test-lock\.mjs lock --files/);
|
||||||
|
assert.match(text, /test-lock\.mjs unlock/);
|
||||||
|
assert.match(text, /"not-locked"/);
|
||||||
|
assert.match(text, /never silent/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P9 (2): the floor guard runs on the step diff and exit 2 is never green', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /lib\/verification\/test-integrity\.mjs --base \{start_sha\}/);
|
||||||
|
assert.match(text, /exit 2[^\n]*never (green|a pass)/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P9 (4): the done line is written by a script, only when the test command exits 0', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /lib\/verification\/done-line\.mjs/);
|
||||||
|
assert.match(text, /done-line\.mjs --check/);
|
||||||
|
assert.match(text, /only (when|if) the test command exits 0/i);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P9 (6): without a run-the-thing check the result is "not verified end to end", never green', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /not verified end to end/);
|
||||||
|
assert.match(text, /end_to_end/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P9: the three new CLIs the prose names exist', () => {
|
||||||
|
for (const f of ['red-proof.mjs', 'test-integrity.mjs', 'done-line.mjs']) {
|
||||||
|
assert.ok(existsSync(join(ROOT, 'lib', 'verification', f)), `lib/verification/${f}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- P4: the closed stop list and the decision log ------------------------------
|
||||||
|
|
||||||
|
test('P4: the run stops only on the closed list of four, and the final report shows it', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
const list = section(text, '### Closed stop list');
|
||||||
|
for (const cls of [/cannot be undone/i, /security-sensitive/i, /outside the working copy/i, /so broken that (every|any)thing (after it )?is guesswork/i]) {
|
||||||
|
assert.match(list, cls);
|
||||||
|
}
|
||||||
|
assert.match(list, /closed/i);
|
||||||
|
// Phase 8's report template is a fence that holds its own "## " heading, so
|
||||||
|
// the phase is read up to Phase 9, not up to the next heading.
|
||||||
|
const report = text.slice(text.indexOf('## Phase 8 — Final report'), text.indexOf('## Phase 9'));
|
||||||
|
assert.match(report, /### Stops/);
|
||||||
|
assert.match(report, /### Decisions taken/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('P4: every decision the executor took is logged with what it costs if it is wrong', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
assert.match(text, /cost_if_wrong/);
|
||||||
|
assert.match(text, /"decisions"/);
|
||||||
|
});
|
||||||
|
|
||||||
|
// --- Credits: every borrowing is named, never silent ----------------------------
|
||||||
|
|
||||||
|
test('every borrowing in the executor is credited', () => {
|
||||||
|
const text = read(CMD);
|
||||||
|
const credits = section(text, '## Credits');
|
||||||
|
for (const src of [/Superpowers/, /subagent-driven-development/, /ai-native-sdlc/, /Kiro/, /Spec Kit/, /agent-skills/, /Cherny/]) {
|
||||||
|
assert.match(credits, src);
|
||||||
|
}
|
||||||
|
assert.match(credits, /MIT/);
|
||||||
|
});
|
||||||
200
tests/lib/red-proof.test.mjs
Normal file
200
tests/lib/red-proof.test.mjs
Normal file
|
|
@ -0,0 +1,200 @@
|
||||||
|
// tests/lib/red-proof.test.mjs
|
||||||
|
// Proof of red (P9 (1)): the step's test is run by a command against the start
|
||||||
|
// point, where it must FAIL, and against the result, where it must PASS. The
|
||||||
|
// model's word that "the test was red first" is never the proof.
|
||||||
|
// Exit 0 = proven, 1 = rejected (green at the start point, or red at the result),
|
||||||
|
// 2 = could not run (never green).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { proveRed } from '../../lib/verification/red-proof.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const CLI = join(ROOT, 'lib', 'verification', 'red-proof.mjs');
|
||||||
|
const TEST_CMD = 'node --test tests/sum.test.mjs';
|
||||||
|
|
||||||
|
const SUM_TEST = [
|
||||||
|
"import { test } from 'node:test';",
|
||||||
|
"import { strict as assert } from 'node:assert';",
|
||||||
|
"import { sum } from '../lib/sum.mjs';",
|
||||||
|
"test('sum adds', () => { assert.equal(sum(1, 2), 3); });",
|
||||||
|
'',
|
||||||
|
].join('\n');
|
||||||
|
|
||||||
|
function sh(cwd, cmd, args) {
|
||||||
|
const r = spawnSync(cmd, args, { cwd, encoding: 'utf8' });
|
||||||
|
assert.equal(r.status, 0, `${cmd} ${args.join(' ')}: ${r.stderr}`);
|
||||||
|
return r.stdout.trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A repo whose base commit has `body` as sum(); returns {dir, base}. */
|
||||||
|
function repo(body) {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'red-proof-'));
|
||||||
|
sh(dir, 'git', ['init', '-q']);
|
||||||
|
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||||
|
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||||
|
mkdirSync(join(dir, 'lib'));
|
||||||
|
writeFileSync(join(dir, 'lib', 'sum.mjs'), `export function sum(a, b) { return ${body}; }\n`);
|
||||||
|
sh(dir, 'git', ['add', 'lib/sum.mjs']);
|
||||||
|
sh(dir, 'git', ['commit', '-q', '-m', 'base']);
|
||||||
|
return { dir, base: sh(dir, 'git', ['rev-parse', 'HEAD']) };
|
||||||
|
}
|
||||||
|
|
||||||
|
function addTest(dir) {
|
||||||
|
mkdirSync(join(dir, 'tests'), { recursive: true });
|
||||||
|
writeFileSync(join(dir, 'tests', 'sum.test.mjs'), SUM_TEST);
|
||||||
|
}
|
||||||
|
|
||||||
|
function cli(args) {
|
||||||
|
const r = spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8' });
|
||||||
|
return { status: r.status, stdout: r.stdout, stderr: r.stderr };
|
||||||
|
}
|
||||||
|
|
||||||
|
test('red proof: red at the start point, green at the result -> proven, exit 0', () => {
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||||
|
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||||
|
assert.equal(r.status, 'proven', JSON.stringify(r));
|
||||||
|
assert.notEqual(r.base.exit, 0);
|
||||||
|
assert.equal(r.result.exit, 0);
|
||||||
|
const c = cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs', '--json']);
|
||||||
|
assert.equal(c.status, 0, c.stdout + c.stderr);
|
||||||
|
assert.equal(JSON.parse(c.stdout).status, 'proven');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof: a test that already PASSES at the start point is rejected, exit 1', () => {
|
||||||
|
const { dir, base } = repo('a + b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||||
|
assert.equal(r.status, 'rejected');
|
||||||
|
assert.equal(r.reason, 'not-red-at-base');
|
||||||
|
const c = cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']);
|
||||||
|
assert.equal(c.status, 1, c.stdout + c.stderr);
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof: a test that still FAILS at the result is rejected, exit 1', () => {
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||||
|
assert.equal(r.status, 'rejected');
|
||||||
|
assert.equal(r.reason, 'not-green-at-result');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof --red-only: the gate before building - red at the start point only, the result is not run', () => {
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir); // test written, nothing built yet: the working tree is still red
|
||||||
|
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'], redOnly: true });
|
||||||
|
assert.equal(r.status, 'proven', JSON.stringify(r));
|
||||||
|
assert.equal(r.result, undefined, '--red-only never runs the result');
|
||||||
|
const c = cli(['--red-only', '--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']);
|
||||||
|
assert.equal(c.status, 0, c.stdout + c.stderr);
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
const green = repo('a + b');
|
||||||
|
try {
|
||||||
|
addTest(green.dir);
|
||||||
|
const r = proveRed({ cwd: green.dir, base: green.base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'], redOnly: true });
|
||||||
|
assert.equal(r.reason, 'not-red-at-base');
|
||||||
|
} finally { rmSync(green.dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof: what cannot be measured is "not-run", exit 2 - never proven', () => {
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||||
|
const args = (over) => ['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs', ...over];
|
||||||
|
// A command outside the test-runner allowlist never reaches a shell.
|
||||||
|
assert.equal(proveRed({ cwd: dir, base, cmd: 'node tests/sum.test.mjs', tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||||
|
assert.equal(proveRed({ cwd: dir, base, cmd: `${TEST_CMD} || true`, tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||||
|
// An unknown start point, a missing test file, no test file at all.
|
||||||
|
assert.equal(proveRed({ cwd: dir, base: 'no-such-ref', cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||||
|
assert.equal(proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/missing.test.mjs'] }).status, 'not-run');
|
||||||
|
assert.equal(proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: [] }).status, 'not-run');
|
||||||
|
assert.equal(cli(args([])).status, 0, 'control: the same arguments do prove');
|
||||||
|
assert.equal(cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD]).status, 2, 'no --tests');
|
||||||
|
assert.equal(cli(['--cwd', join(dir, 'nope'), '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']).status, 2);
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof: --cwd in a subdirectory still runs at the repo root (plan paths are root-relative)', () => {
|
||||||
|
// Measured 2026-10-06: `git archive` run from a subdirectory packs only that
|
||||||
|
// subdirectory, so a start point built from a subdir cwd would miss lib/.
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||||
|
mkdirSync(join(dir, 'docs'));
|
||||||
|
const r = proveRed({ cwd: join(dir, 'docs'), base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||||
|
assert.equal(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('red proof: the start-point run leaves the working tree untouched', () => {
|
||||||
|
const { dir, base } = repo('a - b');
|
||||||
|
try {
|
||||||
|
addTest(dir);
|
||||||
|
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||||
|
const before = sh(dir, 'git', ['status', '--porcelain']);
|
||||||
|
proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||||
|
assert.equal(sh(dir, 'git', ['status', '--porcelain']), before);
|
||||||
|
assert.equal(readFileSync(join(dir, 'lib', 'sum.mjs'), 'utf8'), 'export function sum(a, b) { return a + b; }\n');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
// Reuse of tests/fixtures/red-first/02243c6-always-allow-shim.mjs: the measured
|
||||||
|
// red-first claim of 02243c6 ("6 of 55"), re-run through red-proof instead of by
|
||||||
|
// hand. Needs this repo's git history; a `git archive` tree has none, and there
|
||||||
|
// the test says so and skips rather than pretending to pass.
|
||||||
|
const HAS_HISTORY = spawnSync('git', ['cat-file', '-e', '02243c6^{commit}'], { cwd: ROOT }).status === 0;
|
||||||
|
|
||||||
|
test('red proof: reproduces the 02243c6 red-first measurement (6 of 55) and the bare load error (1 of 1)', { skip: !HAS_HISTORY && 'no git history (git archive tree)' }, () => {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'red-proof-02243c6-'));
|
||||||
|
const RUNNER = 'lib/verification/criteria-runner.mjs';
|
||||||
|
const TEST_FILE = 'tests/lib/criteria-runner.test.mjs';
|
||||||
|
const tar = join(dir, '..', `${dir.split('/').pop()}.tar`);
|
||||||
|
try {
|
||||||
|
sh(ROOT, 'git', ['archive', '--format=tar', '-o', tar, '02243c6']);
|
||||||
|
sh(dir, 'tar', ['-xf', tar]);
|
||||||
|
const parent = spawnSync('git', ['show', `02243c6^:${RUNNER}`], { cwd: ROOT, encoding: 'utf8' }).stdout;
|
||||||
|
const real = readFileSync(join(dir, RUNNER), 'utf8');
|
||||||
|
const shim = readFileSync(join(ROOT, 'tests', 'fixtures', 'red-first', '02243c6-always-allow-shim.mjs'), 'utf8');
|
||||||
|
sh(dir, 'git', ['init', '-q']);
|
||||||
|
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||||
|
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||||
|
sh(dir, 'git', ['add', '.']);
|
||||||
|
// Start point 1: the parent's runner as it was - the test file cannot even load.
|
||||||
|
writeFileSync(join(dir, RUNNER), parent);
|
||||||
|
sh(dir, 'git', ['commit', '-q', '-am', 'parent runner']);
|
||||||
|
const bare = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||||
|
// Start point 2: the parent's runner plus the always-allow shim.
|
||||||
|
writeFileSync(join(dir, RUNNER), parent + shim);
|
||||||
|
sh(dir, 'git', ['commit', '-q', '-am', 'parent runner + shim']);
|
||||||
|
const shimmed = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||||
|
writeFileSync(join(dir, RUNNER), real);
|
||||||
|
const cmd = `node --test --test-reporter=tap ${TEST_FILE}`;
|
||||||
|
const withShim = proveRed({ cwd: dir, base: shimmed, cmd, tests: [TEST_FILE] });
|
||||||
|
assert.equal(withShim.status, 'proven', JSON.stringify(withShim));
|
||||||
|
assert.deepEqual(withShim.base.tap, { tests: 55, fail: 6 });
|
||||||
|
assert.deepEqual(withShim.result.tap, { tests: 55, fail: 0 });
|
||||||
|
const bareRun = proveRed({ cwd: dir, base: bare, cmd, tests: [TEST_FILE] });
|
||||||
|
assert.equal(bareRun.status, 'proven');
|
||||||
|
assert.deepEqual(bareRun.base.tap, { tests: 1, fail: 1 }, 'the parent tree fails as ONE load error, not as named tests');
|
||||||
|
} finally {
|
||||||
|
rmSync(dir, { recursive: true, force: true });
|
||||||
|
rmSync(tar, { force: true });
|
||||||
|
}
|
||||||
|
});
|
||||||
126
tests/lib/test-integrity.test.mjs
Normal file
126
tests/lib/test-integrity.test.mjs
Normal file
|
|
@ -0,0 +1,126 @@
|
||||||
|
// tests/lib/test-integrity.test.mjs
|
||||||
|
// The floor guard (P9 (2)): a diff that weakens the tests is a finding, never a
|
||||||
|
// pass. Exit 0 = clean, 1 = findings, 2 = could not run (never green).
|
||||||
|
// Borrowed: agent-skills "floor guard" (MIT) - see lib/verification/test-integrity.mjs.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import { spawnSync } from 'node:child_process';
|
||||||
|
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||||
|
import { tmpdir } from 'node:os';
|
||||||
|
import { dirname, join } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
import { analyzeDiff, isTestPath } from '../../lib/verification/test-integrity.mjs';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = join(HERE, '..', '..');
|
||||||
|
const CLI = join(ROOT, 'lib', 'verification', 'test-integrity.mjs');
|
||||||
|
const FIX = join(ROOT, 'tests', 'fixtures', 'test-integrity');
|
||||||
|
const fixture = (name) => readFileSync(join(FIX, name), 'utf8');
|
||||||
|
|
||||||
|
function cli(args, opts = {}) {
|
||||||
|
const r = spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8', ...opts });
|
||||||
|
return { status: r.status, stdout: r.stdout, stderr: r.stderr };
|
||||||
|
}
|
||||||
|
|
||||||
|
const kinds = (r) => r.findings.map((f) => f.kind);
|
||||||
|
|
||||||
|
test('floor guard: a clean diff (fix + a new test) is clean -> exit 0', () => {
|
||||||
|
const r = analyzeDiff(fixture('clean.diff'));
|
||||||
|
assert.equal(r.status, 'clean', JSON.stringify(r.findings));
|
||||||
|
assert.deepEqual(r.findings, []);
|
||||||
|
assert.equal(cli(['--diff', join(FIX, 'clean.diff')]).status, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: a diff that removes an assert -> finding removed-assert, exit 1', () => {
|
||||||
|
const r = analyzeDiff(fixture('removed-assert.diff'));
|
||||||
|
assert.equal(r.status, 'findings');
|
||||||
|
assert.deepEqual(kinds(r), ['removed-assert']);
|
||||||
|
assert.equal(r.findings[0].file, 'tests/sum.test.mjs');
|
||||||
|
assert.equal(cli(['--diff', join(FIX, 'removed-assert.diff')]).status, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: a diff that skips a test -> finding skipped-test, exit 1', () => {
|
||||||
|
const r = analyzeDiff(fixture('skipped-test.diff'));
|
||||||
|
assert.deepEqual(kinds(r), ['skipped-test']);
|
||||||
|
assert.equal(cli(['--diff', join(FIX, 'skipped-test.diff')]).status, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: a deleted test, a muted check, a stub and a loosened threshold are each found', () => {
|
||||||
|
assert.deepEqual(kinds(analyzeDiff(fixture('deleted-test.diff'))), ['deleted-test']);
|
||||||
|
assert.deepEqual(kinds(analyzeDiff(fixture('muted-check.diff'))), ['muted-check']);
|
||||||
|
assert.deepEqual(kinds(analyzeDiff(fixture('stub.diff'))), ['stub']);
|
||||||
|
assert.deepEqual(kinds(analyzeDiff(fixture('loosened-threshold.diff'))), ['loosened-threshold']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: a TIGHTENED threshold is not a finding (direction is read, not just change)', () => {
|
||||||
|
assert.deepEqual(analyzeDiff(fixture('tightened-threshold.diff')).findings, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: an unreadable diff -> status unreadable, exit 2 (never green)', () => {
|
||||||
|
assert.equal(analyzeDiff(fixture('unreadable.diff')).status, 'unreadable');
|
||||||
|
const r = cli(['--diff', join(FIX, 'unreadable.diff')]);
|
||||||
|
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||||
|
assert.equal(cli(['--diff', join(FIX, 'does-not-exist.diff')]).status, 2);
|
||||||
|
assert.equal(cli([]).status, 2, 'no input is "could not run", not clean');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: hunk line counts decide what is content - a removed "-- x" line is not a file header', () => {
|
||||||
|
const diff = [
|
||||||
|
'diff --git a/tests/a.test.mjs b/tests/a.test.mjs',
|
||||||
|
'--- a/tests/a.test.mjs',
|
||||||
|
'+++ b/tests/a.test.mjs',
|
||||||
|
'@@ -1,4 +1,3 @@',
|
||||||
|
" test('a', () => {",
|
||||||
|
'--- an old comment',
|
||||||
|
'+++ counter',
|
||||||
|
'- assert.equal(1, 1);',
|
||||||
|
' });',
|
||||||
|
'\\ No newline at end of file',
|
||||||
|
].join('\n');
|
||||||
|
const r = analyzeDiff(diff);
|
||||||
|
assert.deepEqual(r.findings.map((f) => [f.kind, f.file]), [['removed-assert', 'tests/a.test.mjs']]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: an empty diff is clean (nothing changed weakens nothing)', () => {
|
||||||
|
assert.equal(analyzeDiff('').status, 'clean');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: --json prints the findings with file and line', () => {
|
||||||
|
const r = cli(['--diff', join(FIX, 'removed-assert.diff'), '--json']);
|
||||||
|
const out = JSON.parse(r.stdout);
|
||||||
|
assert.equal(out.status, 'findings');
|
||||||
|
assert.equal(out.findings[0].kind, 'removed-assert');
|
||||||
|
assert.equal(out.findings[0].file, 'tests/sum.test.mjs');
|
||||||
|
assert.equal(typeof out.findings[0].line, 'number');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: --base reads the diff from git in --cwd', () => {
|
||||||
|
const dir = mkdtempSync(join(tmpdir(), 'floor-guard-'));
|
||||||
|
const git = (...a) => {
|
||||||
|
const r = spawnSync('git', a, { cwd: dir, encoding: 'utf8' });
|
||||||
|
assert.equal(r.status, 0, `git ${a.join(' ')}: ${r.stderr}`);
|
||||||
|
return r.stdout.trim();
|
||||||
|
};
|
||||||
|
try {
|
||||||
|
git('init', '-q');
|
||||||
|
git('config', 'user.email', 't@example.invalid');
|
||||||
|
git('config', 'user.name', 'test');
|
||||||
|
writeFileSync(join(dir, 'a.test.mjs'), "test('a', () => {\n assert.equal(1, 1);\n assert.equal(2, 2);\n});\n");
|
||||||
|
git('add', 'a.test.mjs');
|
||||||
|
git('commit', '-q', '-m', 'base');
|
||||||
|
const base = git('rev-parse', 'HEAD');
|
||||||
|
writeFileSync(join(dir, 'a.test.mjs'), "test('a', () => {\n assert.equal(1, 1);\n});\n");
|
||||||
|
const r = cli(['--base', base, '--cwd', dir, '--json']);
|
||||||
|
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||||
|
assert.deepEqual(JSON.parse(r.stdout).findings.map((f) => f.kind), ['removed-assert']);
|
||||||
|
assert.equal(cli(['--base', 'no-such-ref', '--cwd', dir]).status, 2, 'an unresolvable base is "could not run"');
|
||||||
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||||
|
});
|
||||||
|
|
||||||
|
test('floor guard: test-path recognition covers the common layouts', () => {
|
||||||
|
for (const p of ['tests/a.mjs', 'test/a.py', 'src/__tests__/a.ts', 'a.test.js', 'pkg/a_test.go', 'app/test_a.py', 'spec/a_spec.rb', 'a.spec.ts']) {
|
||||||
|
assert.ok(isTestPath(p), p);
|
||||||
|
}
|
||||||
|
for (const p of ['lib/a.mjs', 'src/contest.ts', 'README.md']) assert.ok(!isTestPath(p), p);
|
||||||
|
});
|
||||||
Loading…
Add table
Add a link
Reference in a new issue