merge: wave 1 order 1 (executor) into main
CLAUDE.md command table: each row from its owner (trekplan from order 3, trekexecute from order 1, trekreview from order 5). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
ca6ee8779a
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
28 changed files with 2384 additions and 116 deletions
|
|
@ -32,7 +32,7 @@ Voyage — a contract-driven Claude Code pipeline: brief, research, plan, execut
|
|||
| `/trekbrief` | Brief — one question round sized by class (`ceremony: small\|bounded\|architectural`, heaviest in doubt, said aloud, only moved up) produces a task brief with explicit research plan; ends with the operator explicitly approving `## Intent` + `## Goal` («høres bra ut» is not approval), logged in the target repo's hash-chained approval log and committed (the brief too when `artifact_store: git`) (`--approve <dir>` does only the approval, then writes the trekbrief record); optionally orchestrates the pipeline | session |
|
||||
| `/trekresearch` | Research — deep local + external research, produces structured research brief. Opt-in `--engine {swarm\|deep-research}` delegates the external phase to Claude Code's built-in `/deep-research` workflow (swarm default) | session |
|
||||
| `/trekplan` | Plan — brief-reviewer, explore, plan, review. Requires `--brief` or `--project`; halts unless the brief's intent is approved (hash-bound, stale on edit). Auto-discovers `architecture/overview.md` if present. Plan 1.8: `Test first:` per step + one `End-to-end:` step; a design part for `ceremony: architectural` (absent = architectural); critic loop of max 3 rounds, recorded in the plan, and `plan-gate.mjs` (0 go / 1 stop / 2 could not run) decides | session |
|
||||
| `/trekexecute` | Execute — disciplined plan/session-spec executor with failure recovery | session |
|
||||
| `/trekexecute` | Execute — disciplined plan/session-spec executor: one fresh `step-implementer` helper per plan step, one controller that reads the diff (spec + quality verdicts), proof of red / floor guard / done line as commands, fix rounds that go up (max 5: helper → plan → operator), closed stop list + decision log in the final report | session |
|
||||
| `/trekreview` | Review — a gate: Claude Code's `/code-review` + `/security-review` as engine, then Voyage's brief control (which stops); writes `review.md` with a required verdict that `lib/review/verdict-gate.mjs` decides on (0 ALLOW/WARN · 1 BLOCK · 2 never green); proposes recurring finding classes as project memory, line by line (Handover 6) | session |
|
||||
| `/trekcontinue` | Continue — resumes the next session of a multi-session voyage project. Reads `.session-state.local.json` (Handover 7) and immediately begins executing | session |
|
||||
| `/trekendsession` | End-session — mark the current session complete and write session-state pointing at the next session. Helper for informal multi-session flows | session |
|
||||
|
|
@ -96,15 +96,16 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
|
|||
| plan-critic | opus | Adversarial plan review (12 dimensions) |
|
||||
| scope-guardian | opus | Scope alignment (creep + gaps) |
|
||||
| session-decomposer | opus | Splits plans into headless sessions with dependency graph |
|
||||
| step-implementer | opus | `/trekexecute` step helper — one fresh helper per plan step: test first, proof of red, test lock, then exactly the step's Changes; ran / saw / did-NOT-check report (no Agent tool) |
|
||||
| synthesis-agent | opus | Distills Phase-5/7 exploration outputs into a findings digest (NW3 PoC — **dormant**, not wired; delegating Phase 7 alone yields Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`) |
|
||||
| docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) |
|
||||
| community-researcher | opus | Community experience: issues, blogs, discussions |
|
||||
| security-researcher | opus | CVEs, audit history, supply chain risks |
|
||||
| contrarian-researcher | opus | Counter-evidence, overlooked alternatives |
|
||||
|
||||
> **Inventory (S33 reconcile, updated v5.10.1).** 23 agent files = **20 spawnable** (one, `synthesis-agent`, ships **dormant** — Δ≈0, wired to nothing) **+ 3 orchestrator reference docs** (`planning-/research-/review-orchestrator` document the inline `/trek*` workflow, not spawnable capabilities). All 23 stay `model: opus` (operator pin `40d8742`); the glue/mechanical/retrieval/dormant roles were reconsidered for a sonnet downgrade and **kept opus** — decision record: `docs/voyage-vs-cc-balance-analysis.md` §10.
|
||||
> **Inventory (S33 reconcile, updated wave 1).** 24 agent files = **21 spawnable** (one, `synthesis-agent`, ships **dormant** — Δ≈0, wired to nothing) **+ 3 orchestrator reference docs** (`planning-/research-/review-orchestrator` document the inline `/trek*` workflow, not spawnable capabilities). All 24 stay `model: opus` (operator pin `40d8742`); the glue/mechanical/retrieval/dormant roles were reconsidered for a sonnet downgrade and **kept opus** — decision record: `docs/voyage-vs-cc-balance-analysis.md` §10.
|
||||
|
||||
> **Model & effort.** On the Anthropic API, `opus` = Opus 5.5 and `sonnet` = Sonnet 5.5 (both default reasoning effort `medium`); `fable` = Fable 5.1 (the most capable model in Claude Code; default `high`, so xhigh must be chosen explicitly). An agent without `effort:` inherits the session level. Eight agents pin native per-spawn `effort: high` and the rest follow the session; none pins below `high` — a different axis from brief `phase_signals.effort` (orchestration shape: which agents/passes run). Per-agent table + axes → `docs/profiles.md` §Model & effort axes.
|
||||
> **Model & effort.** On the Anthropic API, `opus` = Opus 5.5 and `sonnet` = Sonnet 5.5 (both default reasoning effort `medium`); `fable` = Fable 5.1 (the most capable model in Claude Code; default `high`, so xhigh must be chosen explicitly). An agent without `effort:` inherits the session level. Nine agents pin native per-spawn `effort: high` and the rest follow the session; none pins below `high` — a different axis from brief `phase_signals.effort` (orchestration shape: which agents/passes run). Per-agent table + axes → `docs/profiles.md` §Model & effort axes.
|
||||
|
||||
## Reference docs (read on demand)
|
||||
|
||||
|
|
|
|||
|
|
@ -34,7 +34,7 @@ Changes to this plugin must preserve:
|
|||
| `commands/trekresearch.md` | The `/trekresearch` slash command — research orchestration |
|
||||
| `commands/trekplan.md` | The `/trekplan` slash command — planning orchestration |
|
||||
| `commands/trekexecute.md` | The `/trekexecute` slash command — execution orchestration |
|
||||
| `agents/*.md` | 19 specialized agents for research, exploration, review, and orchestration |
|
||||
| `agents/*.md` | 24 agent files (21 spawnable + 3 orchestrator reference docs) for research, exploration, review, execution, and orchestration |
|
||||
| `templates/plan-template.md` | Structured plan output format |
|
||||
| `templates/research-brief-template.md` | Research brief format with triangulation and confidence |
|
||||
| `templates/spec-template.md` | Spec file format |
|
||||
|
|
|
|||
40
README.md
40
README.md
|
|
@ -317,7 +317,25 @@ Borrowed, with credit: per-step interfaces, Global Constraints and Review Focus
|
|||
|
||||
Reads a plan from `/trekplan` and implements it with strict discipline. No guessing, no improvising — follows the plan exactly.
|
||||
|
||||
Per step: apply Changes exactly as written → run Verify (exit code is truth) → manifest audit (expected paths, forbidden paths, commit pattern) → follow the plan's failure clause if anything fails (revert / retry / skip / escalate) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery.
|
||||
Per step: record the start point → **one fresh helper** (`step-implementer`) writes the step's test first, proves it red, locks it, then applies Changes exactly as written → the executor proves it by command: **red at the start point, green at the result** (`red-proof.mjs`), Verify (exit code is truth), manifest audit, and the **floor guard** on the diff (`test-integrity.mjs`: removed asserts, skipped or deleted tests, muted checks, stubs, loosened thresholds) → **one controller** reads the diff — never the helper's summary — and gives two verdicts, *spec* and *quality* → a script writes the step's **done line** only when the test command exits 0 (`done-line.mjs`) → Checkpoint commit. After all steps: independent Phase 7.5 manifest audit from git log + filesystem (ignoring agent bookkeeping); drift triggers Phase 7.6 bounded recovery. A plan with no passing run-the-thing criterion ends **not verified end to end**, never green.
|
||||
|
||||
### The step loop
|
||||
|
||||
| Part | What it does | Why |
|
||||
|------|--------------|-----|
|
||||
| Fresh helper per step | `voyage:step-implementer` (opus, effort high, no Agent tool) builds one step and writes a report: **ran · saw · did NOT check** | Each step starts from the repository and the plan, not from a long context |
|
||||
| Start point | `start_sha` recorded before the step; every check reads `git diff {start_sha}` | The review package is the diff, not a story |
|
||||
| Proof of red | the step's test must fail at the start point and pass at the result, and a control run (the start point plus the change) must pass too — run by `red-proof.mjs`, not claimed | A test that passes without the change proves nothing; a start point that fails for an environment reason proves nothing either |
|
||||
| Test lock | the helper locks its test files after the red proof (`test-lock.mjs`, when installed; otherwise "not-locked" is reported) | The code moves to meet the test, never the reverse |
|
||||
| Floor guard | `test-integrity.mjs` reads the diff for weakened tests; exit 2 ("could not run") is never green | Cheating the tests is caught by a command |
|
||||
| Controller | one general helper, two verdicts (spec, quality), re-checks only what a fix round changed | An independent judge of the diff |
|
||||
| Done line | `done-line.mjs` appends `DONE step N` only when the test command exits 0 | "Done" is an exit code |
|
||||
|
||||
The helper works in your working tree, not in a worktree of its own: measured 2026-10-06, a worktree-isolated subagent branches from the default branch rather than your `HEAD`, so it would not see the earlier steps (only the operator setting `worktree.baseRef: "head"` changes that).
|
||||
|
||||
### Unattended: what stops a run
|
||||
|
||||
A run stops only on a **closed list**: something that cannot be undone · something security-sensitive · something outside the working copy (merge to main, push, publish) · a plan so broken that everything after it is guesswork. Everything else the executor decides itself and logs in the final report's **Decisions taken** table, each with what it costs if it is wrong.
|
||||
|
||||
### Modes
|
||||
|
||||
|
|
@ -358,11 +376,11 @@ Use `--fg` to run every step sequentially in the current session instead, even w
|
|||
|
||||
### Failure recovery
|
||||
|
||||
- **3-attempt retry cap** — retries twice, then stops (never loops forever)
|
||||
- **On failure: revert** — undo changes, stop
|
||||
- **On failure: retry** — try alternative approach, then revert if still failing
|
||||
- **Fix rounds go up, maximum 5** — rounds 1–3: a fresh helper fixes exactly the controller's findings; after three failed fixes the question is the architecture: round 4 sends the step back to planning (`/trekplan` revises it, then `--resume`), round 5 to you. Never a fourth guess, never a cheaper model
|
||||
- **On failure: revert** — fix rounds, then undo the step's changes when it goes up
|
||||
- **On failure: retry** — the alternative approach goes to the first fix round; revert when it goes up
|
||||
- **On failure: skip** — non-critical step, continue
|
||||
- **On failure: escalate** — stop everything, needs human judgment
|
||||
- **On failure: escalate** — stop at the first failed round, needs human judgment
|
||||
|
||||
### Security hardening
|
||||
|
||||
|
|
@ -686,7 +704,7 @@ Solid arrows are the forward pipeline; dashed are conditional/internal edges (`-
|
|||
|
||||
### Agents per phase
|
||||
|
||||
Every command orchestrates its agents inline from the main context (all sub-agents are `model: opus`-pinned). `/trekexecute` is the exception — it spawns **no** sub-agents.
|
||||
Every command orchestrates its agents inline from the main context (all sub-agents are `model: opus`-pinned). `/trekexecute` spawns no swarm: one fresh `step-implementer` per plan step, in order, and one controller per round.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
|
|
@ -703,8 +721,8 @@ flowchart TB
|
|||
PLC["Phase 9 — parallel · plan-critic · scope-guardian"]
|
||||
PLG --> PLS --> PLC
|
||||
end
|
||||
subgraph EX["/trekexecute · no sub-agents"]
|
||||
EXI["inline step-loop + session orders per wave (worktrees)<br/>+ Phase 7.5/7.6 manifest audit (deterministic)"]
|
||||
subgraph EX["/trekexecute · one helper per step"]
|
||||
EXI["per step: step-implementer → controller (spec + quality)<br/>+ red proof · floor guard · done line (commands)<br/>+ session orders per wave (worktrees) + Phase 7.5/7.6 audit"]
|
||||
end
|
||||
subgraph RV["/trekreview"]
|
||||
RVR["Phase 5 — parallel · code-correctness-reviewer · brief-conformance-reviewer*"]
|
||||
|
|
@ -866,7 +884,7 @@ Default JSONL stats stream (`${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl`) is unchan
|
|||
|
||||
The default `premium` profile runs **Opus on every phase** of the pipeline's agent work — the exploration and review swarms (5–10 sub-agents per command; spawn sites inject the composed brief > profile > frontmatter resolution, with `agents/*.md` `model: opus` pins as the fallback) and the executor (one per plan session). The command orchestrator itself is not profile-controlled: as of v5.9, command frontmatter omits `model:`, so the orchestrator follows the session model. For cheaper runs, opt into `--profile balanced` (Sonnet on brief/research/execute/continue, Opus on plan + review) or `--profile economy` (Sonnet everywhere); for maximum quality, `--profile fable` (Fable 5.1 on every phase). Per-command cost is published in `${CLAUDE_PLUGIN_DATA}/trek*-stats.jsonl` if you want exact numbers.
|
||||
|
||||
On the Anthropic API the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5** (both default to reasoning effort `medium`), and `fable` to **Fable 5.1** (the most capable model in Claude Code; default reasoning effort `high` — xhigh must be chosen explicitly, see [`docs/profiles.md`](docs/profiles.md)). Other providers resolve `opus` and `sonnet` to other versions. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` (eight agents pin `high`; the rest follow the session) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
||||
On the Anthropic API the `opus` alias resolves to **Opus 5.5** and `sonnet` to **Sonnet 5.5** (both default to reasoning effort `medium`), and `fable` to **Fable 5.1** (the most capable model in Claude Code; default reasoning effort `high` — xhigh must be chosen explicitly, see [`docs/profiles.md`](docs/profiles.md)). Other providers resolve `opus` and `sonnet` to other versions. Note two distinct effort axes that share the word "effort": brief `phase_signals.effort` (low/standard/high) tunes *orchestration shape* — how many agents and passes run — while native `effort:` (nine agents pin `high`; the rest follow the session) tunes the *per-spawn reasoning budget*. See [`docs/profiles.md`](docs/profiles.md) § Model & effort axes.
|
||||
|
||||
For per-profile cost estimates, see [`docs/profiles.md`](docs/profiles.md).
|
||||
|
||||
|
|
@ -882,8 +900,8 @@ Top-level layout:
|
|||
|
||||
```
|
||||
trekplan/
|
||||
├── agents/ 23 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
||||
│ └ 20 spawnable (1 dormant: synthesis-agent, Δ≈0) + 3 orchestrator reference docs (not spawned)
|
||||
├── agents/ 24 specialized agents (all `model: opus`-pinned; per-phase model set by `--profile`)
|
||||
│ └ 21 spawnable (1 dormant: synthesis-agent, Δ≈0) + 3 orchestrator reference docs (not spawned)
|
||||
├── commands/ 6 slash commands (trekbrief, trekresearch, trekplan, trekexecute, trekreview, trekcontinue) + trekendsession helper
|
||||
├── templates/ Frontmatter templates for brief, research, plan, session
|
||||
├── hooks/ 12 hooks (pre-bash, pre-write, pre-agent-cap, pre-plan-gate, session-title, post-bash-stats, post-approval-log, post-edit-check, pre-compact-flush, post-compact-flush, otel-export, stop-test-lock)
|
||||
|
|
|
|||
127
agents/step-implementer.md
Normal file
127
agents/step-implementer.md
Normal file
|
|
@ -0,0 +1,127 @@
|
|||
---
|
||||
name: step-implementer
|
||||
description: |
|
||||
Use this agent when /trekexecute hands one plan step to a fresh helper: it
|
||||
writes the step's test first, proves it red, locks it, implements exactly the
|
||||
step's Changes, and writes a ran / saw / did-NOT-check report to a file.
|
||||
model: opus
|
||||
effort: high
|
||||
color: green
|
||||
tools: ["Read", "Write", "Edit", "Bash", "Glob", "Grep"]
|
||||
---
|
||||
|
||||
You are the helper for ONE step of a Voyage plan. You start fresh: you know
|
||||
nothing about earlier steps except what is in the repository and the files the
|
||||
executor names. You do one step, report it in a file, and stop. You never start
|
||||
another helper (you have no Agent tool, by design), never commit, never push,
|
||||
and never touch a file the step does not list.
|
||||
|
||||
## What the executor gives you
|
||||
|
||||
- `plan` — path to the plan; `plan_version`; `step` — the step number; the step's text verbatim
|
||||
(Files, Changes, Reuses, Test first, Verify, and when present Interface in,
|
||||
Interface out, Review focus).
|
||||
- `start_sha` — the commit the step starts from. Everything you change is read
|
||||
later as `git diff {start_sha}`, so the diff is your real report.
|
||||
- `report_path` — where your report goes.
|
||||
- `mode` — `build` (the step's first helper) or `fix` (a fix round: the
|
||||
controller's findings to fix are included; fix only those).
|
||||
- `voyage_root` — the plugin root, for the CLIs below.
|
||||
|
||||
## Mode `build`
|
||||
|
||||
1. **Test first.** Read the step's `Test first:` and write that test (and only
|
||||
that test) first. Follow the `Pattern:` file's style when one is named.
|
||||
Then register every new file you created so the executor's diff sees it:
|
||||
`git add --intent-to-add -- <path>` (this stages nothing).
|
||||
2. **Prove it red before you build.** Run, from the repository root:
|
||||
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files as ONE comma-separated argument: a,b>`
|
||||
Exit 0 = the test fails at the start point: continue. Exit 1 = the test
|
||||
already passes at the start point, so it proves nothing: rewrite the test
|
||||
until it fails for the reason the step exists, and run the check again.
|
||||
Exit 2 = it could not run (the Verify command is not one test-runner
|
||||
invocation, or a file is missing): on `plan_version` ≥ 1.8 do not build;
|
||||
say so in the report and stop. Below 1.8, write "red proof not run:
|
||||
{reason}" in the report and go on to step 4 without a lock.
|
||||
|
||||
**A step with no `Test first:`** (allowed only below plan_version 1.8):
|
||||
skip steps 1–3, write "no Test first: no test written, no proof of red" in
|
||||
the report, and go to step 4.
|
||||
3. **Lock the tests** (when the CLI exists):
|
||||
`node {voyage_root}/lib/util/test-lock.mjs lock --files <test files> --step {step}`.
|
||||
If the file does not exist, write "not-locked (test-lock.mjs missing)" in the
|
||||
report. Never silent. Once locked you cannot edit the tests, and that is the
|
||||
point: the code changes to meet the test, never the reverse.
|
||||
4. **Implement exactly the step's `Changes:`** in the files its `Files:` lists
|
||||
(and its test files from step 1 — nothing else; register any new file
|
||||
with `git add --intent-to-add -- <path>`).
|
||||
Read `Reuses:` first. No extra scope, no refactors on the side, no "while I
|
||||
was here". If the step cannot be done as written, stop and say why in the
|
||||
report: the executor sends it up to the plan; you do not improvise.
|
||||
5. **Run the step's `Verify:` command** yourself, fresh. Its exit code is the
|
||||
truth, not your reading of the output.
|
||||
|
||||
## Mode `fix`
|
||||
|
||||
You get the controller's findings (file, line, problem). Fix those and nothing
|
||||
else. The tests stay locked: if a finding says the test itself is wrong, do not
|
||||
work around it - say so in the report; that goes up to the plan. Run the
|
||||
step's `Verify:` again.
|
||||
|
||||
The one exception: a `not-red-at-base` finding (the test passes without the
|
||||
change). The executor has released the lock for that round. Rewrite the test
|
||||
so it fails for the reason the step exists, pass the `--red-only` gate again
|
||||
(build step 2), and lock it again (build step 3).
|
||||
|
||||
## Never
|
||||
|
||||
- Never edit a test to make it pass: no deleted assert, no skip, no looser
|
||||
threshold, no `|| true`. The executor's floor guard reads your diff for
|
||||
exactly that, and finds it.
|
||||
- Never write a stub ("not implemented", `TODO: implement`) as the step's body.
|
||||
- Never write a `DONE` line. The executor's script writes it, and only when
|
||||
the test command exits 0.
|
||||
- Never commit, push, merge or tag. Never write outside the repository.
|
||||
|
||||
## The report (APPEND it to `report_path`, then answer with one line)
|
||||
|
||||
The report file holds every round of the step and the controller's verdicts.
|
||||
Add your section at the end — with Edit, or by rewriting the file with every
|
||||
earlier section kept word for word. Never remove or rewrite what is there.
|
||||
Use the round label the executor gave you in the heading.
|
||||
|
||||
```markdown
|
||||
## Step {N} - {build | fix round R}
|
||||
|
||||
### Ran
|
||||
- `{command}` -> exit {code}
|
||||
(every command you ran, in order, with its exit code)
|
||||
|
||||
### Saw
|
||||
- {what the output actually showed: the failing test name at the start point,
|
||||
the passing count after, a warning you did not fix}
|
||||
|
||||
### Did NOT check
|
||||
- {what you did not verify, plainly: other platforms, the full suite, an edge
|
||||
case the plan names but no test covers, anything you assumed}
|
||||
|
||||
### Files touched
|
||||
- {path} ({new | modified})
|
||||
|
||||
### Test lock
|
||||
- {locked | not-locked (test-lock.mjs missing) | not-locked (reason)}
|
||||
```
|
||||
|
||||
"Did NOT check" is never empty: if you truly checked everything the step
|
||||
names, say what lies outside the step that you did not look at.
|
||||
|
||||
Your final answer to the executor is one line: `report: {report_path}`. The
|
||||
executor does not read your summary to decide anything; it reads the diff, and a
|
||||
controller judges it.
|
||||
|
||||
## Credits
|
||||
|
||||
The loop this helper lives in is adapted from obra/superpowers@8ca22db (MIT,
|
||||
(c) 2025 Jesse Vincent), subagent-driven-development: a fresh helper per task,
|
||||
hand-over in files. The ran / saw / did-NOT-check report contract is borrowed
|
||||
from bashebr/ai-native-sdlc@11fb8d7 (MIT).
|
||||
|
|
@ -2,8 +2,8 @@
|
|||
name: trekexecute
|
||||
description: Disciplined plan executor — single-session, or multi-session through session orders the user starts, with failure recovery
|
||||
argument-hint: "[--project <dir>] [--fg | --resume | --dry-run | --validate | --step N | --session N] [plan.md]"
|
||||
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion
|
||||
disallowed-tools: Agent, TeamCreate
|
||||
allowed-tools: Read, Write, Edit, Bash, Glob, Grep, AskUserQuestion, Agent
|
||||
disallowed-tools: TeamCreate
|
||||
---
|
||||
|
||||
# Ultraexecute Local
|
||||
|
|
@ -12,9 +12,17 @@ Disciplined executor for trekplan plans. Reads a plan file, detects if it has
|
|||
an Execution Strategy (multi-session), and either executes directly or
|
||||
prepares one session order per session, wave by wave — all to realize one plan.
|
||||
|
||||
Each plan step is built by **one fresh helper** (`voyage:step-implementer`)
|
||||
and judged by **one controller** that reads the step's diff. The executor
|
||||
itself writes no product code: it records the start point, launches the
|
||||
helper, runs the mechanical checks (proof of red, the floor guard, the
|
||||
manifest, the done line), and decides from exit codes and the controller's
|
||||
verdicts (Phase 6).
|
||||
|
||||
Voyage never starts a Claude Code session. Work that needs another session
|
||||
becomes an order plus a starter line the user gives a session they open
|
||||
themselves (Phase 2.6).
|
||||
themselves (Phase 2.6). Helpers inside this session are subagents, and helpers
|
||||
never start helpers.
|
||||
|
||||
## Phase 1 — Parse mode and validate input
|
||||
|
||||
|
|
@ -120,15 +128,34 @@ Extract every `### Step N: {description}` heading (in order). For each step, ext
|
|||
- **Files** — file paths to create or modify
|
||||
- **Changes** — what to modify
|
||||
- **Reuses** — existing code to leverage (informational)
|
||||
- **Test first** — test to run before implementation (optional)
|
||||
- **Verify** — command to run after implementation
|
||||
- **Test first** — the test the step writes before it builds: its `File:`
|
||||
(existing | new), what it `Verifies:`, an optional `Pattern:` (required from
|
||||
plan_version ≥ 1.8, see below)
|
||||
- **Verify** — command to run after implementation. It is also the step's
|
||||
**test command**: proof of red (Sub-step B) and the done line (Sub-step G)
|
||||
run it, so it must be one test-runner invocation
|
||||
- **On failure** — recovery action (revert/retry/skip/escalate)
|
||||
- **Checkpoint** — git commit command after success
|
||||
- **Manifest** — YAML block following Checkpoint (v1.7+)
|
||||
- **Interface in** / **Interface out** / **Review focus** — optional (K4):
|
||||
what the step consumes and produces, and what its controller must look at
|
||||
hardest. Read when present and passed to the helper and the controller;
|
||||
never required, and the executor tolerates their absence in every plan
|
||||
version.
|
||||
|
||||
If a step is missing `On failure`, default to `escalate` and record a parse warning.
|
||||
If a step is missing `Verify`, record a parse warning.
|
||||
|
||||
**`Test first:` is required from plan_version ≥ 1.8.** A ≥ 1.8 step without a
|
||||
`Test first:` field (or whose `File:` is missing) is not executable: STOP with
|
||||
```
|
||||
Error: plan_version={v} but step {N} has no Test first: field.
|
||||
Re-run /trekplan - from plan_version 1.8 every step proves red before it builds.
|
||||
```
|
||||
For plans below 1.8 the field stays optional: a step without it runs, its
|
||||
`red_proof` is recorded as `"not-run"`, and the final report counts it among
|
||||
the steps without proof of red (never hidden, never called proven).
|
||||
|
||||
### Parse the Manifest block (v1.7+)
|
||||
|
||||
Read the plan header for a `plan_version` marker. Treat ≥ `1.7` as strict
|
||||
|
|
@ -1021,18 +1048,39 @@ progress file.
|
|||
"mode": "{execute | resume | step}",
|
||||
"total_steps": 0,
|
||||
"current_step": 0,
|
||||
"session_start_sha": null,
|
||||
"status": "{in_progress | completed | failed | stopped | partial}",
|
||||
"steps": {
|
||||
"1": { "status": "pending", "attempts": 0, "error": null, "completed_at": null, "commit": null }
|
||||
"1": { "status": "pending", "attempts": 0, "fix_rounds": 0, "start_sha": null, "report_path": null, "red_proof": null, "test_lock": null, "error": null, "completed_at": null, "commit": null }
|
||||
},
|
||||
"iterations_remaining": 25,
|
||||
"entry_condition_checked": false,
|
||||
"exit_condition_checked": false,
|
||||
"plan_verification": null,
|
||||
"end_to_end": null,
|
||||
"decisions": [],
|
||||
"summary": null
|
||||
}
|
||||
```
|
||||
|
||||
**The step-loop fields (Handover 5, additive-optional).** Written by Phase 6;
|
||||
a progress file without them still validates and resumes:
|
||||
|
||||
| Field | Written | Meaning |
|
||||
|---|---|---|
|
||||
| `session_start_sha` | Phase 3 | the commit the run started from; Phase 7.5 audits the whole run from it |
|
||||
| `steps.{N}.start_sha` | Sub-step 0 | the commit the step started from; every check of the step reads `git diff {start_sha}` |
|
||||
| `steps.{N}.report_path` | Sub-step 0 | the step's report file (helper rounds + the done line) |
|
||||
| `steps.{N}.red_proof` | Sub-step B | `"proven" \| "rejected" \| "not-run"` — the exit of `red-proof.mjs` |
|
||||
| `steps.{N}.test_lock` | Sub-steps B, G | `"locked" \| "released" \| "not-locked"` (K8) |
|
||||
| `steps.{N}.fix_rounds` | Sub-step F | 0–5: 0 = passed its first control, 1–3 = fix rounds, 4 = sent up to the plan, 5 = sent up to the operator |
|
||||
| `steps.{N}.attempts` | Sub-steps C, F | helpers launched for the step: 1 + the fix rounds that launched one |
|
||||
| `end_to_end` | Phase 7 | `"verified" \| "not-verified"` (P9 (6)) |
|
||||
| `decisions` | any phase | `[{step, decision, cost_if_wrong}]` — every call the executor made on its own (Phase 5.5) |
|
||||
|
||||
`progress-validator.mjs` rejects `fix_rounds` outside 0–5 (there is no sixth
|
||||
round) and a `decisions` entry without its `cost_if_wrong`.
|
||||
|
||||
`plan_version` is the value Phase 2 parsed from the plan header (required by
|
||||
`progress-validator.mjs`). The status words here are the validator's:
|
||||
`PROGRESS_VALID_STATUSES` for the run, `PROGRESS_STEP_STATUSES` for a step.
|
||||
|
|
@ -1047,8 +1095,8 @@ condition instead) and on early stop. Unknown keys are tolerated by
|
|||
recovery/retry budget against the global hard cap (unit = **iterations**, per the
|
||||
brief — token/cost budgeting is out of scope). Seeded to
|
||||
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default 25) at start. **Single writer:** the
|
||||
executor decrements it by 1 on **every recovery/retry iteration** — each step
|
||||
attempt beyond the first, and each Phase 7.6 recovery dispatch. **Backfill-on-absent:**
|
||||
executor decrements it by 1 on **every recovery/retry iteration** — each fix
|
||||
round of a step (Sub-step F), and each Phase 7.6 recovery dispatch. **Backfill-on-absent:**
|
||||
when resuming a legacy `progress.json` that lacks the field, seed it to the cap
|
||||
rather than hard-failing (forward/backward-compatible; shape-validated by
|
||||
`progress-validator.mjs` — non-negative integer, never required-top). The field is
|
||||
|
|
@ -1056,6 +1104,12 @@ present in the spec at minimum — there is **no `(if wired)` conditional**.
|
|||
|
||||
### Mode-specific behavior
|
||||
|
||||
**Run start point (every mode that executes):** a fresh progress file records
|
||||
`session_start_sha` = `git rev-parse HEAD` before step 1; a resumed one keeps
|
||||
the value it has (a legacy file without it is backfilled with the parent of
|
||||
the first completed step's `commit`, or `HEAD` when none is completed). Phase
|
||||
7.5 audits the whole run from it — never from a step's own `start_sha`.
|
||||
|
||||
**mode = execute (fresh):**
|
||||
- If a progress file exists with status `in_progress`, `failed`, `stopped` or `partial`: warn that
|
||||
`--resume` is available, then wait 3 seconds (`sleep 3`) and start fresh.
|
||||
|
|
@ -1160,6 +1214,55 @@ If `has_execution_strategy = true`, show a preview of multi-session orchestratio
|
|||
|
||||
Stop after the dry-run report. Do not execute anything.
|
||||
|
||||
## Phase 5.5 — Autonomy: what stops a run, and the decision log
|
||||
|
||||
A run is long and mostly unattended. It does not stop to ask: what it decides
|
||||
on its own it writes down, and it stops only for the four reasons below.
|
||||
|
||||
### Closed stop list
|
||||
|
||||
The list is **closed**: a run stops on these four classes and on nothing else.
|
||||
Every stop names its class in the final report (Phase 8 `### Stops`).
|
||||
|
||||
1. **Something that cannot be undone.** A step or command whose effect cannot
|
||||
be reverted from git: deleting data outside the repo's tracked files, a
|
||||
migration against a live store, anything the plan marks irreversible — and
|
||||
a dirty main checkout before a multi-session wave (Phase 2.55 Check 1:
|
||||
merging would lose the uncommitted work).
|
||||
2. **Something security-sensitive.** A command the executor denylist blocks
|
||||
(Phase 2.4, Sub-step D1), a write to a security-sensitive path (Hard
|
||||
rule 16), credentials or secrets in a diff.
|
||||
3. **Something outside the working copy.** Merge to the main branch, push,
|
||||
tag, publish, a write outside the repository (Hard rule 15). The main-merge
|
||||
gate (Phase 8) is this class: it always pauses.
|
||||
4. **A plan so broken that everything after it is guesswork.** The plan fails
|
||||
to parse, a ≥ 1.8 step has no `Test first:`, a step's own `On failure:
|
||||
escalate` fired (the plan's author declared that past this point is
|
||||
guesswork), a step went up the fix ladder (Sub-step F), the global
|
||||
iteration budget reached 0 — and the plan's own preconditions failing: an
|
||||
unmet entry condition (Phase 4), or the sandbox pre-flight exiting 77
|
||||
(Sub-step D1: the plan cannot be carried out where it runs).
|
||||
|
||||
Every stop the executor can make maps to one of these four; `stop_class` in
|
||||
the summary is that number.
|
||||
|
||||
Not on the list, and therefore never a stop: a failing test (that is a fix
|
||||
round), an ambiguity the plan leaves open (decide, and log it), a style
|
||||
choice, a missing optional field.
|
||||
|
||||
### Decision log
|
||||
|
||||
Every decision the executor (or a helper, through its report) takes that the
|
||||
plan did not dictate goes into progress `decisions` as one entry:
|
||||
`{"step": N, "decision": "...", "cost_if_wrong": "..."}`. `cost_if_wrong` is
|
||||
concrete: what has to be redone, by whom, how big — "one rename commit",
|
||||
"step 4's API shape; steps 5–7 build on it". The final report prints the log
|
||||
(Phase 8 `### Decisions taken`), so the operator reviews the calls instead of
|
||||
being asked them one by one.
|
||||
|
||||
Borrowed: Superpowers subagent-driven-development (MIT) — the stop classes and
|
||||
the log of choices made while unattended.
|
||||
|
||||
## Phase 6 — Step execution loop
|
||||
|
||||
The core execution phase. Runs for modes: `execute`, `resume`, `step`.
|
||||
|
|
@ -1179,6 +1282,33 @@ Update progress: `steps.{N}.status = "in_progress"`, `current_step = N`, `update
|
|||
--- Step {N}/{total}: {description} ---
|
||||
```
|
||||
|
||||
One step at a time. Inside one session the steps run in order in one working
|
||||
tree; parallel steps are the multi-session wave path (Phase 2.6), never two
|
||||
helpers in one tree.
|
||||
|
||||
#### Sub-step 0 — Record the start point
|
||||
|
||||
Before anything touches the tree, record where the step begins:
|
||||
|
||||
```bash
|
||||
git rev-parse HEAD
|
||||
git status --porcelain
|
||||
```
|
||||
|
||||
- `steps.{N}.start_sha` = the `HEAD` sha. Every later check of this step reads
|
||||
`git diff {start_sha}`: the review package is the diff from this recorded
|
||||
start point, not anything the helper says. On `--resume` of a step that
|
||||
already has a `start_sha`, keep it.
|
||||
- `steps.{N}.report_path` = `{progress-dir}/steps/step-{N}.report.md` (create
|
||||
the directory). Every helper round of the step appends to it, and the done
|
||||
line lands in it (Sub-step G). Hand-over between the executor, the helper
|
||||
and the controller is in files: the plan, this report, the diff.
|
||||
- A non-empty `git status --porcelain` (beyond Voyage's own progress and report
|
||||
files) means work the plan does not know about would mix into this step's
|
||||
diff. Not a stop (Phase 5.5): log it as a decision
|
||||
(`cost_if_wrong`: "the controller judges unrelated changes as part of step N")
|
||||
and name the paths in the report.
|
||||
|
||||
#### Sub-step A — Scope fence check (session specs only)
|
||||
|
||||
Before touching any file, verify that every file in the step's `Files:` field is
|
||||
|
|
@ -1192,24 +1322,118 @@ Escalating — this step cannot be executed within this session's scope.
|
|||
|
||||
Treat this as an automatic `escalate`. Jump to the stop-and-report logic.
|
||||
|
||||
#### Sub-step B — Test first (if present)
|
||||
#### Sub-step B — Test first and proof of red
|
||||
|
||||
If the step has a `Test first:` field:
|
||||
1. If test file is marked `(new)`: note it will be created during implementation.
|
||||
2. If test file exists: run it. Expect failure (RED state).
|
||||
3. If test unexpectedly passes: warn but continue — step may already be done.
|
||||
The step's test is written first, and a command proves it red — never the
|
||||
model's word that "it failed first". This sub-step blocks: a step whose test
|
||||
is not proven red at its start point and green at its result does not pass.
|
||||
|
||||
Do not block on test-first failures — they are expected.
|
||||
**The step's test files** are the `File:` paths under its `Test first:`. The
|
||||
plan template lists them apart from `Files:`, so everywhere below **the step's
|
||||
scope = its `Files:` + its test files**: both are built, both are committed at
|
||||
the checkpoint, and neither is a scope finding.
|
||||
|
||||
#### Sub-step C — Implement changes
|
||||
**The test command** is the step's `Verify:` command. It must be ONE
|
||||
test-runner invocation without shell operators (the allowlist of
|
||||
`lib/verification/criteria-runner.mjs`); the tools below refuse anything else
|
||||
with exit 2.
|
||||
|
||||
Read the step's `Files:` and `Changes:` fields. Implement exactly as described.
|
||||
1. **The helper proves red before it builds** (Sub-step C): it writes the test
|
||||
from `Test first:`, runs the gate
|
||||
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {test files, ONE comma-separated argument}`
|
||||
(the test must FAIL at the start point), locks the test files, and only then
|
||||
implements.
|
||||
2. **The executor proves it both ways after the helper returns:**
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {test files, ONE comma-separated argument} --json
|
||||
```
|
||||
The test must FAIL against the start point and PASS against the result
|
||||
(the working tree), and a **control run** must PASS: the start-point copy
|
||||
with the step's whole change laid on top. The start point is a temp
|
||||
`git clone --shared` checked out at `start_sha` (a real git checkout), with
|
||||
the step's test files copied in and every gitignored top-level entry of the
|
||||
working tree (`node_modules`, `.venv`, …) linked in; `--link a,b` adds
|
||||
deeper dependency paths. A start point that fails for an environment reason
|
||||
fails the control too, so it is never read as red.
|
||||
|
||||
**Rules:**
|
||||
- Follow `Changes:` exactly — do not improvise, add scope, or optimize
|
||||
- Use Edit for modifications, Write for new files
|
||||
- If `Reuses:` references existing code, read that code first for context
|
||||
- Only touch files listed in `Files:` — nothing else
|
||||
| Exit | `steps.{N}.red_proof` | Executor |
|
||||
|------|-----------------------|----------|
|
||||
| 0 | `"proven"` | continue |
|
||||
| 1, `not-red-at-base` | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again |
|
||||
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
|
||||
| 2 | `"not-run"` | never a pass and never proven. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
|
||||
|
||||
The exit code is the verdict, the output text is not. `base.tap` (when the
|
||||
runner prints TAP) says what the red was: `fail 1 of 1` on a node test file is
|
||||
usually a load error, not a named failing test — say so in the step report.
|
||||
|
||||
**The test lock (K8).** After its red gate the helper runs
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs lock --files {test files} --step {N}`;
|
||||
from then on the guard refuses Write/Edit/NotebookEdit on those files, so the
|
||||
code moves to meet the test and never the reverse. The executor reads
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs status` and records
|
||||
`steps.{N}.test_lock = "locked"` when it exits 0 and names the step's files.
|
||||
If `lib/util/test-lock.mjs` is not in this plugin, record `"not-locked"` and
|
||||
write "not locked: test-lock.mjs missing" in the step report — never silent.
|
||||
The floor guard (Sub-step D3) still reads every test change in the diff. The
|
||||
lock is released in Sub-step G, or when the run stops.
|
||||
|
||||
#### Sub-step C — Implement through one fresh helper
|
||||
|
||||
The executor does not write the step's code. **Launch** one fresh helper for
|
||||
the step with the Agent tool:
|
||||
|
||||
Launch `subagent_type: "voyage:step-implementer"` with a short `description`
|
||||
("step {N}: {title}") and a prompt that carries, as files and values, not as a
|
||||
summary: the plan path, `plan_version`, the step number, the step's text
|
||||
verbatim (every field, including `Interface in:`, `Interface out:` and
|
||||
`Review focus:` when present), `start_sha`, `report_path`, `mode: build`, the
|
||||
round label (`build`, or `fix round R`), and `voyage_root: ${CLAUDE_PLUGIN_ROOT}`.
|
||||
|
||||
**A step at `fix_rounds` 5 is never built again** (Sub-step F): skip the
|
||||
launch and run only the checks (B, D1, D2, D3, E) on whatever the operator
|
||||
changed. A step at `fix_rounds` 4 on `--resume` was re-planned: launch a fresh
|
||||
`build` helper for the revised step.
|
||||
|
||||
- **Never pass the Agent tool's `name` parameter.** A named spawn becomes an
|
||||
`in_process_teammate`: it does the work, but its result never reaches this
|
||||
orchestrator (measured: named 0/5 returned, unnamed 3/3 —
|
||||
`docs/agent-return-channel-defect.md`). The same holds for the controller
|
||||
(Sub-step E).
|
||||
- **Never pass `isolation`.** Measured 2026-10-06 (Claude Code 2.1.291): a
|
||||
worktree-isolated helper branched from the repository's default branch
|
||||
(`02f320f`), not from the executor's `HEAD` (`2c89a6b`), so it did not see
|
||||
the earlier steps. Only the operator's `worktree.baseRef: "head"` setting
|
||||
changes that, and a plugin cannot set it. The helper works in this tree,
|
||||
from the recorded `start_sha`.
|
||||
- **Wait for the helper to finish** before anything else; when it runs in the
|
||||
background, its completion notification is the signal. Never start the next
|
||||
sub-step or step while a helper runs.
|
||||
- `steps.{N}.attempts` += 1 for every helper launched for the step — this
|
||||
one, each fix round's, and the `build` helper after a re-plan. It is never
|
||||
reset, and every launch after a step's first also takes 1 from
|
||||
`iterations_remaining`, so the completion gate's reconciliation
|
||||
(`Σ (attempts − 1)`) holds across re-plans.
|
||||
|
||||
The helper's one-line answer is not evidence. After it returns:
|
||||
1. The report at `report_path` must carry a NEW section for this round
|
||||
(`## Step {N} - {round label}`) with its `### Ran`, `### Saw` and
|
||||
`### Did NOT check`; earlier rounds' sections and verdicts stay. A missing
|
||||
section is a finding for Sub-step F (stale sections do not count).
|
||||
2. Register EVERY new file so the diff and every check see it — not only the
|
||||
listed ones: for each path `git ls-files --others --exclude-standard`
|
||||
prints, run `git add --intent-to-add -- {path}` (stages nothing). A new file
|
||||
the step did not list is otherwise invisible to `git diff`, to the floor
|
||||
guard and to the controller. Voyage's own progress file and step reports
|
||||
are the one exception: never registered.
|
||||
3. A changed or new path outside the step's scope (`Files:` + test files) is a
|
||||
scope finding (Hard rule 2) for the controller. The executor does not
|
||||
revert it on its own.
|
||||
|
||||
**Rules the helper is held to** (its agent file carries them; the controller
|
||||
checks them in the diff): follow `Changes:` exactly — no improvising, no added
|
||||
scope; touch only the step's scope (`Files:` + test files); read `Reuses:`
|
||||
first; never weaken a test; never write a stub; never commit.
|
||||
|
||||
#### Sub-step D1 — Command verification
|
||||
|
||||
|
|
@ -1259,7 +1483,9 @@ status = `blocked` and apply `On failure: escalate`. Skip D2 manifest
|
|||
verification (nothing to verify for a read-only sandbox test). Commit none —
|
||||
jump straight to Phase 7 with a structured "sandbox-blocked" reason.
|
||||
|
||||
If **PASS**: proceed to Sub-step D2 (manifest verification).
|
||||
If **PASS**: proceed to Sub-step D2 (manifest verification). If **FAIL**: the
|
||||
failure output is a finding for Sub-step F; still run D2 and D3, so the
|
||||
controller and the next fix round see every failure at once.
|
||||
|
||||
#### Sub-step D2 — Manifest verification
|
||||
|
||||
|
|
@ -1273,10 +1499,10 @@ holds, regardless of Verify exit code.
|
|||
file exists in the repo. Count how many exist.
|
||||
2. **min_file_count satisfied:** count from (1) must be ≥ `min_file_count`.
|
||||
3. **Forbidden paths untouched:** for each `forbidden_paths` entry, run
|
||||
`git diff --name-only HEAD~{attempts} HEAD -- {path}` (since this step
|
||||
`git diff --name-only {start_sha} -- {path}` (since this step
|
||||
began). Any modified forbidden path = fail.
|
||||
4. **Bash syntax check:** for each entry in `bash_syntax_check` AND any
|
||||
`.sh` file that appears in `git diff --name-only HEAD~1 HEAD` (safety
|
||||
`.sh` file that appears in `git diff --name-only {start_sha}` (safety
|
||||
net for unlisted scripts), run `bash -n {path}`. Non-zero exit = fail.
|
||||
5. **must_contain patterns:** for each `{path, pattern}` pair, run
|
||||
`grep -E "{pattern}" {path}`. No match = fail.
|
||||
|
|
@ -1288,45 +1514,160 @@ Manifest verification FAIL for step {N}:
|
|||
- {which check failed with detail}
|
||||
```
|
||||
|
||||
Apply the step's `On failure:` clause (same as Sub-step D1 failure). Increment
|
||||
attempts. Manifest failures count against the retry cap equally with command
|
||||
failures.
|
||||
A manifest failure is a finding for Sub-step F, exactly like a command
|
||||
failure: it counts against the same fix rounds.
|
||||
|
||||
If all manifest checks pass: proceed to Sub-step F (checkpoint).
|
||||
Either way, proceed to Sub-step D3 (floor guard).
|
||||
|
||||
Record per-step manifest audit result in progress file:
|
||||
`steps.{N}.manifest_audit = "pass" | "fail"`,
|
||||
`steps.{N}.manifest_drift = [{check: reason, ...}]` on fail.
|
||||
|
||||
#### Sub-step E — On failure handling
|
||||
#### Sub-step D3 — Floor guard
|
||||
|
||||
If **FAIL**, read the `On failure:` clause. Apply the retry cap: **maximum 2 retries**
|
||||
(3 total attempts). Track attempts in `steps.{N}.attempts`.
|
||||
Read the step's diff for weakened tests — by a command:
|
||||
|
||||
**`On failure: revert`**
|
||||
- If attempts < 3: analyze the failure, re-implement with adjustments, re-verify.
|
||||
```
|
||||
Attempt {A}/3 failed. Retrying...
|
||||
```
|
||||
- If attempts == 3: revert this step's changes:
|
||||
```bash
|
||||
git checkout -- {files from Files: field}
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/test-integrity.mjs --base {start_sha} --json
|
||||
```
|
||||
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.
|
||||
|
||||
**`On failure: retry`**
|
||||
- If attempts < 3: use the alternative approach described in the On failure clause.
|
||||
- If attempts == 3: revert and stop. Jump to Phase 7.
|
||||
It reports a removed assert, a deleted or skipped test, a muted check
|
||||
(`|| true`, continue-on-error, an inline lint/type suppression), a stub body
|
||||
("not implemented") and a threshold loosened in the permissive direction.
|
||||
|
||||
| Exit | Meaning | Executor |
|
||||
|------|---------|----------|
|
||||
| 0 | clean | continue |
|
||||
| 1 | findings | each finding goes to the controller (Sub-step E), which rules it `justified` only by quoting the plan line that asks for the change; every other finding is a `violation` |
|
||||
| 2 | could not run | exit 2 is never green: "floor guard not run: {reason}" is a finding for Sub-step F, and the step cannot pass this round |
|
||||
|
||||
A finding is never dropped silently: the step report and the final report list
|
||||
every one with its ruling.
|
||||
|
||||
#### Sub-step E — Control
|
||||
|
||||
**One controller, two verdicts.** After the mechanical checks (B, D1, D2, D3)
|
||||
— whatever they said — launch one controller with the Agent tool:
|
||||
`subagent_type: "general-purpose"`, `model: "opus"`, no `name` (Sub-step C),
|
||||
no `isolation`. It is a general helper with the prompt below, not a second
|
||||
Voyage agent. The Agent tool sets no reasoning effort, so the controller runs
|
||||
at the session's effort: run `/trekexecute` at effort `high` or above (the
|
||||
standard). A session below `high` is a decision-log entry ("controller ran at
|
||||
effort {level}"), never silent. It reads the diff, not the helper's summary: it is never given
|
||||
the step report, and it judges what changed, not what the helper says changed.
|
||||
|
||||
The prompt (fill in every `{…}`; pass the mechanical results verbatim as JSON):
|
||||
|
||||
```text
|
||||
You are the controller for step {N} of the plan {plan path}. You judge; you
|
||||
change nothing: no Write, no Edit, no git command that writes.
|
||||
|
||||
Read the step's diff yourself: git diff {start_sha} --stat, then
|
||||
git diff {start_sha}. The diff is the evidence. You are not given the
|
||||
helper's report and must not look for it.
|
||||
|
||||
The step, verbatim:
|
||||
{every field of the step}
|
||||
|
||||
Review focus from the plan (look hardest here): {Review focus, or "none"}
|
||||
|
||||
Mechanical results: proof of red {red-proof JSON}; Verify exit {code};
|
||||
manifest {pass | fail + drift}; floor guard {test-integrity JSON}.
|
||||
|
||||
Give two verdicts:
|
||||
- "spec": does the diff do what the step's Changes, Files and Interface say -
|
||||
no more, no less? Name any changed path outside Files.
|
||||
- "quality": does the code hold - correctness, error handling, edge cases the
|
||||
step names, no stub, no weakened test, the repo's own patterns?
|
||||
Rule every floor-guard finding: "justified: <the plan line, quoted>" or
|
||||
"violation". List what you did not judge.
|
||||
|
||||
Answer with exactly one JSON block:
|
||||
{"spec": "pass|fail", "quality": "pass|fail",
|
||||
"findings": [{"verdict": "spec|quality", "file": "...", "line": 0, "problem": "...", "fix": "..."}],
|
||||
"floor_guard": [{"kind": "...", "file": "...", "ruling": "justified: ... | violation"}],
|
||||
"not_judged": ["..."]}
|
||||
```
|
||||
|
||||
Wait for the controller to finish. Parse its JSON block; an answer without a
|
||||
parseable block is a failed control (a finding for Sub-step F), never a pass.
|
||||
Append both verdicts, the findings and `not_judged` to the step report.
|
||||
|
||||
**The round passes** only when ALL hold: `red_proof` is `"proven"` (or, below
|
||||
plan_version 1.8, the tolerated `"not-run"` of Sub-step B), Verify PASS,
|
||||
manifest pass, every floor-guard finding ruled `justified` (and the guard
|
||||
ran), `"spec": "pass"` and `"quality": "pass"`. Then go to Sub-step G.
|
||||
Otherwise go to Sub-step F with every finding.
|
||||
|
||||
**Re-control after a fix** checks only what was fixed. A fresh controller gets
|
||||
the previous round's verdicts, findings and floor-guard rulings, and the
|
||||
round's own change — before the fix round the executor saves
|
||||
`git diff {start_sha}` to `{report dir}/step-{N}-round-{R}.diff`, and afterwards
|
||||
hands over `diff -u` of that file against the new `git diff {start_sha}`. It
|
||||
answers in the SAME JSON block as above:
|
||||
- per previous finding, fixed or not — a finding not fixed stays in
|
||||
`findings`; whether the round changed anything the findings did not name
|
||||
(that is a new finding);
|
||||
- `"spec"` and `"quality"` for the step as a whole: the previous verdict
|
||||
carries over for every part the round did not touch, re-judged only where
|
||||
the round changed something;
|
||||
- `floor_guard`: a finding identical (same kind, file and text) to one the
|
||||
previous round already ruled keeps that ruling; only new floor-guard
|
||||
findings are ruled.
|
||||
It does not re-judge the rest of the step.
|
||||
|
||||
#### Sub-step F — Fix rounds
|
||||
|
||||
A failed round never loops sideways. **The fix rounds are capped at maximum 5 fix rounds per step, and they escalate upward** — to a fresh helper, then to
|
||||
the plan, then to the operator. The executor does not improvise a new design,
|
||||
and there is no ladder of cheaper models: every helper and every controller
|
||||
runs `opus` (the step helper pins effort `high`; the controller follows the
|
||||
session's effort, Sub-step E).
|
||||
|
||||
| `fix_rounds` | Goes to | What happens |
|
||||
|---|---|---|
|
||||
| 1, 2, 3 | a fresh helper | Launch `voyage:step-implementer` again (no `name`, no `isolation`), `mode: fix`, with the findings (file, line, problem, fix) and the mechanical failures. It fixes only those; the tests stay locked (except in a `not-red-at-base` round, Sub-step B). Then B (both ways), D1, D2, D3 run again and a fresh controller re-controls only what was fixed (Sub-step E). `attempts` +1; `iterations_remaining` −1 |
|
||||
| 4 | the plan | **After three failed fix rounds the question is the architecture, not a fourth fix.** The step goes back to planning (below). Stop class 4 |
|
||||
| 5 | the operator | A step already sent to the plan (`fix_rounds` 4) that fails its control again after re-planning goes to the operator. The executor never builds this step again: on a later `--resume` it only runs the checks (B, D1, D2, D3, E) on whatever the operator changed. If they fail again the step stays at 5 — there is no sixth round, `fix_rounds` never exceeds 5 — and the run stops again. Stop class 4 |
|
||||
|
||||
**Back to planning** (`fix_rounds = 4`): save `git diff {start_sha}` to
|
||||
`{report dir}/step-{N}-upward.diff` and write `{report dir}/step-{N}-upward.md`:
|
||||
the step verbatim, every round's findings and verdicts, and the question for
|
||||
the planner — which assumption in the step (its Changes, its Interface, its
|
||||
test) makes it fail. Then stop: step `failed`, run `failed`, and tell the
|
||||
operator to re-plan the step (`/trekplan` revises it) and run
|
||||
`/trekexecute --resume`, which launches a fresh `build` helper for the revised
|
||||
step (`attempts` +1, `iterations_remaining` −1) and keeps `fix_rounds` at 4. A step whose proof of red cannot run on a
|
||||
≥ 1.8 plan because its `Verify:` is not one test-runner invocation goes here
|
||||
directly (Sub-step B).
|
||||
|
||||
**The plan's `On failure:` clause** decides what happens around the ladder:
|
||||
|
||||
**`On failure: revert`** and **`On failure: retry`**
|
||||
- Fix rounds 1–3 run as above; for `retry`, the clause's alternative approach
|
||||
is handed to the round-1 fix helper.
|
||||
- When the step goes up (round 4 or 5), first save the upward files, then
|
||||
revert this step's changes (nothing of the step is committed yet):
|
||||
```bash
|
||||
git checkout -- {the step's tracked files changed since start_sha}
|
||||
git rm --cached -q -- {the step's new files: registered with --intent-to-add since start_sha}
|
||||
```
|
||||
The new files stay on disk, untracked, and `step-{N}-upward.md` names them —
|
||||
nothing the step made is deleted, and none of it is left registered.
|
||||
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.5 (Phase 7
|
||||
runs only when every step passed).
|
||||
|
||||
**`On failure: skip`**
|
||||
- Mark step as skipped regardless of attempt count. Continue to next step.
|
||||
- Mark step as skipped at its first failed round, without fix rounds. Continue
|
||||
to next step.
|
||||
```
|
||||
Step {N}: SKIPPED (non-critical per plan)
|
||||
```
|
||||
Update `steps.{N}.status = "skipped"`.
|
||||
|
||||
**`On failure: escalate`**
|
||||
- Stop immediately regardless of attempt count.
|
||||
- Stop at the first failed round, without fix rounds (stop class 4: the plan's
|
||||
author declared that past this point is guesswork).
|
||||
```
|
||||
Step {N}: ESCALATED — requires human judgment
|
||||
```
|
||||
|
|
@ -1336,14 +1677,46 @@ If **FAIL**, read the `On failure:` clause. Apply the retry cap: **maximum 2 ret
|
|||
```bash
|
||||
git add {files from completed steps' Files: fields} && git commit -m "wip: trekexecute stopped at step {N} — escalation needed"
|
||||
```
|
||||
Jump to Phase 7.
|
||||
Jump to Phase 7.5 (Phase 7 runs only when every step passed).
|
||||
|
||||
#### Sub-step F — Checkpoint
|
||||
The same wip commit of completed work runs whenever a step goes up (round 4
|
||||
or 5). Release the step's test lock before stopping
|
||||
(`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock`, `test_lock` =
|
||||
`"released"`), so the operator and the planner can edit the test.
|
||||
|
||||
Run the `Checkpoint:` git commit command exactly as written in the plan.
|
||||
#### Sub-step G — Checkpoint, done line, unlock
|
||||
|
||||
If the commit fails (nothing to commit, etc.): warn but do NOT fail the step.
|
||||
The step's verification already passed — the commit is bookkeeping.
|
||||
1. **Checkpoint.** Stage the step's scope — its `Files:` and its test files,
|
||||
and only those, never `git add -A` — then run the `Checkpoint:` git commit
|
||||
command exactly as written in the plan. (The test files are listed apart
|
||||
from `Files:` in the plan; staging only `Files:` would leave the step's
|
||||
test uncommitted and charge it to the next step's diff.)
|
||||
If the commit fails (nothing to commit, etc.): warn but do NOT fail the
|
||||
step. The step's verification already passed — the commit is bookkeeping.
|
||||
2. **The done line**, at the step's own commit. A step is done only when a
|
||||
script says so, and the script writes its line only when the test command
|
||||
exits 0:
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --step {N} --cmd "{Verify command}" --report {report_path}
|
||||
```
|
||||
Exit 0 = `DONE step {N}: … exit 0 at {the checkpoint sha} …` appended to the
|
||||
step report. Confirm with
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path} --commit {hash}`
|
||||
(exit 0 = the line names this step's commit). Neither the helper nor the
|
||||
executor ever types a DONE line.
|
||||
- Exit 1 = the test went red after the checkpoint (it passed in Sub-step E
|
||||
moments before: a flaky test or an order dependency). A finding for
|
||||
Sub-step F; the fix lands as a second commit with the same message, and
|
||||
Phase 7.5's commit count reports that extra commit as drift — said, not
|
||||
hidden.
|
||||
- Exit 2 = it could not run: never green — on a ≥ 1.8 plan that is the
|
||||
plan's `Verify:` (up to the plan, as in Sub-step B); below 1.8, a step
|
||||
whose `Verify:` is not a test-runner command records "no done line
|
||||
(Verify is not a test-runner command)" in its report and rests on
|
||||
Sub-step D1's exit code.
|
||||
3. **Release the test lock**:
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock` → `test_lock` =
|
||||
`"released"` (a step that was `"not-locked"` stays so).
|
||||
|
||||
```
|
||||
Step {N}: PASS (committed: {hash})
|
||||
|
|
@ -1474,6 +1847,23 @@ now enforced on the single-session path and not only on the multi-session one.
|
|||
Continue to Phase 7.5 either way: the manifest audit is independent, and a
|
||||
verification failure must not hide a manifest drift.
|
||||
|
||||
### End to end — "run the thing" (P9 (6))
|
||||
|
||||
Tests passing is not the product working. From the runner's JSON, set
|
||||
`end_to_end`:
|
||||
|
||||
- `"verified"` — at least one criterion that **passed** says in its text that
|
||||
it runs the thing end to end (its text contains `end to end`, `end-to-end`
|
||||
or `run the thing`, any case);
|
||||
- `"not-verified"` — otherwise, including a session spec and a plan whose
|
||||
verification did not pass.
|
||||
|
||||
`"not-verified"` is never green: the report says **not verified end to end**
|
||||
in plain words, and the completion gate (Phase 7.5, clause 4) turns
|
||||
`completed` into `partial`. The fix is in the plan — a criterion that runs the
|
||||
built thing — not in the run. (This reads the existing `## Verification`
|
||||
criteria; it adds no plan field.)
|
||||
|
||||
## Phase 7.5 — Manifest audit (independent)
|
||||
|
||||
**Runs for all modes except dry-run.** This is the last-line-of-defense
|
||||
|
|
@ -1492,18 +1882,26 @@ produces an independent truth based on objective state, not self-narrative.
|
|||
2. **Filesystem check:** for each expected_path, confirm the file exists.
|
||||
|
||||
3. **Commit-count check:** count commits since session-start
|
||||
(`git rev-list --count {start_sha}..HEAD`). Expected count =
|
||||
(`git rev-list --count {session_start_sha}..HEAD`). Expected count =
|
||||
number of steps with `status=completed` (excludes Step 0 if sandbox_preflight).
|
||||
|
||||
4. **Commit-message pattern sweep:** walk `git log {start_sha}..HEAD` and
|
||||
4. **Commit-message pattern sweep:** walk `git log {session_start_sha}..HEAD` and
|
||||
confirm each commit matches one of the declared
|
||||
`commit_message_patterns` (in any order — order is advisory).
|
||||
|
||||
5. **Bash syntax sweep:** for every `.sh` file in
|
||||
`git diff --name-only {start_sha} HEAD`, run `bash -n`. Collect failures.
|
||||
`git diff --name-only {session_start_sha} HEAD`, run `bash -n`. Collect failures.
|
||||
|
||||
6. **Forbidden-path sweep:** for each `scope_forbidden` entry, check
|
||||
`git diff --name-only {start_sha} HEAD -- {path}` is empty.
|
||||
`git diff --name-only {session_start_sha} HEAD -- {path}` is empty.
|
||||
|
||||
7. **Done-line sweep:** for every step with `status=completed` that carries a
|
||||
`report_path`, run
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path} --commit {steps.N.commit}`
|
||||
(the line must name the step's own checkpoint commit).
|
||||
Exit 1 on a plan ≥ 1.8 = drift (`{check: "done-line", expected: "DONE step N", actual: "absent"}`);
|
||||
below 1.8 it is drift only when the step's report does not say why it has
|
||||
none (Sub-step G).
|
||||
|
||||
Compare the audit result against the executor's `progress.status`:
|
||||
|
||||
|
|
@ -1548,6 +1946,9 @@ Record in progress file:
|
|||
> result is OVERRIDDEN `completed → partial`, exactly as the audit-override
|
||||
> already does (`:1221`). So the counter's liveness is auditable at the gate,
|
||||
> not merely asserted in prose.
|
||||
> 4. **End to end** — for a trekplan, Phase 7's `end_to_end == "verified"`.
|
||||
> `"not-verified"` OVERRIDES `completed → partial` with the reason
|
||||
> "not verified end to end": tests that pass are not the thing running.
|
||||
>
|
||||
> Termination authority therefore rests on the deterministic audit + an
|
||||
> objective stop-signal, never on the model's narrative. Native `/goal` (when
|
||||
|
|
@ -1611,16 +2012,19 @@ Do NOT dispatch a third recovery. Report to the user.
|
|||
Every recovery/retry loop in this executor has a documented numeric bound.
|
||||
These are **three distinct axes** — not one nested counter:
|
||||
|
||||
- **Per-step retry cap** — `attempts ≤ 3` (initial + **maximum 2 retries**;
|
||||
Hard Rule 5 / Sub-step E). Bounds the retries of a *single* step.
|
||||
- **Per-step fix-round cap** — `fix_rounds ≤ 5`: maximum 5 fix rounds per step, escalating upward
|
||||
(rounds 1–3 fix with a fresh helper, 4 goes to the plan, 5 to the operator;
|
||||
Hard Rule 5 / Sub-step F). `attempts` counts the helpers a step launched.
|
||||
Bounds the rounds of a *single* step.
|
||||
- **Per-session recovery cap** — `recovery_depth < 2` (Phase 7.6 hard cap).
|
||||
Bounds how deep recovery *dispatch* nests.
|
||||
- **Global recovery/retry budget** — a new env-overridable constant
|
||||
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` (default **25**) — the aggregate ceiling
|
||||
on the **total count of recovery + retry iterations across the whole
|
||||
execution**. This is the case the two caps above do NOT bound: many steps each
|
||||
retrying up to 3× can sum unbounded. It primarily backstops the **retry
|
||||
aggregate** (recovery dispatch is already hard-capped at 2). Surfaced and
|
||||
using up to 3 fix rounds can sum unbounded. It primarily backstops the **fix-round
|
||||
aggregate** (recovery dispatch is already hard-capped at 2). Reaching 0 stops
|
||||
the run (stop class 4: a plan that needs that many fixes is guesswork). Surfaced and
|
||||
enforced via the `iterations_remaining` counter (Phase 3 schema; decremented
|
||||
on every recovery/retry iteration; reconciled at the Phase 7.5 completion
|
||||
gate).
|
||||
|
|
@ -1787,15 +2191,37 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
|||
**Type:** {plan | session-spec}
|
||||
**Mode:** {execute | resume | step N}
|
||||
**Result:** {COMPLETED | FAILED at step N | STOPPED (escalation) | PARTIAL (N/total passed)}
|
||||
**End to end:** {verified | not verified end to end}
|
||||
|
||||
### Step Results
|
||||
|
||||
| Step | Description | Result | Attempts | Commit | Manifest |
|
||||
|------|-------------|--------|----------|--------|----------|
|
||||
| 0 | Sandbox pre-flight | PASS | 1 | — | n/a |
|
||||
| 1 | {desc} | PASS | 1 | abc1234 | pass |
|
||||
| 2 | {desc} | FAIL | 3 | — | — |
|
||||
| 3 | {desc} | — | 0 | — | — |
|
||||
| Step | Description | Result | Red proof | Fix rounds | Test lock | Done line | Attempts | Commit | Manifest |
|
||||
|------|-------------|--------|-----------|------------|-----------|-----------|----------|--------|----------|
|
||||
| 0 | Sandbox pre-flight | PASS | n/a | 0 | n/a | n/a | 1 | — | n/a |
|
||||
| 1 | {desc} | PASS | proven | 0 | released | yes | 1 | abc1234 | pass |
|
||||
| 2 | {desc} | FAIL | rejected | 4 (sent to the plan) | released | no | 4 | — | — |
|
||||
| 3 | {desc} | — | — | 0 | — | — | 0 | — | — |
|
||||
|
||||
Steps without proof of red: {N} ({step numbers}; below plan_version 1.8 only)
|
||||
|
||||
### Controller verdicts
|
||||
|
||||
Per step and round: spec {pass | fail}, quality {pass | fail}, findings (file:line — problem),
|
||||
floor-guard rulings, and what the controller did not judge. Report paths: {report_path per step}.
|
||||
|
||||
### Stops
|
||||
|
||||
{none — the run ended without a stop | the stop that ended the run, with its class from the
|
||||
closed stop list (Phase 5.5): 1 cannot be undone · 2 security-sensitive · 3 outside the working
|
||||
copy · 4 plan so broken that everything after it is guesswork, and the file or command that shows it}
|
||||
|
||||
### Decisions taken
|
||||
|
||||
| Step | Decision | Cost if wrong |
|
||||
|------|----------|---------------|
|
||||
| {N} | {what the executor or a helper decided that the plan did not dictate} | {what must be redone, by whom, how big} |
|
||||
|
||||
{none, when the plan dictated every call}
|
||||
|
||||
### Plan Verification (Phase 7, trekplans only)
|
||||
|
||||
|
|
@ -1819,8 +2245,12 @@ Phase 2.3 (validate exit) and Phase 5 (dry-run) intentionally do not write
|
|||
- Not reached: {N}
|
||||
- Blocked (sandbox): {N}
|
||||
|
||||
{if all passed + exit condition passed + plan verification passed}:
|
||||
All steps completed. Exit condition: PASS. Plan verification: PASS.
|
||||
{if all passed + exit condition passed + plan verification passed + end to end verified}:
|
||||
All steps completed. Exit condition: PASS. Plan verification: PASS. End to end: verified.
|
||||
|
||||
{if all passed but end_to_end = not-verified}:
|
||||
All steps passed their checks, but the plan has no passing run-the-thing criterion:
|
||||
not verified end to end. Result: PARTIAL.
|
||||
|
||||
{if failed/stopped}:
|
||||
### Failure Details
|
||||
|
|
@ -1842,10 +2272,11 @@ To continue (multi-session waves): re-run the same /trekexecute command (not --r
|
|||
- `completed` — all steps passed AND Phase 7.5 manifest audit passed AND
|
||||
(for trekplans) Phase 7 plan verification passed
|
||||
- `partial` — steps passed per executor but Phase 7.5 found drift, OR
|
||||
Phase 7.6 recovery incomplete
|
||||
Phase 7.6 recovery incomplete, OR the run is not verified end to end
|
||||
(Phase 7, gate clause 4)
|
||||
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
|
||||
- `failed` — a step failed and On failure was revert/retry (retry cap hit)
|
||||
- `stopped` — On failure: escalate triggered
|
||||
- `failed` — a step failed and went up the fix ladder (On failure revert/retry: round 4 to the plan, or 5 to the operator)
|
||||
- `stopped` — On failure: escalate triggered, or another stop on the closed stop list (Phase 5.5)
|
||||
|
||||
**JSON summary block** (always at the end, machine-parseable):
|
||||
|
||||
|
|
@ -1871,7 +2302,11 @@ To continue (multi-session waves): re-run the same /trekexecute command (not --r
|
|||
"recovery_depth": 0,
|
||||
"iterations_remaining": 25,
|
||||
"legacy_plan": false,
|
||||
"progress_file": "{path}"
|
||||
"progress_file": "{path}",
|
||||
"end_to_end": "{verified | not-verified | n/a}",
|
||||
"steps_without_red_proof": 0,
|
||||
"stop_class": "{none | 1 | 2 | 3 | 4}",
|
||||
"decisions": 0
|
||||
}
|
||||
}
|
||||
```
|
||||
|
|
@ -1968,9 +2403,10 @@ strategy (gates auto-escalation, parallel-wave choice — see High-effort
|
|||
behavior below). The resolver does NOT control the orchestrator's own
|
||||
model — that is fixed at invocation time (command frontmatter omits
|
||||
`model:`, so it follows the session model) and cannot be switched mid-turn.
|
||||
`/trekexecute` spawns no sub-agent swarm (Hard Rule 10), so
|
||||
`phase_signal_result.model` has no spawn site here; it is returned for
|
||||
cross-command uniformity and stats.
|
||||
`/trekexecute` spawns no swarm: one step helper at a time and one controller
|
||||
(Hard Rule 10), and both run `opus` — the step agent's pin and the
|
||||
controller's `model: "opus"` — so `phase_signal_result.model` still has no
|
||||
spawn site here; it is returned for cross-command uniformity and stats.
|
||||
|
||||
For `/trekexecute` specifically: `effort == 'low'` activates `--gates open`
|
||||
+ sequential-only execution (no worktree-isolated parallel waves — runs
|
||||
|
|
@ -2022,7 +2458,9 @@ code-path.
|
|||
|
||||
1. **No AskUserQuestion for execution decisions.** All execution decisions come
|
||||
from the plan's On failure clauses. If the plan says escalate, stop and
|
||||
report — never ask.
|
||||
report — never ask. What the plan leaves open, the executor decides and
|
||||
writes into the decision log with its `cost_if_wrong`; it stops only on the
|
||||
closed stop list (Phase 5.5).
|
||||
|
||||
2. **No scope creep.** Only touch files listed in the step's `Files:` field.
|
||||
If a file outside the list seems to need changing, record it as a finding
|
||||
|
|
@ -2034,7 +2472,10 @@ code-path.
|
|||
4. **Fresh verification.** Re-run the Verify command from scratch every time.
|
||||
Never trust cached or prior results.
|
||||
|
||||
5. **Retry cap = 3 attempts.** Initial + 2 retries, then stop. Never loop forever.
|
||||
5. **Fix rounds go up — maximum 5 fix rounds per step, escalating upward.**
|
||||
Rounds 1–3 are fixes by a fresh helper, round 4 sends the step back to
|
||||
planning, round 5 to the operator (Sub-step F). Never a fourth guess at the
|
||||
same level, never a cheaper model, never a loop.
|
||||
|
||||
6. **Never corrupt completed work.** Only revert files from the failing step.
|
||||
Never touch files from earlier passed steps.
|
||||
|
|
@ -2047,11 +2488,15 @@ code-path.
|
|||
|
||||
9. **Progress file is ground truth.** Resume uses the progress file, not git log.
|
||||
|
||||
10. **No sub-agents.** The executor reads and implements directly.
|
||||
No Agent tool, no TeamCreate, no delegation. Enforced (not just
|
||||
documented) via `disallowed-tools: Agent, TeamCreate` in the frontmatter,
|
||||
which removes both tools from the pool — `allowed-tools` alone does not,
|
||||
since omission leaves a tool callable (CC 2.1.152).
|
||||
10. **One fresh helper per plan step, one controller, nothing more.** The
|
||||
executor writes no product code: each step is built by a fresh
|
||||
`voyage:step-implementer` (Sub-step C) and judged by one controller, a
|
||||
general-purpose helper that reads the diff (Sub-step E).
|
||||
Helpers never start helpers: the step agent has no Agent tool, and the
|
||||
controller is told it changes nothing. Never the Agent tool's `name` (a named spawn's
|
||||
result never returns), never `isolation` (measured: wrong base). No
|
||||
TeamCreate — kept out via `disallowed-tools: TeamCreate`, since omission
|
||||
from `allowed-tools` alone leaves a tool callable (CC 2.1.152).
|
||||
|
||||
11. **Worktree isolation is mandatory for multi-session execution.** Every
|
||||
session order names its own git worktree. Never run two or more sessions
|
||||
|
|
@ -2123,14 +2568,49 @@ code-path.
|
|||
failure (worktree removed, branch deleted, work lost) into a recoverable
|
||||
one (push succeeded, branch preserved on remote). Source: research/02 R3.
|
||||
|
||||
20. **Three distinct iteration axes.** Per-step `attempts` (≤3), per-session
|
||||
`recovery_depth` (<2), and the global `TREKEXECUTE_MAX_RECOVERY_ITERATIONS`
|
||||
budget (default 25 — aggregate recovery+retry backstop) are independent
|
||||
bounds, not one nested counter. Every recovery/retry loop is bounded by at
|
||||
least one of them. See "Iteration caps + global recovery/retry budget".
|
||||
20. **Three distinct iteration axes.** Per-step `fix_rounds` (≤5, upward),
|
||||
per-session `recovery_depth` (<2), and the global
|
||||
`TREKEXECUTE_MAX_RECOVERY_ITERATIONS` budget (default 25 — aggregate
|
||||
fix-round + recovery backstop) are independent bounds, not one nested
|
||||
counter. Every recovery/retry loop is bounded by at least one of them. See
|
||||
"Iteration caps + global recovery/retry budget".
|
||||
|
||||
21. **Never start Claude Code.** No `claude` CLI invocation, no Agent SDK, no
|
||||
background process that runs a session — not for waves, not for recovery,
|
||||
not for a test. Work that needs another session becomes a session order
|
||||
plus a starter line the user gives a session they open themselves
|
||||
(Phase 2.6, Phase 7.6).
|
||||
|
||||
22. **Proof, not word.** Red before green, an unweakened test and "done" are
|
||||
each a command's exit code — `red-proof.mjs`, `test-integrity.mjs`,
|
||||
`done-line.mjs` — never the helper's or the executor's statement. Exit 2
|
||||
from any of them ("could not run") is never green.
|
||||
|
||||
## Credits
|
||||
|
||||
Every borrowing in this executor is named here, never silent. All sources are
|
||||
MIT-licensed open source except Kiro (a closed product: only its published
|
||||
documentation was read) and Cherny (public talks and posts). Read 2026-10-06;
|
||||
pins and quotes in the local practice analysis.
|
||||
|
||||
- **Superpowers** — adapted from obra/superpowers@8ca22db (MIT, © 2025 Jesse
|
||||
Vincent): `subagent-driven-development` — a fresh helper per task, hand-over
|
||||
in files, a review package from a recorded start point, a reviewer between
|
||||
tasks, the stop classes and the decision log of an unattended run (P8, P4);
|
||||
`executing-plans/scripts/task-done` — the completion line appended only when
|
||||
the test command exits 0, here `done-line.mjs` (P9 (4));
|
||||
`systematic-debugging` — after three failed fixes, question the
|
||||
architecture (P9 (7)). Not borrowed: its ladder of cheaper models.
|
||||
- **ai-native-sdlc** — bashebr/ai-native-sdlc@11fb8d7 (MIT): the helper's
|
||||
report contract, ran / saw / did NOT check (P8).
|
||||
- **Kiro** (documentation only): the task runner takes the diff, not the
|
||||
agent's account of it, as the reviewer's input (P8). Its background daemon is
|
||||
not borrowed; only the step loop.
|
||||
- **GitHub Spec Kit** (MIT), github/spec-kit@2e941fe: test-first as a rule,
|
||||
mechanized here as `red-proof.mjs` (P9 (1)).
|
||||
- **agent-skills** — addyosmani/agent-skills@9d0c60d (MIT): the floor guard of
|
||||
constraint-driven development, as `test-integrity.mjs` (P9 (2)).
|
||||
- **Cherny** (public primary sources): "run the thing" — a plan is verified
|
||||
end to end only by a criterion that runs what was built (P9 (6)).
|
||||
- **Claude Code** itself: subagents for the helper and the controller; the
|
||||
`PreToolUse` guard behind the test lock (K8).
|
||||
|
|
|
|||
|
|
@ -299,7 +299,7 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
|
||||
## Handover 5 — `progress.json` (resume contract)
|
||||
|
||||
**Producer:** `/trekexecute` per-step (after Verify + Manifest audit + Checkpoint).
|
||||
**Producer:** `/trekexecute` per-step (start point recorded before the step; then proof of red + Verify + Manifest audit + floor guard + control + done line + Checkpoint).
|
||||
|
||||
**Consumer:** `/trekexecute --resume` (re-entry) + `pre-compact-flush` hook (drift detection before context compaction).
|
||||
|
||||
|
|
@ -325,6 +325,8 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
| `session_start_sha` | string | optional | git sha at execute start |
|
||||
| `session_end_sha` | string | optional | git sha at execute end |
|
||||
| `steps` | object | yes | Map of step number → step record |
|
||||
| `end_to_end` | string | optional | `verified \| not-verified` — a passing run-the-thing criterion in the plan's `## Verification`; `not-verified` turns `completed` into `partial` (wave 1, P9 (6)) |
|
||||
| `decisions` | array | optional | `[{step, decision, cost_if_wrong}]` — every call the executor made that the plan did not dictate; both strings non-empty (wave 1, P4) |
|
||||
|
||||
**Per-step record:**
|
||||
|
||||
|
|
@ -337,6 +339,15 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
| `commit` | string \| null | git sha after Checkpoint |
|
||||
| `manifest_audit` | string | `pass \| fail \| pass-with-note \| n/a` |
|
||||
| `note` | string | optional human-readable annotation |
|
||||
| `start_sha` | string \| null | optional (wave 1) — git sha the step started from; every check of the step reads `git diff {start_sha}` |
|
||||
| `report_path` | string \| null | optional (wave 1) — the step's report file: the helper's ran / saw / did-NOT-check rounds and the script-written `DONE step N` line |
|
||||
| `red_proof` | string \| null | optional (wave 1) — `proven \| rejected \| not-run`, the exit of `lib/verification/red-proof.mjs` |
|
||||
| `test_lock` | string \| null | optional (wave 1) — `locked \| released \| not-locked` (K8, `lib/util/test-lock.mjs`) |
|
||||
| `fix_rounds` | number | optional (wave 1) — 0..5: 0 = passed its first control, 1–3 = fix rounds, 4 = sent up to the plan, 5 = sent up to the operator. 6 is invalid |
|
||||
|
||||
The wave-1 fields are **additive-optional on read**: a `progress.json` written
|
||||
before them validates and resumes unchanged (no `schema_version` bump). `null`
|
||||
means "not set yet". When present, `progress-validator.mjs` shape-checks them.
|
||||
|
||||
**Validation strategy:** `progress-validator.mjs` runs at:
|
||||
1. `/trekexecute --validate` (alongside plan-validator)
|
||||
|
|
@ -354,6 +365,9 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
- `PROGRESS_STEP_RANGE` → `current_step` outside `[0, total_steps]`; resume halts
|
||||
- `PROGRESS_ALREADY_DONE` → `status === completed`; nothing to resume
|
||||
- `PROGRESS_STEP_COUNT_MISMATCH` → warning; not a blocker
|
||||
- `PROGRESS_FIX_ROUNDS_INVALID` → `fix_rounds` not an integer 0..5; resume halts
|
||||
- `PROGRESS_START_SHA_INVALID` / `PROGRESS_REPORT_PATH_INVALID` / `PROGRESS_RED_PROOF_INVALID` / `PROGRESS_TEST_LOCK_INVALID` → a wave-1 step field of the wrong shape; resume halts
|
||||
- `PROGRESS_DECISION_INVALID` → `decisions` is not an array of `{decision, cost_if_wrong}`; resume halts
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
|
|
@ -73,10 +73,11 @@ native `effort:`.
|
|||
**Native `effort:` on agents.** Voyage sets the reasoning axis statically on
|
||||
selected agents, overriding the session level:
|
||||
|
||||
- **Eight agents pin `high`:** the retrieval agents `task-finder`,
|
||||
`git-historian`, `dependency-tracer`, `architecture-mapper`, and the
|
||||
- **Nine agents pin `high`:** the retrieval agents `task-finder`,
|
||||
`git-historian`, `dependency-tracer`, `architecture-mapper`, the
|
||||
adversarial-reasoning agents `plan-critic`, `risk-assessor`,
|
||||
`contrarian-researcher`, `review-coordinator`. No agent pins below `high`
|
||||
`contrarian-researcher`, `review-coordinator`, and `/trekexecute`'s step
|
||||
helper `step-implementer`. No agent pins below `high`
|
||||
(operator rule, 2026-10-05: standard is Opus 5.5 at effort High).
|
||||
- **All other agents:** unset → inherits the session's effort level (a session
|
||||
that nothing sets runs at its model's default — `medium` on Opus 5.5).
|
||||
|
|
|
|||
|
|
@ -6,6 +6,9 @@
|
|||
// PROGRESS_REQUIRED_TOP, so legacy progress.json without it still validates. The field's
|
||||
// liveness (never-decremented) is enforced deterministically by the Phase 7.5 completion-gate
|
||||
// cross-check in trekexecute, not by this per-validation shape check.
|
||||
// The same holds for the wave-1 step-loop fields (per step: start_sha, report_path,
|
||||
// red_proof, test_lock, fix_rounds 0..5; per run: decisions): shape-checked when
|
||||
// present, never required.
|
||||
|
||||
import { readFileSync, existsSync } from 'node:fs';
|
||||
import { issue, fail } from '../util/result.mjs';
|
||||
|
|
@ -17,6 +20,45 @@ export const PROGRESS_VALID_STATUSES = ['pending', 'in_progress', 'completed', '
|
|||
// know (lib/util/session-orders.mjs ENDED_BAD).
|
||||
export const PROGRESS_STEP_STATUSES = ['completed', 'in_progress', 'failed', 'pending', 'deferred', 'skipped', 'blocked'];
|
||||
|
||||
// Wave 1 (P8/P9) per-step fields, all optional on read so an older progress.json
|
||||
// still resumes; `null` means "not set yet". fix_rounds is the step's place on
|
||||
// the upward ladder: 0 = passed its first control, 1-3 = fix rounds, 4 = sent up
|
||||
// to the plan, 5 = sent up to the operator. There is no sixth round.
|
||||
export const PROGRESS_MAX_FIX_ROUNDS = 5;
|
||||
export const PROGRESS_RED_PROOF_VALUES = ['proven', 'rejected', 'not-run'];
|
||||
export const PROGRESS_TEST_LOCK_VALUES = ['locked', 'released', 'not-locked'];
|
||||
|
||||
function checkStepFields(k, s, errors) {
|
||||
const set = (v) => v !== undefined && v !== null;
|
||||
if (s.fix_rounds !== undefined &&
|
||||
!(Number.isInteger(s.fix_rounds) && s.fix_rounds >= 0 && s.fix_rounds <= PROGRESS_MAX_FIX_ROUNDS)) {
|
||||
errors.push(issue('PROGRESS_FIX_ROUNDS_INVALID',
|
||||
`steps["${k}"].fix_rounds=${JSON.stringify(s.fix_rounds)} must be an integer 0..${PROGRESS_MAX_FIX_ROUNDS}`));
|
||||
}
|
||||
if (set(s.start_sha) && !(typeof s.start_sha === 'string' && /^[0-9a-f]{7,64}$/.test(s.start_sha))) {
|
||||
errors.push(issue('PROGRESS_START_SHA_INVALID', `steps["${k}"].start_sha must be a git sha, got ${JSON.stringify(s.start_sha)}`));
|
||||
}
|
||||
if (set(s.report_path) && !(typeof s.report_path === 'string' && s.report_path.trim() !== '')) {
|
||||
errors.push(issue('PROGRESS_REPORT_PATH_INVALID', `steps["${k}"].report_path must be a non-empty path`));
|
||||
}
|
||||
if (set(s.red_proof) && !PROGRESS_RED_PROOF_VALUES.includes(s.red_proof)) {
|
||||
errors.push(issue('PROGRESS_RED_PROOF_INVALID', `steps["${k}"].red_proof "${s.red_proof}" not in [${PROGRESS_RED_PROOF_VALUES.join(', ')}]`));
|
||||
}
|
||||
if (set(s.test_lock) && !PROGRESS_TEST_LOCK_VALUES.includes(s.test_lock)) {
|
||||
errors.push(issue('PROGRESS_TEST_LOCK_INVALID', `steps["${k}"].test_lock "${s.test_lock}" not in [${PROGRESS_TEST_LOCK_VALUES.join(', ')}]`));
|
||||
}
|
||||
}
|
||||
|
||||
// The run-level decision log (P4): every decision the executor took on its own,
|
||||
// with what it costs if it is wrong. Optional; when present, every entry carries both.
|
||||
function checkDecisions(decisions, errors) {
|
||||
if (decisions === undefined) return;
|
||||
const ok = Array.isArray(decisions) && decisions.every((d) => d && typeof d === 'object' &&
|
||||
typeof d.decision === 'string' && d.decision.trim() !== '' &&
|
||||
typeof d.cost_if_wrong === 'string' && d.cost_if_wrong.trim() !== '');
|
||||
if (!ok) errors.push(issue('PROGRESS_DECISION_INVALID', 'decisions must be an array of {decision, cost_if_wrong} with both non-empty'));
|
||||
}
|
||||
|
||||
export function validateProgressContent(jsonText, opts = {}) {
|
||||
let parsed;
|
||||
try { parsed = JSON.parse(jsonText); }
|
||||
|
|
@ -82,9 +124,12 @@ export function validateProgressObject(parsed, opts = {}) {
|
|||
if (s.status !== undefined && !PROGRESS_STEP_STATUSES.includes(s.status)) {
|
||||
warnings.push(issue('PROGRESS_STEP_BAD_STATUS', `steps["${k}"].status "${s.status}" unrecognized`));
|
||||
}
|
||||
checkStepFields(k, s, errors);
|
||||
}
|
||||
}
|
||||
|
||||
checkDecisions(parsed.decisions, errors);
|
||||
|
||||
return { valid: errors.length === 0, errors, warnings, parsed };
|
||||
}
|
||||
|
||||
|
|
|
|||
106
lib/verification/done-line.mjs
Normal file
106
lib/verification/done-line.mjs
Normal file
|
|
@ -0,0 +1,106 @@
|
|||
// lib/verification/done-line.mjs
|
||||
// The done line (P9 (4)). A step is "done" only when this script has written
|
||||
// its line, and it writes the line only when the test command exits 0:
|
||||
//
|
||||
// DONE step <N>: `<command>` exit 0 at <HEAD sha> <ISO time>
|
||||
//
|
||||
// appended to the step's report file. Exit 0 = written, 1 = the command failed
|
||||
// (nothing written), 2 = could not run (nothing written; never green): no
|
||||
// --step, a command outside the test-runner allowlist, a command that cannot
|
||||
// start, a working tree that is not a git repo.
|
||||
//
|
||||
// `--check --step <N> --report <file> [--commit <sha>]`: exit 0 when the report
|
||||
// carries a well-formed line for step N (at exactly that commit, with
|
||||
// --commit), 1 when it does not. /trekexecute writes the line right after the
|
||||
// step's checkpoint and checks it with --commit, so the line is bound to the
|
||||
// commit the test passed on.
|
||||
//
|
||||
// Adapted from obra/superpowers@8ca22db (MIT, (c) 2025 Jesse Vincent),
|
||||
// skills/executing-plans/scripts/task-done: the completion line is appended
|
||||
// only when the test command exits 0.
|
||||
|
||||
import { appendFileSync, existsSync, readFileSync, realpathSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { commandEnv, DEFAULT_TIMEOUT_MS, repoRoot } from './red-proof.mjs';
|
||||
|
||||
export function doneLineRe(step, commit) {
|
||||
const sha = commit ? String(commit).replace(/[^0-9a-f]/g, '') : '[0-9a-f]{40}';
|
||||
return new RegExp(`^DONE step ${step}: \`.+\` exit 0 at ${sha} \\d{4}-\\d{2}-\\d{2}T\\S+$`, 'm');
|
||||
}
|
||||
|
||||
/**
|
||||
* With `commit` (the step's checkpoint sha, full 40 hex), the line must name
|
||||
* exactly that commit. A regex cannot tell a typed line from a written one; the
|
||||
* commit binds the line to the state the test ran on (/trekexecute writes the
|
||||
* line right after the checkpoint, so HEAD is the step's own commit).
|
||||
*/
|
||||
export function hasDoneLine(reportPath, step, commit) {
|
||||
if (!existsSync(reportPath)) return false;
|
||||
return doneLineRe(step, commit).test(readFileSync(reportPath, 'utf8'));
|
||||
}
|
||||
|
||||
/** @returns {{status: 'written'|'failed'|'not-run', reason: string, line?: string, exit?: number|null}} */
|
||||
export function writeDoneLine({ cwd, step, cmd, report, timeoutMs = DEFAULT_TIMEOUT_MS }) {
|
||||
if (!/^\d+$/.test(String(step ?? ''))) return { status: 'not-run', reason: '--step must be a step number' };
|
||||
if (!report) return { status: 'not-run', reason: '--report is required' };
|
||||
// The plan's Verify command is relative to the repo root, as in red-proof.mjs.
|
||||
const dir = repoRoot(resolve(cwd ?? process.cwd()));
|
||||
if (!dir) return { status: 'not-run', reason: `${cwd ?? process.cwd()} is not inside a git repository` };
|
||||
const allowed = allowedCommand(cmd, { cwd: dir });
|
||||
if (!allowed.allowed) return { status: 'not-run', reason: `command not run: ${allowed.reason}` };
|
||||
const screen = screenCommand(cmd);
|
||||
if (!screen.allowed) return { status: 'not-run', reason: `command blocked: ${screen.rule}` };
|
||||
const head = spawnSync('git', ['rev-parse', '--verify', '--quiet', 'HEAD'], { cwd: dir, encoding: 'utf8' });
|
||||
if (head.status !== 0) return { status: 'not-run', reason: `no git HEAD in ${dir}` };
|
||||
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd: dir, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
if (r.error || r.status === null || r.status === 126 || r.status === 127) {
|
||||
return { status: 'not-run', reason: `the command did not run to an exit code (${r.error?.message ?? r.signal ?? `exit ${r.status}`})`, exit: r.status };
|
||||
}
|
||||
if (r.status !== 0) return { status: 'failed', reason: `the test command exited ${r.status}`, exit: r.status };
|
||||
const line = `DONE step ${step}: \`${cmd}\` exit 0 at ${head.stdout.trim()} ${new Date().toISOString()}`;
|
||||
appendFileSync(report, `${line}\n`);
|
||||
return { status: 'written', reason: 'the test command exited 0', line, exit: 0 };
|
||||
}
|
||||
|
||||
const USAGE =
|
||||
'usage: done-line.mjs --step <N> --cmd "<test command>" --report <file> [--cwd <dir>] [--json]\n' +
|
||||
' done-line.mjs --check --step <N> --report <file> [--commit <sha>]';
|
||||
|
||||
export function main(argv) {
|
||||
const o = {};
|
||||
let check = false;
|
||||
let json = false;
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--step' && argv[i + 1]) o.step = argv[++i];
|
||||
else if (a === '--cmd' && argv[i + 1]) o.cmd = argv[++i];
|
||||
else if (a === '--report' && argv[i + 1]) o.report = resolve(argv[++i]);
|
||||
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||
else if (a === '--check') check = true;
|
||||
else if (a === '--commit' && argv[i + 1]) o.commit = argv[++i];
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`done-line: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if (check) {
|
||||
if (!/^\d+$/.test(String(o.step ?? '')) || !o.report) { process.stderr.write(`done-line: --check needs --step and --report\n${USAGE}\n`); return 2; }
|
||||
const ok = hasDoneLine(o.report, o.step, o.commit);
|
||||
process.stdout.write(json ? `${JSON.stringify({ step: Number(o.step), done: ok })}\n` : `step ${o.step}: ${ok ? 'DONE line present' : 'no DONE line - not done'}\n`);
|
||||
return ok ? 0 : 1;
|
||||
}
|
||||
let r;
|
||||
try { r = writeDoneLine(o); }
|
||||
catch (e) { r = { status: 'not-run', reason: `done-line crashed: ${e.message}` }; }
|
||||
process.stdout.write(json ? `${JSON.stringify(r)}\n` : `done line: ${r.status.toUpperCase()} - ${r.line ?? r.reason}\n`);
|
||||
return r.status === 'written' ? 0 : r.status === 'failed' ? 1 : 2;
|
||||
}
|
||||
|
||||
function isMain() {
|
||||
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||
catch { return false; }
|
||||
}
|
||||
|
||||
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||
251
lib/verification/red-proof.mjs
Normal file
251
lib/verification/red-proof.mjs
Normal file
|
|
@ -0,0 +1,251 @@
|
|||
// lib/verification/red-proof.mjs
|
||||
// Proof of red (P9 (1)). A step's test is run by a command:
|
||||
//
|
||||
// 1. against the START POINT (the step's recorded start_sha, with the step's
|
||||
// test files copied in): it must FAIL - otherwise the test proves nothing
|
||||
// about the change;
|
||||
// 2. against the RESULT (the working tree as the step left it): it must PASS;
|
||||
// 3. CONTROL: the start-point copy with the step's whole change laid on top
|
||||
// must PASS too. A copy that cannot go green even with the change failed
|
||||
// step 1 for an environment reason (a missing dependency, a path, a tool),
|
||||
// not because the change was missing - that is "not-run", never proven.
|
||||
//
|
||||
// `--red-only` runs step 1 alone: the gate the step helper passes BEFORE it
|
||||
// builds (and locks the test). The executor's own check afterwards is the full
|
||||
// run, and only the full run is authoritative.
|
||||
//
|
||||
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),
|
||||
// 2 = could not run (`not-run`: no test files, an unknown start point, a command
|
||||
// outside the test-runner allowlist, a command that cannot start, a control run
|
||||
// that stays red). 2 is never a pass and never "proven".
|
||||
//
|
||||
// The start point is a temp `git clone --shared` of the repo checked out at
|
||||
// start_sha - a real git checkout, so a git-reading command behaves as it does
|
||||
// in the repo, and nothing in the working tree or the repo's worktree list is
|
||||
// touched. Every gitignored top-level entry of the working tree (node_modules,
|
||||
// .venv, a build cache) is symlinked in, so the copy has the same dependencies;
|
||||
// `--link a,b` adds deeper ones. Measured 2026-10-06 (review): with a bare
|
||||
// `git archive` copy, a test that needs node_modules "proved red" with no change.
|
||||
// Known limit, failing safe: a dependency that points back INTO the working
|
||||
// tree (an editable `pip install -e .`, a workspace symlink) makes the start
|
||||
// point see the new code, and a correct step reads `not-red-at-base`.
|
||||
//
|
||||
// Everything runs at the root of --cwd's repository: plan paths and commands
|
||||
// are root-relative, and `git archive`/`clone` paths from a subdirectory are
|
||||
// not (measured 2026-10-06).
|
||||
//
|
||||
// The command must be ONE test-runner invocation without shell operators
|
||||
// (criteria-runner.mjs `allowedCommand`), and it is screened through the
|
||||
// plugin's own PreToolUse denylist (`screenCommand`): a command spawned from
|
||||
// node never passes through the Bash tool, so the hook cannot fire by itself.
|
||||
//
|
||||
// When the output is TAP (`--test-reporter=tap`, pytest-tap, ...), the counts
|
||||
// `# tests N` / `# fail N` are reported per run. They show WHAT the red was:
|
||||
// "fail 1 of 1" on a node test file is usually a load error (an import the
|
||||
// start point lacks), not named failing tests - see
|
||||
// tests/fixtures/red-first/02243c6-always-allow-shim.mjs, re-measured through
|
||||
// this tool in tests/lib/red-proof.test.mjs.
|
||||
//
|
||||
// Borrowed: GitHub Spec Kit (MIT, github/spec-kit@2e941fe) - its test-first
|
||||
// rule is an instruction there; here a command proves it.
|
||||
|
||||
import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
|
||||
export const DEFAULT_TIMEOUT_MS = 900_000;
|
||||
const TAIL = 2000;
|
||||
|
||||
function tap(output) {
|
||||
const tests = String(output).match(/^# tests (\d+)$/m);
|
||||
const fail = String(output).match(/^# fail (\d+)$/m);
|
||||
return tests && fail ? { tests: Number(tests[1]), fail: Number(fail[1]) } : null;
|
||||
}
|
||||
|
||||
// The command's exit code must be its own. Under `node --test` a child inherits
|
||||
// NODE_TEST_CONTEXT, and a nested `node --test` then reports to that parent and
|
||||
// exits 0 whatever its tests did (measured 2026-10-06: a red start point read as
|
||||
// green). So the variable never reaches the command.
|
||||
export function commandEnv(env = process.env) {
|
||||
const { NODE_TEST_CONTEXT, ...rest } = env;
|
||||
return rest;
|
||||
}
|
||||
|
||||
export function repoRoot(dir) {
|
||||
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: dir, encoding: 'utf8' });
|
||||
return top.status === 0 ? top.stdout.trim() : null;
|
||||
}
|
||||
|
||||
function run(cmd, cwd, timeoutMs) {
|
||||
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
const output = `${r.stdout ?? ''}${r.stderr ?? ''}`;
|
||||
return { exit: r.status, signal: r.signal ?? null, tap: tap(output), tail: output.slice(-TAIL), spawnError: r.error ? r.error.message : null };
|
||||
}
|
||||
|
||||
const notRun = (reason, extra = {}) => ({ status: 'not-run', reason, ...extra });
|
||||
|
||||
// A shell that could not find or execute the command (126/127), a signal (the
|
||||
// timeout), or a spawn error is not a test verdict at all.
|
||||
function unrunnable(r) {
|
||||
if (r.spawnError) return `the command could not start: ${r.spawnError}`;
|
||||
if (r.exit === null) return `the command was killed (${r.signal ?? 'timeout'})`;
|
||||
if (r.exit === 126 || r.exit === 127) return `the shell could not run the command (exit ${r.exit})`;
|
||||
return null;
|
||||
}
|
||||
|
||||
const git = (cwd, args) => spawnSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
|
||||
const lines = (s) => String(s ?? '').split('\n').filter(Boolean);
|
||||
|
||||
function safeRel(p) {
|
||||
return p && !isAbsolute(p) && !p.split('/').includes('..');
|
||||
}
|
||||
|
||||
function copyIn(from, to, rel) {
|
||||
mkdirSync(dirname(join(to, rel)), { recursive: true });
|
||||
rmSync(join(to, rel), { recursive: true, force: true });
|
||||
cpSync(join(from, rel), join(to, rel), { recursive: true, verbatimSymlinks: true });
|
||||
}
|
||||
|
||||
/** Symlink every gitignored top-level entry of the working tree, plus `extra`, into `tree`. */
|
||||
function linkDependencies(root, tree, extra) {
|
||||
const linked = [];
|
||||
const ignored = lines(git(root, ['ls-files', '--others', '--ignored', '--exclude-standard', '--directory']).stdout)
|
||||
.map((p) => p.replace(/\/$/, '').split('/')[0]);
|
||||
for (const l of new Set([...ignored, ...extra])) {
|
||||
if (!safeRel(l) || l === '.git') continue;
|
||||
if (existsSync(join(root, l)) && !existsSync(join(tree, l))) {
|
||||
mkdirSync(dirname(join(tree, l)), { recursive: true });
|
||||
symlinkSync(join(root, l), join(tree, l));
|
||||
linked.push(l);
|
||||
}
|
||||
}
|
||||
return linked;
|
||||
}
|
||||
|
||||
/** Lay the working tree's change since `base` onto `tree` (tracked edits, deletions, untracked files). */
|
||||
function applyResult(root, tree, base) {
|
||||
const status = lines(git(root, ['diff', '--name-status', '--no-renames', base]).stdout);
|
||||
for (const s of status) {
|
||||
const [code, rel] = s.split('\t');
|
||||
if (!safeRel(rel)) continue;
|
||||
if (code.startsWith('D')) rmSync(join(tree, rel), { recursive: true, force: true });
|
||||
else if (existsSync(join(root, rel)) || lstatSync(join(root, rel), { throwIfNoEntry: false })) copyIn(root, tree, rel);
|
||||
}
|
||||
for (const rel of lines(git(root, ['ls-files', '--others', '--exclude-standard']).stdout)) {
|
||||
if (safeRel(rel)) copyIn(root, tree, rel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], redOnly?: boolean, timeoutMs?: number}} o
|
||||
* @returns {{status: 'proven'|'rejected'|'not-run', reason: string, base?: object, result?: object, control?: object}}
|
||||
*/
|
||||
export function proveRed(o) {
|
||||
const given = resolve(o.cwd ?? process.cwd());
|
||||
const timeoutMs = o.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
const tests = (o.tests ?? []).filter(Boolean);
|
||||
if (!existsSync(given)) return notRun(`--cwd ${given} does not exist`);
|
||||
const cwd = repoRoot(given);
|
||||
if (!cwd) return notRun(`--cwd ${given} is not inside a git repository`);
|
||||
if (tests.length === 0) return notRun('no test files given: a step without its test cannot prove red');
|
||||
for (const t of tests) {
|
||||
if (!safeRel(t)) return notRun(`test path must be repo-relative: ${t}`);
|
||||
if (!existsSync(join(cwd, t))) return notRun(`test file not found in the working tree: ${t}`);
|
||||
}
|
||||
for (const l of o.link ?? []) if (!safeRel(l)) return notRun(`--link must be repo-relative: ${l}`);
|
||||
const allowed = allowedCommand(o.cmd, { cwd });
|
||||
if (!allowed.allowed) return notRun(`command not run: ${allowed.reason}`);
|
||||
const screen = screenCommand(o.cmd);
|
||||
if (!screen.allowed) return notRun(`command blocked: ${screen.rule}`);
|
||||
const sha = git(cwd, ['rev-parse', '--verify', '--quiet', `${o.base}^{commit}`]);
|
||||
if (sha.status !== 0) return notRun(`cannot resolve the start point ${o.base} in ${cwd}`);
|
||||
const baseSha = sha.stdout.trim();
|
||||
|
||||
// The result first: it is the working tree as it stands. `redOnly` is the
|
||||
// gate before building (the step helper's own check): the result is not run.
|
||||
let result;
|
||||
if (!o.redOnly) {
|
||||
result = run(o.cmd, cwd, timeoutMs);
|
||||
const resultBroken = unrunnable(result);
|
||||
if (resultBroken) return notRun(`at the result: ${resultBroken}`, { result });
|
||||
}
|
||||
|
||||
const tmp = mkdtempSync(join(tmpdir(), 'voyage-red-proof-'));
|
||||
try {
|
||||
const tree = join(tmp, 'tree');
|
||||
const c = git(tmp, ['clone', '--quiet', '--shared', '--no-checkout', cwd, tree]);
|
||||
if (c.status !== 0) return notRun(`could not clone the start point: ${String(c.stderr).trim()}`, { result });
|
||||
const co = git(tree, ['checkout', '--quiet', '--detach', baseSha]);
|
||||
if (co.status !== 0) return notRun(`could not check out ${baseSha}: ${String(co.stderr).trim()}`, { result });
|
||||
for (const t of tests) copyIn(cwd, tree, t);
|
||||
const linked = linkDependencies(cwd, tree, o.link ?? []);
|
||||
|
||||
const base = run(o.cmd, tree, timeoutMs);
|
||||
const baseBroken = unrunnable(base);
|
||||
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result, linked });
|
||||
const out = { base_sha: baseSha, linked, base, result };
|
||||
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...out };
|
||||
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', base_sha: baseSha, linked, base };
|
||||
if (result.exit !== 0) return { status: 'rejected', reason: 'not-green-at-result', ...out };
|
||||
|
||||
// Control: the same copy with the whole change laid on. Red here means the
|
||||
// copy itself cannot run the test green, so its red proved nothing.
|
||||
applyResult(cwd, tree, baseSha);
|
||||
const control = run(o.cmd, tree, timeoutMs);
|
||||
if (control.exit !== 0) {
|
||||
return notRun('the control run (start-point copy + the change) stayed red: the start point failed for an environment reason, not for the missing change', { ...out, control });
|
||||
}
|
||||
return { status: 'proven', reason: 'red at the start point, green at the result, green in the control', ...out, control };
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
// --- CLI ------------------------------------------------------------------------
|
||||
|
||||
const USAGE =
|
||||
'usage: red-proof.mjs --base <start_sha> --cmd "<test command>" --tests <a,b> [--red-only] [--cwd <dir>] [--link <dir,...>] [--timeout <ms>] [--json]';
|
||||
|
||||
const list = (v) => String(v).split(',').map((s) => s.trim()).filter(Boolean);
|
||||
|
||||
export function main(argv) {
|
||||
const o = { cwd: process.cwd(), tests: [], link: [] };
|
||||
let json = false;
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--base' && argv[i + 1]) o.base = argv[++i];
|
||||
else if (a === '--cmd' && argv[i + 1]) o.cmd = argv[++i];
|
||||
else if (a === '--tests' && argv[i + 1]) o.tests = list(argv[++i]);
|
||||
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||
else if (a === '--link' && argv[i + 1]) o.link = list(argv[++i]);
|
||||
else if (a === '--red-only') o.redOnly = true;
|
||||
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`red-proof: unknown argument ${a} (--tests takes ONE comma-separated list)\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if (!o.base || !o.cmd) { process.stderr.write(`red-proof: --base and --cmd are required\n${USAGE}\n`); return 2; }
|
||||
if (o.timeoutMs !== undefined && !Number.isFinite(o.timeoutMs)) { process.stderr.write(`red-proof: --timeout must be a number\n${USAGE}\n`); return 2; }
|
||||
let r;
|
||||
try { r = proveRed(o); }
|
||||
catch (e) { r = notRun(`red-proof crashed: ${e.message}`); }
|
||||
if (json) process.stdout.write(JSON.stringify(r, null, 2) + '\n');
|
||||
else {
|
||||
const counts = (x) => (x?.tap ? ` (tap: ${x.tap.fail} of ${x.tap.tests} failed)` : '');
|
||||
process.stdout.write(`red proof: ${r.status.toUpperCase()} - ${r.reason}\n`);
|
||||
if (r.base) process.stdout.write(` start point ${r.base_sha ?? o.base}: exit ${r.base.exit}${counts(r.base)}\n`);
|
||||
if (r.result) process.stdout.write(` result: exit ${r.result.exit}${counts(r.result)}\n`);
|
||||
if (r.control) process.stdout.write(` control: exit ${r.control.exit}${counts(r.control)}\n`);
|
||||
if (r.linked?.length) process.stdout.write(` linked into the start point: ${r.linked.join(', ')}\n`);
|
||||
}
|
||||
return r.status === 'proven' ? 0 : r.status === 'rejected' ? 1 : 2;
|
||||
}
|
||||
|
||||
function isMain() {
|
||||
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||
catch { return false; }
|
||||
}
|
||||
|
||||
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||
290
lib/verification/test-integrity.mjs
Normal file
290
lib/verification/test-integrity.mjs
Normal file
|
|
@ -0,0 +1,290 @@
|
|||
// lib/verification/test-integrity.mjs
|
||||
// The floor guard (P9 (2)): read a step's diff and report every change that
|
||||
// makes the tests weaker instead of the code better.
|
||||
//
|
||||
// removed-assert a test file loses more assertions than it gains
|
||||
// deleted-test a test file loses more test declarations than it gains
|
||||
// skipped-test a test is skipped, marked todo/xfail, or focused (.only)
|
||||
// muted-check a check is silenced: `|| true`, continue-on-error,
|
||||
// --passWithNoTests, an inline lint/type suppression ...
|
||||
// stub non-test code gains a "not implemented" body
|
||||
// loosened-threshold a numeric bound in a test or a check config moves in the
|
||||
// permissive direction (`ms < 200` -> `ms < 2000`)
|
||||
//
|
||||
// Exit 0 = clean, 1 = findings, 2 = could not run (no input, unreadable input,
|
||||
// an unresolvable --base). 2 is never green: an absent measurement is reported
|
||||
// as absent, not as a pass.
|
||||
//
|
||||
// A finding is not a verdict. /trekexecute hands every finding to its
|
||||
// controller, which either quotes the plan line that asked for the change or
|
||||
// fails the step; nothing is dropped silently.
|
||||
//
|
||||
// Borrowed: the floor guard of addyosmani/agent-skills@9d0c60d (MIT),
|
||||
// constraint-driven development - mechanized here as a diff reader so a
|
||||
// command, not the model's word, says whether a test was weakened.
|
||||
|
||||
import { readFileSync, realpathSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
// --- what counts as a test / a check config ----------------------------------
|
||||
|
||||
const TEST_PATH_RE = [
|
||||
/(^|\/)(tests?|__tests__|spec|specs)\//,
|
||||
/\.(test|spec)\.[a-z0-9]+$/i,
|
||||
/_(test|spec)\.[a-z0-9]+$/i,
|
||||
/(^|\/)test_[^/]*\.py$/,
|
||||
];
|
||||
|
||||
const CHECK_CONFIG_RE =
|
||||
/(^|\/)(\.github|\.forgejo|\.gitea|\.woodpecker)\/|(^|\/)(\.gitlab-ci\.yml|Makefile|package\.json|pyproject\.toml|setup\.cfg|tox\.ini|\.pre-commit-config\.yaml|tsconfig[^/]*\.json|\.eslintrc[^/]*|eslint\.config\.[a-z]+|(jest|vitest|playwright)\.config\.[a-z]+|\.c8rc[^/]*|\.nycrc[^/]*|codecov\.ya?ml)$/;
|
||||
|
||||
export function isTestPath(p) {
|
||||
return TEST_PATH_RE.some((re) => re.test(p));
|
||||
}
|
||||
|
||||
const isCheckPath = (p) => isTestPath(p) || CHECK_CONFIG_RE.test(p);
|
||||
const PROSE_RE = /\.(md|markdown|mdx|txt|rst|adoc)$/i;
|
||||
|
||||
// --- the patterns -------------------------------------------------------------
|
||||
|
||||
const ASSERT_RE =
|
||||
/\bassert\w*\b|\bexpect\s*\(|\.should\b|\bt\.(equal|is|ok|true|false|deepEqual|throws|notThrows|match)\b|\bself\.assert\w+|\brequire\.\w+\(|\bt\.(Error|Fatal)f?\(/g;
|
||||
|
||||
const TEST_DECL_RE =
|
||||
/^\s*(test|it|describe|suite)(\.\w+)?\s*\(|^\s*(async\s+)?def\s+test_\w*|^\s*func\s+Test\w*|^\s*#\[test\]|@Test\b/;
|
||||
|
||||
const SKIP_RE =
|
||||
/\b(test|it|describe|suite)\.(skip|skipIf|todo|only)\s*\(|\bt\.(skip|todo)\s*\(|\bx(it|describe|test)\s*\(|\bf(it|describe)\s*\(|\{\s*(skip|todo)\s*:\s*(true|['"`])|@pytest\.mark\.(skip|skipif|xfail)\b|\bpytest\.skip\s*\(|@unittest\.skip|\bt\.Skip(Now|f)?\s*\(|#\[ignore\]|@(Disabled|Ignore)\b/;
|
||||
|
||||
// Inline suppressions silence a check wherever they appear.
|
||||
const SUPPRESS_RE =
|
||||
/@ts-ignore|@ts-nocheck|eslint-disable|#\s*type:\s*ignore|#\s*noqa\b|\/\/\s*nolint\b|#\s*pragma:\s*no\s*cover|istanbul\s+ignore|c8\s+ignore/;
|
||||
|
||||
// Exit-code mutes count only where a check runs: tests, scripts under tests/, CI
|
||||
// and check configs. In ordinary code `|| true` is a normal shell idiom.
|
||||
const MUTE_RE =
|
||||
/\|\|\s*true\b|\|\|\s*:\s*($|[;)])|\|\|\s*exit\s+0\b|continue-on-error\s*:\s*true|--passWithNoTests|--pass-with-no-tests|\bset\s+\+e\b|--no-verify\b|allow_failure\s*:\s*true/;
|
||||
|
||||
const STUB_RE =
|
||||
/throw\s+new\s+Error\s*\(\s*['"`]\s*(not\s+implemented|unimplemented|todo)\b|\braise\s+NotImplementedError\b|\b(unimplemented|todo)!\s*\(|panic\s*\(\s*"(not\s+implemented|todo)|(\/\/|#)\s*stub\b|TODO:?\s*implement/i;
|
||||
|
||||
// Number after a "smaller is stricter" bound: bigger = looser.
|
||||
const UPPER_BOUND_RE = /(<=?|toBeLessThan(OrEqual)?|lessThan|\bmax\w*|timeout\w*|tolerance|epsilon|delta|within|retries)\W*$/i;
|
||||
// Number after a "bigger is stricter" bound: smaller = looser.
|
||||
const LOWER_BOUND_RE = /(>=?|toBeGreaterThan(OrEqual)?|greaterThan|\bmin\w*|threshold|coverage|branches|lines|functions|statements)\W*$/i;
|
||||
|
||||
const NUM_RE = /\d+(?:\.\d+)?/g;
|
||||
|
||||
// --- diff parsing ---------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Parse a unified diff into files with hunks of {type: ' '|'-'|'+', text, line}.
|
||||
* Returns null when non-empty input carries no diff header at all.
|
||||
*/
|
||||
export function parseDiff(text) {
|
||||
const src = String(text ?? '');
|
||||
if (src.trim() === '') return [];
|
||||
if (!/^(diff --git |--- |\+\+\+ )/m.test(src)) return null;
|
||||
const files = [];
|
||||
let file = null;
|
||||
let hunk = null;
|
||||
let oldLine = 0;
|
||||
let newLine = 0;
|
||||
// Lines still owed to the current hunk, from its @@ header. While a hunk is
|
||||
// owed lines, every line is content - even one that reads "--- x" or
|
||||
// "+++ y" (a removed "-- x" / an added "++ y"). Headers only between hunks.
|
||||
let oldLeft = 0;
|
||||
let newLeft = 0;
|
||||
const lines = src.split('\n');
|
||||
for (const l of lines) {
|
||||
if (hunk && (oldLeft > 0 || newLeft > 0)) {
|
||||
if (l.startsWith('\\')) continue; // "\ No newline at end of file"
|
||||
if (l.startsWith('-')) { hunk.push({ type: '-', text: l.slice(1), line: oldLine++ }); oldLeft--; continue; }
|
||||
if (l.startsWith('+')) { hunk.push({ type: '+', text: l.slice(1), line: newLine++ }); newLeft--; continue; }
|
||||
if (l.startsWith(' ') || l === '') {
|
||||
hunk.push({ type: ' ', text: l.slice(1), line: newLine });
|
||||
oldLine++; newLine++; oldLeft--; newLeft--;
|
||||
continue;
|
||||
}
|
||||
// Anything else ends a malformed hunk early; fall through to headers.
|
||||
oldLeft = 0; newLeft = 0;
|
||||
}
|
||||
if (l.startsWith('diff --git ')) {
|
||||
const m = l.match(/^diff --git a\/(.+) b\/(.+)$/);
|
||||
file = { path: m ? m[2] : l.slice(11), deleted: false, hunks: [] };
|
||||
files.push(file);
|
||||
hunk = null;
|
||||
continue;
|
||||
}
|
||||
if (l.startsWith('--- ')) {
|
||||
if (!file || file.hunks.length > 0) { file = { path: '', deleted: false, hunks: [] }; files.push(file); }
|
||||
const oldPath = l.slice(4).replace(/^a\//, '');
|
||||
if (!file.path && oldPath !== '/dev/null') file.path = oldPath;
|
||||
hunk = null;
|
||||
continue;
|
||||
}
|
||||
if (l.startsWith('+++ ') && file) {
|
||||
const newPath = l.slice(4).replace(/^b\//, '');
|
||||
if (newPath === '/dev/null') file.deleted = true;
|
||||
else file.path = newPath;
|
||||
continue;
|
||||
}
|
||||
const h = l.match(/^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@/);
|
||||
if (h && file) {
|
||||
oldLine = Number(h[1]);
|
||||
newLine = Number(h[3]);
|
||||
oldLeft = h[2] === undefined ? 1 : Number(h[2]);
|
||||
newLeft = h[4] === undefined ? 1 : Number(h[4]);
|
||||
hunk = [];
|
||||
file.hunks.push(hunk);
|
||||
}
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
// --- detectors ------------------------------------------------------------------
|
||||
|
||||
const count = (lines, re) => lines.reduce((n, l) => n + (l.text.match(new RegExp(re.source, 'g')) || []).length, 0);
|
||||
const norm = (s) => s.trim().replace(/\s+/g, ' ');
|
||||
|
||||
function addedNotRemoved(removed, added, re) {
|
||||
const before = new Set(removed.filter((l) => re.test(l.text)).map((l) => norm(l.text)));
|
||||
return added.filter((l) => re.test(l.text) && !before.has(norm(l.text)));
|
||||
}
|
||||
|
||||
/** A paired -/+ line whose only change is numbers; returns the looser numbers. */
|
||||
function loosened(oldText, newText) {
|
||||
const shape = (s) => norm(s).replace(NUM_RE, '#');
|
||||
if (shape(oldText) !== shape(newText)) return null;
|
||||
const a = [...norm(oldText).matchAll(NUM_RE)];
|
||||
const b = [...norm(newText).matchAll(NUM_RE)];
|
||||
for (let i = 0; i < a.length; i++) {
|
||||
const before = Number(a[i][0]);
|
||||
const after = Number(b[i][0]);
|
||||
if (before === after) continue;
|
||||
const lead = norm(newText).slice(Math.max(0, b[i].index - 30), b[i].index);
|
||||
if (UPPER_BOUND_RE.test(lead) && after > before) return `${before} -> ${after}`;
|
||||
if (LOWER_BOUND_RE.test(lead) && after < before) return `${before} -> ${after}`;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
function finding(kind, file, l, detail) {
|
||||
return { kind, file, line: l ? l.line : null, text: l ? l.text.trim() : '', ...(detail ? { detail } : {}) };
|
||||
}
|
||||
|
||||
export function analyzeDiff(text) {
|
||||
const files = parseDiff(text);
|
||||
if (files === null) return { status: 'unreadable', findings: [], reason: 'the input is not a unified diff' };
|
||||
const findings = [];
|
||||
for (const f of files) {
|
||||
const all = f.hunks.flat();
|
||||
const removed = all.filter((l) => l.type === '-');
|
||||
const added = all.filter((l) => l.type === '+');
|
||||
const test = isTestPath(f.path);
|
||||
// Prose (docs, notes) describes code; a "not implemented" or an
|
||||
// eslint-disable written in it is text, not a stub or a muted check.
|
||||
const prose = PROSE_RE.test(f.path);
|
||||
|
||||
if (test) {
|
||||
const lostAsserts = count(removed, ASSERT_RE) - count(added, ASSERT_RE);
|
||||
if (lostAsserts > 0) {
|
||||
findings.push(finding('removed-assert', f.path, removed.find((l) => new RegExp(ASSERT_RE.source).test(l.text)), `${lostAsserts} fewer`));
|
||||
}
|
||||
const lostTests = removed.filter((l) => TEST_DECL_RE.test(l.text)).length - added.filter((l) => TEST_DECL_RE.test(l.text)).length;
|
||||
if (lostTests > 0 || (f.deleted && removed.some((l) => TEST_DECL_RE.test(l.text)))) {
|
||||
findings.push(finding('deleted-test', f.path, removed.find((l) => TEST_DECL_RE.test(l.text)), f.deleted ? 'test file deleted' : `${lostTests} fewer`));
|
||||
}
|
||||
for (const l of addedNotRemoved(removed, added, SKIP_RE)) findings.push(finding('skipped-test', f.path, l));
|
||||
}
|
||||
|
||||
if (!prose) for (const l of addedNotRemoved(removed, added, SUPPRESS_RE)) findings.push(finding('muted-check', f.path, l));
|
||||
if (isCheckPath(f.path)) {
|
||||
for (const l of addedNotRemoved(removed, added, MUTE_RE)) findings.push(finding('muted-check', f.path, l));
|
||||
for (const h of f.hunks) {
|
||||
const minus = h.filter((l) => l.type === '-');
|
||||
const plus = h.filter((l) => l.type === '+');
|
||||
for (const m of minus) {
|
||||
for (const p of plus) {
|
||||
const d = loosened(m.text, p.text);
|
||||
if (d) { findings.push(finding('loosened-threshold', f.path, p, d)); break; }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (!test && !prose) for (const l of addedNotRemoved(removed, added, STUB_RE)) findings.push(finding('stub', f.path, l));
|
||||
}
|
||||
return { status: findings.length ? 'findings' : 'clean', findings };
|
||||
}
|
||||
|
||||
// --- CLI ------------------------------------------------------------------------
|
||||
|
||||
const USAGE = 'usage: test-integrity.mjs (--diff <file|-> | --base <ref> [--head <ref>]) [--cwd <dir>] [--json]';
|
||||
|
||||
function diffFromGit(cwd, base, head) {
|
||||
const ok = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${base}^{commit}`], { cwd, encoding: 'utf8' });
|
||||
if (ok.status !== 0) return { error: `cannot resolve --base ${base} in ${cwd}` };
|
||||
if (head) {
|
||||
const okHead = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${head}^{commit}`], { cwd, encoding: 'utf8' });
|
||||
if (okHead.status !== 0) return { error: `cannot resolve --head ${head} in ${cwd}` };
|
||||
}
|
||||
// --no-renames: a rename is read as a deletion plus an addition. Paired as a
|
||||
// rename, `foo.test.js -> foo.test.js.bak` showed no removed line at all and
|
||||
// switched a test off silently (review, 2026-10-06).
|
||||
const args = ['diff', '--no-color', '--no-ext-diff', '--no-renames', base, ...(head ? [head] : [])];
|
||||
const r = spawnSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
|
||||
if (r.status !== 0) return { error: `git diff failed: ${String(r.stderr).trim()}` };
|
||||
return { text: r.stdout };
|
||||
}
|
||||
|
||||
export function main(argv) {
|
||||
let diffPath = null;
|
||||
let base = null;
|
||||
let head = null;
|
||||
let cwd = process.cwd();
|
||||
let json = false;
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
if (a === '--diff' && argv[i + 1] !== undefined) diffPath = argv[++i];
|
||||
else if (a === '--base' && argv[i + 1]) base = argv[++i];
|
||||
else if (a === '--head' && argv[i + 1]) head = argv[++i];
|
||||
else if (a === '--cwd' && argv[i + 1]) cwd = resolve(argv[++i]);
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`test-integrity: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if ((diffPath === null) === (base === null)) {
|
||||
process.stderr.write(`test-integrity: give exactly one of --diff or --base\n${USAGE}\n`);
|
||||
return 2;
|
||||
}
|
||||
let text;
|
||||
if (diffPath !== null) {
|
||||
try { text = readFileSync(diffPath === '-' ? 0 : diffPath, 'utf8'); }
|
||||
catch (e) { return out({ status: 'unreadable', findings: [], reason: `cannot read ${diffPath}: ${e.message}` }, json); }
|
||||
} else {
|
||||
const g = diffFromGit(cwd, base, head);
|
||||
if (g.error) return out({ status: 'unreadable', findings: [], reason: g.error }, json);
|
||||
text = g.text;
|
||||
}
|
||||
return out(analyzeDiff(text), json);
|
||||
}
|
||||
|
||||
function out(r, json) {
|
||||
if (json) process.stdout.write(JSON.stringify(r, null, 2) + '\n');
|
||||
else if (r.status === 'unreadable') process.stdout.write(`floor guard: NOT RUN - ${r.reason} (exit 2, never green)\n`);
|
||||
else if (r.status === 'clean') process.stdout.write('floor guard: clean\n');
|
||||
else {
|
||||
process.stdout.write(`floor guard: ${r.findings.length} finding(s)\n`);
|
||||
for (const f of r.findings) process.stdout.write(` ${f.kind} ${f.file}:${f.line ?? '-'} ${f.text}${f.detail ? ` (${f.detail})` : ''}\n`);
|
||||
}
|
||||
return r.status === 'clean' ? 0 : r.status === 'findings' ? 1 : 2;
|
||||
}
|
||||
|
||||
function isMain() {
|
||||
try { return realpathSync(process.argv[1]) === realpathSync(fileURLToPath(import.meta.url)); }
|
||||
catch { return false; }
|
||||
}
|
||||
|
||||
if (process.argv[1] && isMain()) process.exitCode = main(process.argv.slice(2));
|
||||
|
|
@ -168,3 +168,61 @@ test('SC5 control: the extractors report the old vocabulary', () => {
|
|||
const bad = offenders(writtenStatuses(old)).join('\n');
|
||||
for (const word of ['"in-progress"', '"passed"', '"running"']) assert.ok(bad.includes(word), `control must flag ${word}:\n${bad}`);
|
||||
});
|
||||
|
||||
// --- Handover 5, wave 1 (P8/P9): the per-step fields of the step loop ---------
|
||||
// fix_rounds counts the rounds a step went through after its first control:
|
||||
// 0 = passed first time, 1-3 = fix rounds, 4 = sent up to the plan, 5 = sent up
|
||||
// to the operator. There is no sixth round, so 6 is invalid, not a warning.
|
||||
// All new fields are optional on read: a progress.json written before them
|
||||
// still resumes (SC4 above validates the old shape).
|
||||
|
||||
const { validateProgressObject } = progressValidator;
|
||||
|
||||
function withStep(extra) {
|
||||
return {
|
||||
schema_version: '1', plan: 'plan.md', plan_version: '1.8', mode: 'execute', status: 'in_progress',
|
||||
total_steps: 1, current_step: 1,
|
||||
steps: { 1: { status: 'in_progress', attempts: 1, ...extra } },
|
||||
};
|
||||
}
|
||||
|
||||
const codes = (r) => r.errors.map((e) => e.code);
|
||||
|
||||
test('H5: fix_rounds 0-5 is accepted, 6 is rejected', () => {
|
||||
for (const n of [0, 1, 2, 3, 4, 5]) {
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ fix_rounds: n }))), [], `fix_rounds ${n}`);
|
||||
}
|
||||
for (const bad of [6, -1, 2.5, '3']) {
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ fix_rounds: bad }))), ['PROGRESS_FIX_ROUNDS_INVALID'], `fix_rounds ${JSON.stringify(bad)}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('H5: start_sha, report_path, red_proof and test_lock are shape-checked when present', () => {
|
||||
const ok = { start_sha: '2c89a6b580f689070638f3ff83aea82d355f312f', report_path: '.claude/projects/x/steps/step-1.report.md', red_proof: 'proven', test_lock: 'not-locked' };
|
||||
assert.deepEqual(codes(validateProgressObject(withStep(ok))), []);
|
||||
for (const v of ['proven', 'rejected', 'not-run']) assert.deepEqual(codes(validateProgressObject(withStep({ red_proof: v }))), [], v);
|
||||
for (const v of ['locked', 'released', 'not-locked']) assert.deepEqual(codes(validateProgressObject(withStep({ test_lock: v }))), [], v);
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ start_sha: 'HEAD' }))), ['PROGRESS_START_SHA_INVALID']);
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ report_path: '' }))), ['PROGRESS_REPORT_PATH_INVALID']);
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ red_proof: 'green' }))), ['PROGRESS_RED_PROOF_INVALID']);
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ test_lock: 'yes' }))), ['PROGRESS_TEST_LOCK_INVALID']);
|
||||
// null = not set yet (the schema fence's fresh step).
|
||||
assert.deepEqual(codes(validateProgressObject(withStep({ start_sha: null, report_path: null, red_proof: null, test_lock: null }))), []);
|
||||
});
|
||||
|
||||
test('H5: the run-level decision log is an array of {decision, cost_if_wrong}', () => {
|
||||
const run = (decisions) => ({ ...withStep({}), decisions });
|
||||
assert.deepEqual(codes(validateProgressObject(run([{ step: 2, decision: 'kept the old name', cost_if_wrong: 'one rename commit' }]))), []);
|
||||
assert.deepEqual(codes(validateProgressObject(run([{ step: 2, decision: 'kept the old name' }]))), ['PROGRESS_DECISION_INVALID']);
|
||||
assert.deepEqual(codes(validateProgressObject(run('none'))), ['PROGRESS_DECISION_INVALID']);
|
||||
});
|
||||
|
||||
test('H5: the progress schema fence carries the new per-step fields and the decision log', () => {
|
||||
const schema = schemaObject(read());
|
||||
const step = schema.steps['1'];
|
||||
for (const k of ['start_sha', 'report_path', 'red_proof', 'test_lock', 'fix_rounds']) {
|
||||
assert.ok(k in step, `the schema's step record must carry ${k}`);
|
||||
}
|
||||
assert.equal(step.fix_rounds, 0);
|
||||
assert.ok(Array.isArray(schema.decisions), 'the schema must carry the run-level decisions log');
|
||||
});
|
||||
|
|
|
|||
|
|
@ -109,11 +109,16 @@ test('trekexecute — global recovery/retry budget TREKEXECUTE_MAX_RECOVERY_ITER
|
|||
|
||||
test('trekexecute — every recovery/retry loop bounded + 3-axis cap hierarchy (SC b)', () => {
|
||||
const text = read();
|
||||
assert.match(text, /maximum 2 retries|Retry cap = 3 attempts/, 'per-step retry cap must be explicit');
|
||||
// Wave 1 (P8): the per-step cap is at most 5 fix rounds, and they go UP (to the
|
||||
// plan, then to the operator) - never sideways into a fourth guess, never down
|
||||
// a ladder of cheaper models. The old "3 attempts" cap is gone.
|
||||
assert.match(text, /maximum 5 fix rounds[^\n]*upward/i, 'per-step cap must be "maximum 5 fix rounds", escalating upward');
|
||||
assert.doesNotMatch(text, /maximum 2 retries|Retry cap = 3 attempts/, 'the old 3-attempt retry cap must be gone');
|
||||
assert.match(text, /recovery_depth < 2/, 'recovery-dispatch cap must be explicit');
|
||||
const capIdx = text.indexOf('Iteration caps');
|
||||
assert.ok(capIdx >= 0, 'cap-hierarchy section ("Iteration caps") must exist');
|
||||
const section = text.slice(capIdx, capIdx + 1500);
|
||||
assert.match(section, /fix_rounds/, 'hierarchy must name the per-step fix_rounds axis');
|
||||
assert.match(section, /attempts/, 'hierarchy must name per-step attempts axis');
|
||||
assert.match(section, /recovery_depth/, 'hierarchy must name per-session recovery_depth axis');
|
||||
assert.match(section, /TREKEXECUTE_MAX_RECOVERY_ITERATIONS/, 'hierarchy must name the global budget axis');
|
||||
|
|
|
|||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -22,7 +22,7 @@
|
|||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1124-1139 | profile, phase_models, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:1890-1912 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2325-2347 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
||||
|
|
|
|||
22
tests/fixtures/test-integrity/clean.diff
vendored
Normal file
22
tests/fixtures/test-integrity/clean.diff
vendored
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
diff --git a/lib/sum.mjs b/lib/sum.mjs
|
||||
index 1111111..2222222 100644
|
||||
--- a/lib/sum.mjs
|
||||
+++ b/lib/sum.mjs
|
||||
@@ -1,3 +1,4 @@
|
||||
export function sum(a, b) {
|
||||
- return a - b;
|
||||
+ // fixed: addition, not subtraction
|
||||
+ return a + b;
|
||||
}
|
||||
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||
index 3333333..4444444 100644
|
||||
--- a/tests/sum.test.mjs
|
||||
+++ b/tests/sum.test.mjs
|
||||
@@ -3,3 +3,7 @@ import { sum } from '../lib/sum.mjs';
|
||||
test('sum adds', () => {
|
||||
assert.equal(sum(1, 2), 3);
|
||||
});
|
||||
+
|
||||
+test('sum adds negatives', () => {
|
||||
+ assert.equal(sum(-1, -2), -3);
|
||||
+});
|
||||
11
tests/fixtures/test-integrity/deleted-test.diff
vendored
Normal file
11
tests/fixtures/test-integrity/deleted-test.diff
vendored
Normal file
|
|
@ -0,0 +1,11 @@
|
|||
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||
index 3333333..4444444 100644
|
||||
--- a/tests/sum.test.mjs
|
||||
+++ b/tests/sum.test.mjs
|
||||
@@ -3,6 +3,3 @@ import { sum } from '../lib/sum.mjs';
|
||||
test('sum adds', () => {
|
||||
assert.equal(sum(1, 2), 3);
|
||||
});
|
||||
-test('sum adds negatives', () => {
|
||||
- sum(-1, -2);
|
||||
-});
|
||||
9
tests/fixtures/test-integrity/loosened-threshold.diff
vendored
Normal file
9
tests/fixtures/test-integrity/loosened-threshold.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
diff --git a/tests/latency.test.mjs b/tests/latency.test.mjs
|
||||
index 7777777..8888888 100644
|
||||
--- a/tests/latency.test.mjs
|
||||
+++ b/tests/latency.test.mjs
|
||||
@@ -4,3 +4,3 @@ test('tail latency', () => {
|
||||
const ms = measure();
|
||||
- assert.ok(ms < 200, `tail latency ${ms}ms`);
|
||||
+ assert.ok(ms < 2000, `tail latency ${ms}ms`);
|
||||
});
|
||||
8
tests/fixtures/test-integrity/muted-check.diff
vendored
Normal file
8
tests/fixtures/test-integrity/muted-check.diff
vendored
Normal file
|
|
@ -0,0 +1,8 @@
|
|||
diff --git a/tests/run.sh b/tests/run.sh
|
||||
index 5555555..6666666 100644
|
||||
--- a/tests/run.sh
|
||||
+++ b/tests/run.sh
|
||||
@@ -1,2 +1,2 @@
|
||||
#!/bin/sh
|
||||
-node --test tests/
|
||||
+node --test tests/ || true
|
||||
9
tests/fixtures/test-integrity/removed-assert.diff
vendored
Normal file
9
tests/fixtures/test-integrity/removed-assert.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||
index 3333333..4444444 100644
|
||||
--- a/tests/sum.test.mjs
|
||||
+++ b/tests/sum.test.mjs
|
||||
@@ -3,5 +3,4 @@ import { sum } from '../lib/sum.mjs';
|
||||
test('sum adds', () => {
|
||||
assert.equal(sum(1, 2), 3);
|
||||
- assert.equal(sum(2, 2), 4);
|
||||
});
|
||||
9
tests/fixtures/test-integrity/skipped-test.diff
vendored
Normal file
9
tests/fixtures/test-integrity/skipped-test.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
diff --git a/tests/sum.test.mjs b/tests/sum.test.mjs
|
||||
index 3333333..4444444 100644
|
||||
--- a/tests/sum.test.mjs
|
||||
+++ b/tests/sum.test.mjs
|
||||
@@ -3,3 +3,3 @@ import { sum } from '../lib/sum.mjs';
|
||||
-test('sum adds', () => {
|
||||
+test.skip('sum adds', () => {
|
||||
assert.equal(sum(1, 2), 3);
|
||||
});
|
||||
9
tests/fixtures/test-integrity/stub.diff
vendored
Normal file
9
tests/fixtures/test-integrity/stub.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
diff --git a/lib/sum.mjs b/lib/sum.mjs
|
||||
index 1111111..2222222 100644
|
||||
--- a/lib/sum.mjs
|
||||
+++ b/lib/sum.mjs
|
||||
@@ -1,3 +1,3 @@
|
||||
export function sum(a, b) {
|
||||
- return a - b;
|
||||
+ throw new Error('not implemented');
|
||||
}
|
||||
9
tests/fixtures/test-integrity/tightened-threshold.diff
vendored
Normal file
9
tests/fixtures/test-integrity/tightened-threshold.diff
vendored
Normal file
|
|
@ -0,0 +1,9 @@
|
|||
diff --git a/tests/latency.test.mjs b/tests/latency.test.mjs
|
||||
index 7777777..8888888 100644
|
||||
--- a/tests/latency.test.mjs
|
||||
+++ b/tests/latency.test.mjs
|
||||
@@ -4,3 +4,3 @@ test('tail latency', () => {
|
||||
const ms = measure();
|
||||
- assert.ok(ms < 2000, `tail latency ${ms}ms`);
|
||||
+ assert.ok(ms < 200, `tail latency ${ms}ms`);
|
||||
});
|
||||
2
tests/fixtures/test-integrity/unreadable.diff
vendored
Normal file
2
tests/fixtures/test-integrity/unreadable.diff
vendored
Normal file
|
|
@ -0,0 +1,2 @@
|
|||
this is not a diff
|
||||
just prose that a broken pipe left behind
|
||||
|
|
@ -1668,9 +1668,11 @@ for (const cmd of ['trekresearch', 'trekplan', 'trekreview', 'trekexecute']) {
|
|||
// The derived set is asserted exactly, so the pin cannot go vacuous if a
|
||||
// command stops spawning or a new spawning command is added.
|
||||
|
||||
const SPAWNING_COMMANDS = ['trekbrief', 'trekplan', 'trekresearch', 'trekreview'];
|
||||
// Wave 1 (P8): /trekexecute joined the set - one fresh step helper per plan step
|
||||
// plus one controller - so the no-name rule now covers five commands.
|
||||
const SPAWNING_COMMANDS = ['trekbrief', 'trekexecute', 'trekplan', 'trekresearch', 'trekreview'];
|
||||
|
||||
test('S85: the set of agent-spawning commands is exactly the four that carry the no-name rule', () => {
|
||||
test('S85: the set of agent-spawning commands is exactly the five that carry the no-name rule', () => {
|
||||
const spawns = listMd('commands')
|
||||
.filter((f) => /^Launch\b/m.test(read(`commands/${f}`)))
|
||||
.map((f) => f.replace(/\.md$/, ''))
|
||||
|
|
|
|||
93
tests/lib/done-line.test.mjs
Normal file
93
tests/lib/done-line.test.mjs
Normal file
|
|
@ -0,0 +1,93 @@
|
|||
// tests/lib/done-line.test.mjs
|
||||
// The done line (P9 (4)): a step's "done" is written by a script, and only when
|
||||
// the test command exits 0. Exit 0 = written, 1 = the command failed (nothing
|
||||
// written), 2 = could not run (nothing written, never green).
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = join(HERE, '..', '..');
|
||||
const CLI = join(ROOT, 'lib', 'verification', 'done-line.mjs');
|
||||
|
||||
function sh(cwd, cmd, args) {
|
||||
const r = spawnSync(cmd, args, { cwd, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `${cmd} ${args.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
function repo() {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'done-line-'));
|
||||
sh(dir, 'git', ['init', '-q']);
|
||||
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'ok.test.mjs'), "import { test } from 'node:test';\ntest('ok', () => {});\n");
|
||||
writeFileSync(join(dir, 'tests', 'bad.test.mjs'), "import { test } from 'node:test';\ntest('bad', () => { throw new Error('red'); });\n");
|
||||
sh(dir, 'git', ['add', 'tests']);
|
||||
sh(dir, 'git', ['commit', '-q', '-m', 'tests']);
|
||||
return dir;
|
||||
}
|
||||
|
||||
const run = (args) => spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8' });
|
||||
|
||||
test('done line: the test command exits 0 -> one DONE line appended, exit 0', () => {
|
||||
const dir = repo();
|
||||
const report = join(dir, 'step-3.report.md');
|
||||
try {
|
||||
writeFileSync(report, '# Step 3 report\n');
|
||||
const r = run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/ok.test.mjs', '--report', report]);
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const head = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
const lines = readFileSync(report, 'utf8').split('\n').filter((l) => l.startsWith('DONE '));
|
||||
assert.equal(lines.length, 1);
|
||||
assert.match(lines[0], new RegExp(`^DONE step 3: \`node --test tests/ok\\.test\\.mjs\` exit 0 at ${head} \\d{4}-\\d{2}-\\d{2}T`));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('done line: the test command fails -> nothing written, exit 1', () => {
|
||||
const dir = repo();
|
||||
const report = join(dir, 'step-3.report.md');
|
||||
try {
|
||||
writeFileSync(report, '# Step 3 report\n');
|
||||
const r = run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/bad.test.mjs', '--report', report]);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.equal(readFileSync(report, 'utf8'), '# Step 3 report\n');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('done line: a command outside the test-runner allowlist is not run -> exit 2, nothing written', () => {
|
||||
const dir = repo();
|
||||
const report = join(dir, 'step-3.report.md');
|
||||
try {
|
||||
for (const cmd of ['echo done', 'node --test tests/bad.test.mjs || true', 'true']) {
|
||||
const r = run(['--cwd', dir, '--step', '3', '--cmd', cmd, '--report', report]);
|
||||
assert.equal(r.status, 2, `${cmd}: ${r.stdout}${r.stderr}`);
|
||||
}
|
||||
assert.equal(existsSync(report), false);
|
||||
assert.equal(run(['--cwd', dir, '--cmd', 'node --test tests/ok.test.mjs', '--report', report]).status, 2, 'no --step');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('done line: --check finds the line for that step only', () => {
|
||||
const dir = repo();
|
||||
const report = join(dir, 'step-3.report.md');
|
||||
try {
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'no report file = not done');
|
||||
writeFileSync(report, '# Step 3 report\nDONE step 3 (typed by hand)\n');
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'a hand-typed line is not the script line');
|
||||
// A well-formed line typed by hand, at a sha that is not the step's commit.
|
||||
writeFileSync(report, `# Step 3 report\nDONE step 3: \`node --test tests/ok.test.mjs\` exit 0 at ${'0'.repeat(40)} 2026-10-06T12:00:00.000Z\n`);
|
||||
const head = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report, '--commit', head]).status, 1, 'the line must be at the step commit');
|
||||
run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/ok.test.mjs', '--report', report]);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 0);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report, '--commit', head]).status, 0);
|
||||
assert.equal(run(['--check', '--step', '4', '--report', report]).status, 1);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
244
tests/lib/executor-prose-pins.test.mjs
Normal file
244
tests/lib/executor-prose-pins.test.mjs
Normal file
|
|
@ -0,0 +1,244 @@
|
|||
// tests/lib/executor-prose-pins.test.mjs
|
||||
// Wave 1, process 1 (the executor): prose pins for P8 (fresh helper per step,
|
||||
// one controller with two verdicts, fix rounds that go up), P9 on the executor
|
||||
// side (proof of red, the floor guard, the done line, end to end, back to the
|
||||
// architecture after three failed fixes) and P4 (the closed stop list and the
|
||||
// decision log). The mechanisms themselves are tested as code in
|
||||
// tests/lib/{red-proof,test-integrity,done-line}.test.mjs; these pins hold the
|
||||
// command prose to them. Fix the SOURCE a pin guards, not the pin.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = join(HERE, '..', '..');
|
||||
const read = (rel) => readFileSync(join(ROOT, rel), 'utf8');
|
||||
const CMD = 'commands/trekexecute.md';
|
||||
const AGENT = 'agents/step-implementer.md';
|
||||
|
||||
function frontmatter(text) {
|
||||
const m = text.match(/^---\n([\s\S]*?)\n---/);
|
||||
assert.ok(m, 'frontmatter missing');
|
||||
return m[1];
|
||||
}
|
||||
const fmValue = (fm, key) => (fm.match(new RegExp(`^${key}:\\s*(.+)$`, 'm')) || [])[1];
|
||||
|
||||
/** The text between `## {start}` (or `#### {start}`) and the next heading of the same or higher level. */
|
||||
function section(text, heading) {
|
||||
const at = text.indexOf(heading);
|
||||
assert.ok(at >= 0, `${CMD} must have "${heading}"`);
|
||||
const level = heading.match(/^#+/)[0].length;
|
||||
const rest = text.slice(at + heading.length);
|
||||
const next = rest.search(new RegExp(`\\n#{1,${level}} `));
|
||||
return heading + (next >= 0 ? rest.slice(0, next) : rest);
|
||||
}
|
||||
|
||||
// --- P8: the Agent tool is back, for one purpose -------------------------------
|
||||
|
||||
test('P8: /trekexecute may spawn - Agent is allowed and no longer disallowed', () => {
|
||||
const fm = frontmatter(read(CMD));
|
||||
assert.match(fmValue(fm, 'allowed-tools'), /\bAgent\b/, 'allowed-tools must list Agent');
|
||||
const disallowed = fmValue(fm, 'disallowed-tools') || '';
|
||||
assert.doesNotMatch(disallowed, /\bAgent\b/, 'disallowed-tools must no longer remove Agent');
|
||||
assert.match(disallowed, /\bTeamCreate\b/, 'TeamCreate stays removed - helpers, never a team');
|
||||
});
|
||||
|
||||
test('P8: Hard rule 10 is rewritten - one fresh helper per plan step, and helpers never start helpers', () => {
|
||||
const text = read(CMD);
|
||||
assert.doesNotMatch(text, /\*\*No sub-agents\.\*\*/, 'the old "No sub-agents" rule must be gone');
|
||||
const rule = text.slice(text.indexOf('10. **'), text.indexOf('11. **'));
|
||||
assert.match(rule, /fresh helper per plan step/i);
|
||||
assert.match(rule, /voyage:step-implementer/);
|
||||
assert.match(rule, /helpers never start helpers/i);
|
||||
assert.match(rule, /controller/i);
|
||||
});
|
||||
|
||||
test('P8: the step agent exists, is opus at effort >= high, and cannot spawn', () => {
|
||||
assert.ok(existsSync(join(ROOT, AGENT)), `${AGENT} must exist`);
|
||||
const fm = frontmatter(read(AGENT));
|
||||
assert.equal(fmValue(fm, 'name'), 'step-implementer');
|
||||
assert.equal(fmValue(fm, 'model'), 'opus');
|
||||
assert.ok(['high', 'xhigh', 'max'].includes(fmValue(fm, 'effort')), 'effort must be >= high');
|
||||
assert.doesNotMatch(fmValue(fm, 'tools') || '', /\bAgent\b/, 'a helper must not hold the Agent tool');
|
||||
// Measured 2026-10-06 (CC 2.1.291): a worktree-isolated helper branched from
|
||||
// the default branch (02f320f), not from the executor's HEAD (2c89a6b), and so
|
||||
// did not see the earlier steps. A plugin cannot set worktree.baseRef. So no
|
||||
// isolation: the helper works in the executor's tree, from a recorded start_sha.
|
||||
assert.doesNotMatch(fm, /^isolation:/m, 'no isolation: worktree on the step agent (measured: wrong base)');
|
||||
});
|
||||
|
||||
test('P8: the helper report contract is ran / saw / did NOT check', () => {
|
||||
const body = read(AGENT);
|
||||
for (const h of ['### Ran', '### Saw', '### Did NOT check']) assert.ok(body.includes(h), `${AGENT} report must have "${h}"`);
|
||||
assert.ok(read(CMD).includes('Did NOT check'), 'the command must name the report contract too');
|
||||
});
|
||||
|
||||
test('P8: hand-over is in files, from a recorded start point', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /start_sha/);
|
||||
assert.match(text, /report_path/);
|
||||
assert.match(text, /git diff \{start_sha\}/, 'the review package is the diff from the recorded start point');
|
||||
});
|
||||
|
||||
test('P8: one controller, two verdicts, reads the diff not the summary, re-checks only what was fixed', () => {
|
||||
const ctl = section(read(CMD), '#### Sub-step E — Control');
|
||||
assert.match(ctl, /one controller/i);
|
||||
assert.match(ctl, /two verdicts/i);
|
||||
assert.match(ctl, /"spec"/);
|
||||
assert.match(ctl, /"quality"/);
|
||||
assert.match(ctl, /reads the diff, not the helper's (summary|report)/i);
|
||||
assert.match(ctl, /only what (was|the round) fixed/i);
|
||||
assert.match(ctl, /general-purpose/, 'the controller is a general helper with a prompt from the command, not agent no. 2');
|
||||
});
|
||||
|
||||
test('P8: fix rounds go up - maximum 5, plan then operator, never a cheaper-model ladder', () => {
|
||||
const fix = section(read(CMD), '#### Sub-step F — Fix rounds');
|
||||
assert.match(fix, /maximum 5 fix rounds[^\n]*upward/i);
|
||||
assert.match(fix, /back to (the )?planning/i);
|
||||
assert.match(fix, /operator/i);
|
||||
assert.match(fix, /does not improvise/i);
|
||||
assert.match(fix, /no ladder of cheaper models/i);
|
||||
// P9 (7): after three failed fixes the question is the architecture, not a fourth fix.
|
||||
assert.match(fix, /three failed fix(es| rounds)[^\n]*architecture/i);
|
||||
});
|
||||
|
||||
test('P8: the no-name rule covers the step helper and the controller', () => {
|
||||
const text = read(CMD);
|
||||
assert.ok(text.includes('in_process_teammate'));
|
||||
assert.match(text, /never pass .*`name`/i);
|
||||
assert.match(text, /^Launch\b/m, 'the spawn site starts with "Launch" (the S85 detector reads it)');
|
||||
});
|
||||
|
||||
// --- P9 (executor side) ---------------------------------------------------------
|
||||
|
||||
test('P9 (1): proof of red is a command and it blocks - the old "do not block" sentence is gone', () => {
|
||||
const text = read(CMD);
|
||||
assert.doesNotMatch(text, /Do not block on test-first failures/);
|
||||
assert.match(text, /lib\/verification\/red-proof\.mjs/);
|
||||
const red = section(text, '#### Sub-step B — Test first and proof of red');
|
||||
assert.match(red, /must FAIL/);
|
||||
assert.match(red, /must PASS/);
|
||||
assert.match(red, /not-run/);
|
||||
assert.match(red, /never (a pass|green|proven)/i);
|
||||
});
|
||||
|
||||
test('K4: Test first is required from plan_version 1.8; Interface in/out and Review focus are read, not required', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /plan_version ≥ 1\.8[^\n]*`Test first:`|`Test first:`[^\n]*plan_version ≥ 1\.8/);
|
||||
for (const f of ['Interface in:', 'Interface out:', 'Review focus:']) assert.ok(text.includes(f), `${CMD} must read ${f}`);
|
||||
assert.match(text, /tolerat(e|es) (their|its) absence|never required/i);
|
||||
});
|
||||
|
||||
test('K8: the test lock is used through its CLI, and a missing CLI is said out loud', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /lib\/util\/test-lock\.mjs lock --files/);
|
||||
assert.match(text, /test-lock\.mjs unlock/);
|
||||
assert.match(text, /"not-locked"/);
|
||||
assert.match(text, /never silent/i);
|
||||
});
|
||||
|
||||
test('P9 (2): the floor guard runs on the step diff and exit 2 is never green', () => {
|
||||
const d3 = section(read(CMD), '#### Sub-step D3 — Floor guard');
|
||||
assert.match(d3, /lib\/verification\/test-integrity\.mjs --base \{start_sha\}/);
|
||||
assert.match(d3, /exit 2 is never green/i);
|
||||
assert.match(d3, /justified/);
|
||||
});
|
||||
|
||||
test('P9 (4): the done line is written by a script, only when the test command exits 0, at the step commit', () => {
|
||||
const g = section(read(CMD), '#### Sub-step G — Checkpoint, done line, unlock');
|
||||
assert.match(g, /lib\/verification\/done-line\.mjs --step \{N\}/);
|
||||
assert.match(g, /done-line\.mjs --check --step \{N\} --report \{report_path\} --commit \{hash\}/);
|
||||
assert.match(g, /writes its line only when the test command\s+exits 0/i);
|
||||
// The checkpoint comes first, so the line names the step's own commit.
|
||||
assert.ok(g.indexOf('**Checkpoint.**') < g.indexOf('**The done line**'), 'checkpoint before the done line');
|
||||
});
|
||||
|
||||
// --- the independent review of 2026-10-06 (15 findings): what the fixes must keep
|
||||
|
||||
test('review: the step test files are part of the scope and committed at the checkpoint', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(section(text, '#### Sub-step B — Test first and proof of red'), /scope = its `Files:` \+ its test files/);
|
||||
assert.match(section(text, '#### Sub-step G — Checkpoint, done line, unlock'), /its `Files:` and its test files/);
|
||||
});
|
||||
|
||||
test('review: every new file is registered (not only listed ones), so no check is blind to it', () => {
|
||||
const c = section(read(CMD), '#### Sub-step C — Implement through one fresh helper');
|
||||
assert.match(c, /git ls-files --others --exclude-standard/);
|
||||
assert.match(c, /git add --intent-to-add -- \{path\}/);
|
||||
});
|
||||
|
||||
test('review: proof of red has a control run and a real git checkout as its start point', () => {
|
||||
const b = section(read(CMD), '#### Sub-step B — Test first and proof of red');
|
||||
assert.match(b, /control run/);
|
||||
assert.match(b, /git clone --shared/);
|
||||
});
|
||||
|
||||
test('review: Phase 7.5 audits the run from session_start_sha, never a step start_sha', () => {
|
||||
const text = read(CMD);
|
||||
const audit = text.slice(text.indexOf('## Phase 7.5'), text.indexOf('### Machine-verifiable completion gate'));
|
||||
assert.match(audit, /\{session_start_sha\}\.\.HEAD/);
|
||||
assert.doesNotMatch(audit, /\{start_sha\}/);
|
||||
});
|
||||
|
||||
test('review: no sixth round, attempts never reset, a step at 5 is never built again', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(section(text, '#### Sub-step F — Fix rounds'), /no sixth round/);
|
||||
const c = section(text, '#### Sub-step C — Implement through one fresh helper');
|
||||
assert.match(c, /never\s+reset/);
|
||||
assert.match(c, /A step at `fix_rounds` 5 is never built again/);
|
||||
});
|
||||
|
||||
test('review: the helper appends its report, and passes tests as one comma-separated list', () => {
|
||||
const agent = read(AGENT);
|
||||
assert.match(agent, /APPEND it to `report_path`/);
|
||||
assert.match(agent, /ONE comma-separated argument/);
|
||||
assert.match(agent, /A step with no `Test first:`/);
|
||||
});
|
||||
|
||||
test('P9 (6): without a run-the-thing check the result is "not verified end to end", never green', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /not verified end to end/);
|
||||
assert.match(text, /end_to_end/);
|
||||
});
|
||||
|
||||
test('P9: the three new CLIs the prose names exist', () => {
|
||||
for (const f of ['red-proof.mjs', 'test-integrity.mjs', 'done-line.mjs']) {
|
||||
assert.ok(existsSync(join(ROOT, 'lib', 'verification', f)), `lib/verification/${f}`);
|
||||
}
|
||||
});
|
||||
|
||||
// --- P4: the closed stop list and the decision log ------------------------------
|
||||
|
||||
test('P4: the run stops only on the closed list of four, and the final report shows it', () => {
|
||||
const text = read(CMD);
|
||||
const list = section(text, '### Closed stop list');
|
||||
for (const cls of [/cannot be undone/i, /security-sensitive/i, /outside the working copy/i, /so broken that (every|any)thing (after it )?is guesswork/i]) {
|
||||
assert.match(list, cls);
|
||||
}
|
||||
assert.match(list, /closed/i);
|
||||
// Phase 8's report template is a fence that holds its own "## " heading, so
|
||||
// the phase is read up to Phase 9, not up to the next heading.
|
||||
const report = text.slice(text.indexOf('## Phase 8 — Final report'), text.indexOf('## Phase 9'));
|
||||
assert.match(report, /### Stops/);
|
||||
assert.match(report, /### Decisions taken/);
|
||||
});
|
||||
|
||||
test('P4: every decision the executor took is logged with what it costs if it is wrong', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /cost_if_wrong/);
|
||||
assert.match(text, /"decisions"/);
|
||||
});
|
||||
|
||||
// --- Credits: every borrowing is named, never silent ----------------------------
|
||||
|
||||
test('every borrowing in the executor is credited', () => {
|
||||
const text = read(CMD);
|
||||
const credits = section(text, '## Credits');
|
||||
for (const src of [/Superpowers/, /subagent-driven-development/, /ai-native-sdlc/, /Kiro/, /Spec Kit/, /agent-skills/, /Cherny/]) {
|
||||
assert.match(credits, src);
|
||||
}
|
||||
assert.match(credits, /MIT/);
|
||||
});
|
||||
271
tests/lib/red-proof.test.mjs
Normal file
271
tests/lib/red-proof.test.mjs
Normal file
|
|
@ -0,0 +1,271 @@
|
|||
// tests/lib/red-proof.test.mjs
|
||||
// Proof of red (P9 (1)): the step's test is run by a command against the start
|
||||
// point, where it must FAIL, and against the result, where it must PASS. The
|
||||
// model's word that "the test was red first" is never the proof.
|
||||
// Exit 0 = proven, 1 = rejected (green at the start point, or red at the result),
|
||||
// 2 = could not run (never green).
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { proveRed } from '../../lib/verification/red-proof.mjs';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = join(HERE, '..', '..');
|
||||
const CLI = join(ROOT, 'lib', 'verification', 'red-proof.mjs');
|
||||
const TEST_CMD = 'node --test tests/sum.test.mjs';
|
||||
|
||||
const SUM_TEST = [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"import { sum } from '../lib/sum.mjs';",
|
||||
"test('sum adds', () => { assert.equal(sum(1, 2), 3); });",
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
function sh(cwd, cmd, args) {
|
||||
const r = spawnSync(cmd, args, { cwd, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `${cmd} ${args.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
/** A repo whose base commit has `body` as sum(); returns {dir, base}. */
|
||||
function repo(body) {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'red-proof-'));
|
||||
sh(dir, 'git', ['init', '-q']);
|
||||
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||
mkdirSync(join(dir, 'lib'));
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), `export function sum(a, b) { return ${body}; }\n`);
|
||||
sh(dir, 'git', ['add', 'lib/sum.mjs']);
|
||||
sh(dir, 'git', ['commit', '-q', '-m', 'base']);
|
||||
return { dir, base: sh(dir, 'git', ['rev-parse', 'HEAD']) };
|
||||
}
|
||||
|
||||
function addTest(dir) {
|
||||
mkdirSync(join(dir, 'tests'), { recursive: true });
|
||||
writeFileSync(join(dir, 'tests', 'sum.test.mjs'), SUM_TEST);
|
||||
}
|
||||
|
||||
function cli(args) {
|
||||
const r = spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8' });
|
||||
return { status: r.status, stdout: r.stdout, stderr: r.stderr };
|
||||
}
|
||||
|
||||
test('red proof: red at the start point, green at the result -> proven, exit 0', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'proven', JSON.stringify(r));
|
||||
assert.notEqual(r.base.exit, 0);
|
||||
assert.equal(r.result.exit, 0);
|
||||
const c = cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs', '--json']);
|
||||
assert.equal(c.status, 0, c.stdout + c.stderr);
|
||||
assert.equal(JSON.parse(c.stdout).status, 'proven');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: a test that already PASSES at the start point is rejected, exit 1', () => {
|
||||
const { dir, base } = repo('a + b');
|
||||
try {
|
||||
addTest(dir);
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'rejected');
|
||||
assert.equal(r.reason, 'not-red-at-base');
|
||||
const c = cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']);
|
||||
assert.equal(c.status, 1, c.stdout + c.stderr);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: a test that still FAILS at the result is rejected, exit 1', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'rejected');
|
||||
assert.equal(r.reason, 'not-green-at-result');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof --red-only: the gate before building - red at the start point only, the result is not run', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir); // test written, nothing built yet: the working tree is still red
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'], redOnly: true });
|
||||
assert.equal(r.status, 'proven', JSON.stringify(r));
|
||||
assert.equal(r.result, undefined, '--red-only never runs the result');
|
||||
const c = cli(['--red-only', '--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']);
|
||||
assert.equal(c.status, 0, c.stdout + c.stderr);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
const green = repo('a + b');
|
||||
try {
|
||||
addTest(green.dir);
|
||||
const r = proveRed({ cwd: green.dir, base: green.base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'], redOnly: true });
|
||||
assert.equal(r.reason, 'not-red-at-base');
|
||||
} finally { rmSync(green.dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: what cannot be measured is "not-run", exit 2 - never proven', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
const args = (over) => ['--cwd', dir, '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs', ...over];
|
||||
// A command outside the test-runner allowlist never reaches a shell.
|
||||
assert.equal(proveRed({ cwd: dir, base, cmd: 'node tests/sum.test.mjs', tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||
assert.equal(proveRed({ cwd: dir, base, cmd: `${TEST_CMD} || true`, tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||
// An unknown start point, a missing test file, no test file at all.
|
||||
assert.equal(proveRed({ cwd: dir, base: 'no-such-ref', cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] }).status, 'not-run');
|
||||
assert.equal(proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/missing.test.mjs'] }).status, 'not-run');
|
||||
assert.equal(proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: [] }).status, 'not-run');
|
||||
assert.equal(cli(args([])).status, 0, 'control: the same arguments do prove');
|
||||
assert.equal(cli(['--cwd', dir, '--base', base, '--cmd', TEST_CMD]).status, 2, 'no --tests');
|
||||
assert.equal(cli(['--cwd', join(dir, 'nope'), '--base', base, '--cmd', TEST_CMD, '--tests', 'tests/sum.test.mjs']).status, 2);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: --cwd in a subdirectory still runs at the repo root (plan paths are root-relative)', () => {
|
||||
// Measured 2026-10-06: `git archive` run from a subdirectory packs only that
|
||||
// subdirectory, so a start point built from a subdir cwd would miss lib/.
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
mkdirSync(join(dir, 'docs'));
|
||||
const r = proveRed({ cwd: join(dir, 'docs'), base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
// Review finding 2 (2026-10-06): a start-point copy that cannot run the test at
|
||||
// all failed "red" for the wrong reason and was called proven. Three guards:
|
||||
// the copy is a real git checkout, gitignored dependency dirs are linked in, and
|
||||
// the same copy WITH the change must go green (the control run).
|
||||
|
||||
test('red proof: a gitignored dependency the test needs is linked into the start point - no false red', () => {
|
||||
const { dir, base } = repo('a + b');
|
||||
try {
|
||||
writeFileSync(join(dir, '.gitignore'), 'node_modules/\n');
|
||||
sh(dir, 'git', ['add', '.gitignore']);
|
||||
sh(dir, 'git', ['commit', '-q', '-m', 'ignore']);
|
||||
const base2 = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
mkdirSync(join(dir, 'node_modules', 'dep'), { recursive: true });
|
||||
writeFileSync(join(dir, 'node_modules', 'dep', 'index.mjs'), 'export const one = 1;\n');
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'dep.test.mjs'), [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"import { one } from '../node_modules/dep/index.mjs';",
|
||||
"import { sum } from '../lib/sum.mjs';",
|
||||
"test('dep', () => { assert.equal(sum(one, one), 2); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
// The test already passes without any change: it proves nothing.
|
||||
const r = proveRed({ cwd: dir, base: base2, cmd: 'node --test tests/dep.test.mjs', tests: ['tests/dep.test.mjs'] });
|
||||
assert.equal(r.status, 'rejected', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
assert.equal(r.reason, 'not-red-at-base');
|
||||
void base;
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: the start point is a git checkout - a git-reading command does not fail there for lack of .git', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
const r = proveRed({ cwd: dir, base, cmd: 'git status', tests: ['tests/sum.test.mjs'] });
|
||||
assert.notEqual(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: a copy that cannot go green even with the change is "not-run" (environment), never proven', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
// Red anywhere outside the real working tree, whatever the code says.
|
||||
writeFileSync(join(dir, 'tests', 'env.test.mjs'), [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"test('env', () => { assert.ok(!process.cwd().includes('voyage-red-proof-')); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
const r = proveRed({ cwd: dir, base, cmd: 'node --test tests/env.test.mjs', tests: ['tests/env.test.mjs'] });
|
||||
assert.equal(r.status, 'not-run', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
assert.match(r.reason, /control/);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: an untracked file the change adds (not a listed test) is part of the result, so the control run goes green', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
// The fix lives in a NEW untracked module the step adds.
|
||||
writeFileSync(join(dir, 'lib', 'add.mjs'), 'export const add = (a, b) => a + b;\n');
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), "export { add as sum } from './add.mjs';\n");
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: the start-point run leaves the working tree untouched', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
const before = sh(dir, 'git', ['status', '--porcelain']);
|
||||
proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(sh(dir, 'git', ['status', '--porcelain']), before);
|
||||
assert.equal(readFileSync(join(dir, 'lib', 'sum.mjs'), 'utf8'), 'export function sum(a, b) { return a + b; }\n');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
// Reuse of tests/fixtures/red-first/02243c6-always-allow-shim.mjs: the measured
|
||||
// red-first claim of 02243c6 ("6 of 55"), re-run through red-proof instead of by
|
||||
// hand. Needs this repo's git history; a `git archive` tree has none, and there
|
||||
// the test says so and skips rather than pretending to pass.
|
||||
const HAS_HISTORY = spawnSync('git', ['cat-file', '-e', '02243c6^{commit}'], { cwd: ROOT }).status === 0;
|
||||
|
||||
test('red proof: reproduces the 02243c6 red-first measurement (6 of 55) and the bare load error (1 of 1)', { skip: !HAS_HISTORY && 'no git history (git archive tree)' }, () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'red-proof-02243c6-'));
|
||||
const RUNNER = 'lib/verification/criteria-runner.mjs';
|
||||
const TEST_FILE = 'tests/lib/criteria-runner.test.mjs';
|
||||
const tar = join(dir, '..', `${dir.split('/').pop()}.tar`);
|
||||
try {
|
||||
sh(ROOT, 'git', ['archive', '--format=tar', '-o', tar, '02243c6']);
|
||||
sh(dir, 'tar', ['-xf', tar]);
|
||||
const parent = spawnSync('git', ['show', `02243c6^:${RUNNER}`], { cwd: ROOT, encoding: 'utf8' }).stdout;
|
||||
const real = readFileSync(join(dir, RUNNER), 'utf8');
|
||||
const shim = readFileSync(join(ROOT, 'tests', 'fixtures', 'red-first', '02243c6-always-allow-shim.mjs'), 'utf8');
|
||||
sh(dir, 'git', ['init', '-q']);
|
||||
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||
sh(dir, 'git', ['add', '.']);
|
||||
// Start point 1: the parent's runner as it was - the test file cannot even load.
|
||||
writeFileSync(join(dir, RUNNER), parent);
|
||||
sh(dir, 'git', ['commit', '-q', '-am', 'parent runner']);
|
||||
const bare = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
// Start point 2: the parent's runner plus the always-allow shim.
|
||||
writeFileSync(join(dir, RUNNER), parent + shim);
|
||||
sh(dir, 'git', ['commit', '-q', '-am', 'parent runner + shim']);
|
||||
const shimmed = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
writeFileSync(join(dir, RUNNER), real);
|
||||
const cmd = `node --test --test-reporter=tap ${TEST_FILE}`;
|
||||
const withShim = proveRed({ cwd: dir, base: shimmed, cmd, tests: [TEST_FILE] });
|
||||
assert.equal(withShim.status, 'proven', JSON.stringify(withShim));
|
||||
assert.deepEqual(withShim.base.tap, { tests: 55, fail: 6 });
|
||||
assert.deepEqual(withShim.result.tap, { tests: 55, fail: 0 });
|
||||
// Only the start point matters here; the result run above already showed 0 of 55.
|
||||
const bareRun = proveRed({ cwd: dir, base: bare, cmd, tests: [TEST_FILE], redOnly: true });
|
||||
assert.equal(bareRun.status, 'proven');
|
||||
assert.deepEqual(bareRun.base.tap, { tests: 1, fail: 1 }, 'the parent tree fails as ONE load error, not as named tests');
|
||||
} finally {
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
rmSync(tar, { force: true });
|
||||
}
|
||||
});
|
||||
174
tests/lib/test-integrity.test.mjs
Normal file
174
tests/lib/test-integrity.test.mjs
Normal file
|
|
@ -0,0 +1,174 @@
|
|||
// tests/lib/test-integrity.test.mjs
|
||||
// The floor guard (P9 (2)): a diff that weakens the tests is a finding, never a
|
||||
// pass. Exit 0 = clean, 1 = findings, 2 = could not run (never green).
|
||||
// Borrowed: agent-skills "floor guard" (MIT) - see lib/verification/test-integrity.mjs.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { analyzeDiff, isTestPath } from '../../lib/verification/test-integrity.mjs';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = join(HERE, '..', '..');
|
||||
const CLI = join(ROOT, 'lib', 'verification', 'test-integrity.mjs');
|
||||
const FIX = join(ROOT, 'tests', 'fixtures', 'test-integrity');
|
||||
const fixture = (name) => readFileSync(join(FIX, name), 'utf8');
|
||||
|
||||
function cli(args, opts = {}) {
|
||||
const r = spawnSync(process.execPath, [CLI, ...args], { encoding: 'utf8', ...opts });
|
||||
return { status: r.status, stdout: r.stdout, stderr: r.stderr };
|
||||
}
|
||||
|
||||
const kinds = (r) => r.findings.map((f) => f.kind);
|
||||
|
||||
test('floor guard: a clean diff (fix + a new test) is clean -> exit 0', () => {
|
||||
const r = analyzeDiff(fixture('clean.diff'));
|
||||
assert.equal(r.status, 'clean', JSON.stringify(r.findings));
|
||||
assert.deepEqual(r.findings, []);
|
||||
assert.equal(cli(['--diff', join(FIX, 'clean.diff')]).status, 0);
|
||||
});
|
||||
|
||||
test('floor guard: a diff that removes an assert -> finding removed-assert, exit 1', () => {
|
||||
const r = analyzeDiff(fixture('removed-assert.diff'));
|
||||
assert.equal(r.status, 'findings');
|
||||
assert.deepEqual(kinds(r), ['removed-assert']);
|
||||
assert.equal(r.findings[0].file, 'tests/sum.test.mjs');
|
||||
assert.equal(cli(['--diff', join(FIX, 'removed-assert.diff')]).status, 1);
|
||||
});
|
||||
|
||||
test('floor guard: a diff that skips a test -> finding skipped-test, exit 1', () => {
|
||||
const r = analyzeDiff(fixture('skipped-test.diff'));
|
||||
assert.deepEqual(kinds(r), ['skipped-test']);
|
||||
assert.equal(cli(['--diff', join(FIX, 'skipped-test.diff')]).status, 1);
|
||||
});
|
||||
|
||||
test('floor guard: a deleted test, a muted check, a stub and a loosened threshold are each found', () => {
|
||||
assert.deepEqual(kinds(analyzeDiff(fixture('deleted-test.diff'))), ['deleted-test']);
|
||||
assert.deepEqual(kinds(analyzeDiff(fixture('muted-check.diff'))), ['muted-check']);
|
||||
assert.deepEqual(kinds(analyzeDiff(fixture('stub.diff'))), ['stub']);
|
||||
assert.deepEqual(kinds(analyzeDiff(fixture('loosened-threshold.diff'))), ['loosened-threshold']);
|
||||
});
|
||||
|
||||
test('floor guard: a TIGHTENED threshold is not a finding (direction is read, not just change)', () => {
|
||||
assert.deepEqual(analyzeDiff(fixture('tightened-threshold.diff')).findings, []);
|
||||
});
|
||||
|
||||
test('floor guard: an unreadable diff -> status unreadable, exit 2 (never green)', () => {
|
||||
assert.equal(analyzeDiff(fixture('unreadable.diff')).status, 'unreadable');
|
||||
const r = cli(['--diff', join(FIX, 'unreadable.diff')]);
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
assert.equal(cli(['--diff', join(FIX, 'does-not-exist.diff')]).status, 2);
|
||||
assert.equal(cli([]).status, 2, 'no input is "could not run", not clean');
|
||||
});
|
||||
|
||||
test('floor guard: hunk line counts decide what is content - a removed "-- x" line is not a file header', () => {
|
||||
const diff = [
|
||||
'diff --git a/tests/a.test.mjs b/tests/a.test.mjs',
|
||||
'--- a/tests/a.test.mjs',
|
||||
'+++ b/tests/a.test.mjs',
|
||||
'@@ -1,4 +1,3 @@',
|
||||
" test('a', () => {",
|
||||
'--- an old comment',
|
||||
'+++ counter',
|
||||
'- assert.equal(1, 1);',
|
||||
' });',
|
||||
'\\ No newline at end of file',
|
||||
].join('\n');
|
||||
const r = analyzeDiff(diff);
|
||||
assert.deepEqual(r.findings.map((f) => [f.kind, f.file]), [['removed-assert', 'tests/a.test.mjs']]);
|
||||
});
|
||||
|
||||
test('floor guard: skipIf and node\'s lowercase t.skip() / t.todo() are skips too', () => {
|
||||
const diff = (line) => [
|
||||
'diff --git a/tests/a.test.mjs b/tests/a.test.mjs',
|
||||
'--- a/tests/a.test.mjs',
|
||||
'+++ b/tests/a.test.mjs',
|
||||
'@@ -1,1 +1,2 @@',
|
||||
" test('a', (t) => {",
|
||||
`+${line}`,
|
||||
].join('\n');
|
||||
for (const l of [" it.skipIf(process.env.CI)('x', () => {});", ' t.skip();', " t.todo('later');", "test('b', { todo: true }, () => {});"]) {
|
||||
assert.deepEqual(analyzeDiff(diff(l)).findings.map((f) => f.kind), ['skipped-test'], l);
|
||||
}
|
||||
});
|
||||
|
||||
test('floor guard: renaming a test out of the runner\'s pattern is a deleted test (--base never pairs renames)', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'floor-guard-rename-'));
|
||||
const git = (...a) => {
|
||||
const r = spawnSync('git', a, { cwd: dir, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `git ${a.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
};
|
||||
try {
|
||||
git('init', '-q');
|
||||
git('config', 'user.email', 't@example.invalid');
|
||||
git('config', 'user.name', 'test');
|
||||
writeFileSync(join(dir, 'foo.test.js'), "test('a', () => {\n assert.equal(1, 1);\n});\n");
|
||||
git('add', 'foo.test.js');
|
||||
git('commit', '-q', '-m', 'base');
|
||||
const base = git('rev-parse', 'HEAD');
|
||||
git('mv', 'foo.test.js', 'foo.test.js.bak');
|
||||
const r = cli(['--base', base, '--cwd', dir, '--json']);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.ok(JSON.parse(r.stdout).findings.some((f) => f.kind === 'deleted-test'), r.stdout);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('floor guard: prose files are not code - "not implemented" in a .md is not a stub', () => {
|
||||
const diff = [
|
||||
'diff --git a/docs/guide.md b/docs/guide.md',
|
||||
'--- a/docs/guide.md',
|
||||
'+++ b/docs/guide.md',
|
||||
'@@ -1,1 +1,2 @@',
|
||||
' # Guide',
|
||||
"+Never write `throw new Error('not implemented')` or an eslint-disable as a fix.",
|
||||
].join('\n');
|
||||
assert.deepEqual(analyzeDiff(diff).findings, []);
|
||||
});
|
||||
|
||||
test('floor guard: an empty diff is clean (nothing changed weakens nothing)', () => {
|
||||
assert.equal(analyzeDiff('').status, 'clean');
|
||||
});
|
||||
|
||||
test('floor guard: --json prints the findings with file and line', () => {
|
||||
const r = cli(['--diff', join(FIX, 'removed-assert.diff'), '--json']);
|
||||
const out = JSON.parse(r.stdout);
|
||||
assert.equal(out.status, 'findings');
|
||||
assert.equal(out.findings[0].kind, 'removed-assert');
|
||||
assert.equal(out.findings[0].file, 'tests/sum.test.mjs');
|
||||
assert.equal(typeof out.findings[0].line, 'number');
|
||||
});
|
||||
|
||||
test('floor guard: --base reads the diff from git in --cwd', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'floor-guard-'));
|
||||
const git = (...a) => {
|
||||
const r = spawnSync('git', a, { cwd: dir, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `git ${a.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
};
|
||||
try {
|
||||
git('init', '-q');
|
||||
git('config', 'user.email', 't@example.invalid');
|
||||
git('config', 'user.name', 'test');
|
||||
writeFileSync(join(dir, 'a.test.mjs'), "test('a', () => {\n assert.equal(1, 1);\n assert.equal(2, 2);\n});\n");
|
||||
git('add', 'a.test.mjs');
|
||||
git('commit', '-q', '-m', 'base');
|
||||
const base = git('rev-parse', 'HEAD');
|
||||
writeFileSync(join(dir, 'a.test.mjs'), "test('a', () => {\n assert.equal(1, 1);\n});\n");
|
||||
const r = cli(['--base', base, '--cwd', dir, '--json']);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.deepEqual(JSON.parse(r.stdout).findings.map((f) => f.kind), ['removed-assert']);
|
||||
assert.equal(cli(['--base', 'no-such-ref', '--cwd', dir]).status, 2, 'an unresolvable base is "could not run"');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('floor guard: test-path recognition covers the common layouts', () => {
|
||||
for (const p of ['tests/a.mjs', 'test/a.py', 'src/__tests__/a.ts', 'a.test.js', 'pkg/a_test.go', 'app/test_a.py', 'spec/a_spec.rb', 'a.spec.ts']) {
|
||||
assert.ok(isTestPath(p), p);
|
||||
}
|
||||
for (const p of ['lib/a.mjs', 'src/contest.ts', 'README.md']) assert.ok(!isTestPath(p), p);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue