fix(execute): the step loop survives an adversarial review
An independent review of abc3083 found 15 defects, the two worst verified
in scratch repos. Each fix started with a test that failed.
- Proof of red could be "proven" by the environment: a start point
unpacked with git archive had no node_modules and no .git, so any test
failed there. The start point is now a git clone --shared checked out at
start_sha, gitignored top-level entries are linked in, and a control run
(start point + the change) must pass, or the result is not-run.
- The Test first file was never committed (the template lists it apart
from Files:) and leaked into the next step's diff. The step's scope is
now Files: + test files, staged at the checkpoint.
- New files outside Files: were invisible to every check. Every untracked
file is registered with --intent-to-add; a revert unregisters them.
- The floor guard paired renames (foo.test.js -> .bak read as clean) and
missed skipIf and node's t.skip/t.todo; prose files no longer count as
stubs.
- Phase 7.5 audited from the last step's start_sha; it now uses the run's
session_start_sha.
- The done line is written after the checkpoint and checked with --commit.
- fix_rounds never passes 5; attempts are never reset; re-control keeps
the spec/quality verdicts and earlier floor-guard rulings; the closed
stop list maps every existing stop; failed vs stopped made consistent;
the controller's effort is said, not assumed.
Tests: 1470 (1468 pass, 0 fail, 2 skipped).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
382d5ccba2
commit
44dcc779a9
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 466 additions and 148 deletions
|
|
@ -304,7 +304,7 @@ Per step: record the start point → **one fresh helper** (`step-implementer`) w
|
|||
|------|--------------|-----|
|
||||
| Fresh helper per step | `voyage:step-implementer` (opus, effort high, no Agent tool) builds one step and writes a report: **ran · saw · did NOT check** | Each step starts from the repository and the plan, not from a long context |
|
||||
| Start point | `start_sha` recorded before the step; every check reads `git diff {start_sha}` | The review package is the diff, not a story |
|
||||
| Proof of red | the step's test must fail at the start point and pass at the result — run by `red-proof.mjs`, not claimed | A test that passes without the change proves nothing |
|
||||
| Proof of red | the step's test must fail at the start point and pass at the result, and a control run (the start point plus the change) must pass too — run by `red-proof.mjs`, not claimed | A test that passes without the change proves nothing; a start point that fails for an environment reason proves nothing either |
|
||||
| Test lock | the helper locks its test files after the red proof (`test-lock.mjs`, when installed; otherwise "not-locked" is reported) | The code moves to meet the test, never the reverse |
|
||||
| Floor guard | `test-integrity.mjs` reads the diff for weakened tests; exit 2 ("could not run") is never green | Cheating the tests is caught by a command |
|
||||
| Controller | one general helper, two verdicts (spec, quality), re-checks only what a fix round changed | An independent judge of the diff |
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@ and never touch a file the step does not list.
|
|||
|
||||
## What the executor gives you
|
||||
|
||||
- `plan` — path to the plan; `step` — the step number; the step's text verbatim
|
||||
- `plan` — path to the plan; `plan_version`; `step` — the step number; the step's text verbatim
|
||||
(Files, Changes, Reuses, Test first, Verify, and when present Interface in,
|
||||
Interface out, Review focus).
|
||||
- `start_sha` — the commit the step starts from. Everything you change is read
|
||||
|
|
@ -33,20 +33,28 @@ and never touch a file the step does not list.
|
|||
1. **Test first.** Read the step's `Test first:` and write that test (and only
|
||||
that test) first. Follow the `Pattern:` file's style when one is named.
|
||||
Then register every new file you created so the executor's diff sees it:
|
||||
`git add --intent-to-add <path>` (this stages nothing).
|
||||
`git add --intent-to-add -- <path>` (this stages nothing).
|
||||
2. **Prove it red before you build.** Run, from the repository root:
|
||||
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files>`
|
||||
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files as ONE comma-separated argument: a,b>`
|
||||
Exit 0 = the test fails at the start point: continue. Exit 1 = the test
|
||||
already passes at the start point, so it proves nothing: rewrite the test
|
||||
until it fails for the reason the step exists, and run the check again.
|
||||
Exit 2 = it could not run (the Verify command is not one test-runner
|
||||
invocation, or a file is missing): do not build; say so in the report and stop.
|
||||
invocation, or a file is missing): on `plan_version` ≥ 1.8 do not build;
|
||||
say so in the report and stop. Below 1.8, write "red proof not run:
|
||||
{reason}" in the report and go on to step 4 without a lock.
|
||||
|
||||
**A step with no `Test first:`** (allowed only below plan_version 1.8):
|
||||
skip steps 1–3, write "no Test first: no test written, no proof of red" in
|
||||
the report, and go to step 4.
|
||||
3. **Lock the tests** (when the CLI exists):
|
||||
`node {voyage_root}/lib/util/test-lock.mjs lock --files <test files> --step {step}`.
|
||||
If the file does not exist, write "not-locked (test-lock.mjs missing)" in the
|
||||
report. Never silent. Once locked you cannot edit the tests, and that is the
|
||||
point: the code changes to meet the test, never the reverse.
|
||||
4. **Implement exactly the step's `Changes:`** in the files its `Files:` lists.
|
||||
4. **Implement exactly the step's `Changes:`** in the files its `Files:` lists
|
||||
(and its test files from step 1 — nothing else; register any new file
|
||||
with `git add --intent-to-add -- <path>`).
|
||||
Read `Reuses:` first. No extra scope, no refactors on the side, no "while I
|
||||
was here". If the step cannot be done as written, stop and say why in the
|
||||
report: the executor sends it up to the plan; you do not improvise.
|
||||
|
|
@ -60,6 +68,11 @@ else. The tests stay locked: if a finding says the test itself is wrong, do not
|
|||
work around it - say so in the report; that goes up to the plan. Run the
|
||||
step's `Verify:` again.
|
||||
|
||||
The one exception: a `not-red-at-base` finding (the test passes without the
|
||||
change). The executor has released the lock for that round. Rewrite the test
|
||||
so it fails for the reason the step exists, pass the `--red-only` gate again
|
||||
(build step 2), and lock it again (build step 3).
|
||||
|
||||
## Never
|
||||
|
||||
- Never edit a test to make it pass: no deleted assert, no skip, no looser
|
||||
|
|
@ -70,7 +83,12 @@ step's `Verify:` again.
|
|||
the test command exits 0.
|
||||
- Never commit, push, merge or tag. Never write outside the repository.
|
||||
|
||||
## The report (write it to `report_path`, then answer with one line)
|
||||
## The report (APPEND it to `report_path`, then answer with one line)
|
||||
|
||||
The report file holds every round of the step and the controller's verdicts.
|
||||
Add your section at the end — with Edit, or by rewriting the file with every
|
||||
earlier section kept word for word. Never remove or rewrite what is there.
|
||||
Use the round label the executor gave you in the heading.
|
||||
|
||||
```markdown
|
||||
## Step {N} - {build | fix round R}
|
||||
|
|
|
|||
|
|
@ -1048,6 +1048,7 @@ progress file.
|
|||
"mode": "{execute | resume | step}",
|
||||
"total_steps": 0,
|
||||
"current_step": 0,
|
||||
"session_start_sha": null,
|
||||
"status": "{in_progress | completed | failed | stopped | partial}",
|
||||
"steps": {
|
||||
"1": { "status": "pending", "attempts": 0, "fix_rounds": 0, "start_sha": null, "report_path": null, "red_proof": null, "test_lock": null, "error": null, "completed_at": null, "commit": null }
|
||||
|
|
@ -1067,7 +1068,8 @@ a progress file without them still validates and resumes:
|
|||
|
||||
| Field | Written | Meaning |
|
||||
|---|---|---|
|
||||
| `steps.{N}.start_sha` | Sub-step 0 | the commit the step started from; every check reads `git diff {start_sha}` |
|
||||
| `session_start_sha` | Phase 3 | the commit the run started from; Phase 7.5 audits the whole run from it |
|
||||
| `steps.{N}.start_sha` | Sub-step 0 | the commit the step started from; every check of the step reads `git diff {start_sha}` |
|
||||
| `steps.{N}.report_path` | Sub-step 0 | the step's report file (helper rounds + the done line) |
|
||||
| `steps.{N}.red_proof` | Sub-step B | `"proven" \| "rejected" \| "not-run"` — the exit of `red-proof.mjs` |
|
||||
| `steps.{N}.test_lock` | Sub-steps B, G | `"locked" \| "released" \| "not-locked"` (K8) |
|
||||
|
|
@ -1102,6 +1104,12 @@ present in the spec at minimum — there is **no `(if wired)` conditional**.
|
|||
|
||||
### Mode-specific behavior
|
||||
|
||||
**Run start point (every mode that executes):** a fresh progress file records
|
||||
`session_start_sha` = `git rev-parse HEAD` before step 1; a resumed one keeps
|
||||
the value it has (a legacy file without it is backfilled with the parent of
|
||||
the first completed step's `commit`, or `HEAD` when none is completed). Phase
|
||||
7.5 audits the whole run from it — never from a step's own `start_sha`.
|
||||
|
||||
**mode = execute (fresh):**
|
||||
- If a progress file exists with status `in_progress`, `failed`, `stopped` or `partial`: warn that
|
||||
`--resume` is available, then wait 3 seconds (`sleep 3`) and start fresh.
|
||||
|
|
@ -1218,7 +1226,9 @@ Every stop names its class in the final report (Phase 8 `### Stops`).
|
|||
|
||||
1. **Something that cannot be undone.** A step or command whose effect cannot
|
||||
be reverted from git: deleting data outside the repo's tracked files, a
|
||||
migration against a live store, anything the plan marks irreversible.
|
||||
migration against a live store, anything the plan marks irreversible — and
|
||||
a dirty main checkout before a multi-session wave (Phase 2.55 Check 1:
|
||||
merging would lose the uncommitted work).
|
||||
2. **Something security-sensitive.** A command the executor denylist blocks
|
||||
(Phase 2.4, Sub-step D1), a write to a security-sensitive path (Hard
|
||||
rule 16), credentials or secrets in a diff.
|
||||
|
|
@ -1228,7 +1238,13 @@ Every stop names its class in the final report (Phase 8 `### Stops`).
|
|||
4. **A plan so broken that everything after it is guesswork.** The plan fails
|
||||
to parse, a ≥ 1.8 step has no `Test first:`, a step's own `On failure:
|
||||
escalate` fired (the plan's author declared that past this point is
|
||||
guesswork), or a step went up the fix ladder to the plan (Sub-step F).
|
||||
guesswork), a step went up the fix ladder (Sub-step F), the global
|
||||
iteration budget reached 0 — and the plan's own preconditions failing: an
|
||||
unmet entry condition (Phase 4), or the sandbox pre-flight exiting 77
|
||||
(Sub-step D1: the plan cannot be carried out where it runs).
|
||||
|
||||
Every stop the executor can make maps to one of these four; `stop_class` in
|
||||
the summary is that number.
|
||||
|
||||
Not on the list, and therefore never a stop: a failing test (that is a fix
|
||||
round), an ambiguity the plan leaves open (decide, and log it), a style
|
||||
|
|
@ -1311,8 +1327,11 @@ Treat this as an automatic `escalate`. Jump to the stop-and-report logic.
|
|||
The step's test is written first, and a command proves it red — never the
|
||||
model's word that "it failed first". This sub-step blocks: a step whose test
|
||||
is not proven red at its start point and green at its result does not pass.
|
||||
(A plan below 1.8 whose step has no `Test first:` skips it: `red_proof` =
|
||||
`"not-run"`, counted in the final report, never called proven.)
|
||||
|
||||
**The step's test files** are the `File:` paths under its `Test first:`. The
|
||||
plan template lists them apart from `Files:`, so everywhere below **the step's
|
||||
scope = its `Files:` + its test files**: both are built, both are committed at
|
||||
the checkpoint, and neither is a scope finding.
|
||||
|
||||
**The test command** is the step's `Verify:` command. It must be ONE
|
||||
test-runner invocation without shell operators (the allowlist of
|
||||
|
|
@ -1321,25 +1340,28 @@ with exit 2.
|
|||
|
||||
1. **The helper proves red before it builds** (Sub-step C): it writes the test
|
||||
from `Test first:`, runs the gate
|
||||
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {files}`
|
||||
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {test files, ONE comma-separated argument}`
|
||||
(the test must FAIL at the start point), locks the test files, and only then
|
||||
implements.
|
||||
2. **The executor proves it both ways after the helper returns:**
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {Test first File paths, comma-separated} --json
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {test files, ONE comma-separated argument} --json
|
||||
```
|
||||
The test must FAIL against the start point (`start_sha` unpacked with
|
||||
`git archive`, the step's test files copied in) and must PASS against the
|
||||
result (the working tree). The start point holds tracked files only: add
|
||||
`--link node_modules` (or the repo's untracked dependency directory) when
|
||||
the test needs it.
|
||||
The test must FAIL against the start point and PASS against the result
|
||||
(the working tree), and a **control run** must PASS: the start-point copy
|
||||
with the step's whole change laid on top. The start point is a temp
|
||||
`git clone --shared` checked out at `start_sha` (a real git checkout), with
|
||||
the step's test files copied in and every gitignored top-level entry of the
|
||||
working tree (`node_modules`, `.venv`, …) linked in; `--link a,b` adds
|
||||
deeper dependency paths. A start point that fails for an environment reason
|
||||
fails the control too, so it is never read as red.
|
||||
|
||||
| Exit | `steps.{N}.red_proof` | Executor |
|
||||
|------|-----------------------|----------|
|
||||
| 0 | `"proven"` | continue |
|
||||
| 1, `not-red-at-base` | `"rejected"` | the test passes without the change, so it proves nothing: a finding for Sub-step F |
|
||||
| 1, `not-red-at-base` | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again |
|
||||
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
|
||||
| 2 | `"not-run"` | never a pass and never proven. When the reason is the command (the `Verify:` is not one test-runner invocation) on a plan ≥ 1.8, the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4). Any other reason (a missing test file, an unresolvable start point) is a finding for Sub-step F |
|
||||
| 2 | `"not-run"` | never a pass and never proven. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
|
||||
|
||||
The exit code is the verdict, the output text is not. `base.tap` (when the
|
||||
runner prints TAP) says what the red was: `fail 1 of 1` on a node test file is
|
||||
|
|
@ -1363,10 +1385,15 @@ the step with the Agent tool:
|
|||
|
||||
Launch `subagent_type: "voyage:step-implementer"` with a short `description`
|
||||
("step {N}: {title}") and a prompt that carries, as files and values, not as a
|
||||
summary: the plan path, the step number, the step's text verbatim (every
|
||||
field, including `Interface in:`, `Interface out:` and `Review focus:` when
|
||||
present), `start_sha`, `report_path`, `mode: build`, and
|
||||
`voyage_root: ${CLAUDE_PLUGIN_ROOT}`.
|
||||
summary: the plan path, `plan_version`, the step number, the step's text
|
||||
verbatim (every field, including `Interface in:`, `Interface out:` and
|
||||
`Review focus:` when present), `start_sha`, `report_path`, `mode: build`, the
|
||||
round label (`build`, or `fix round R`), and `voyage_root: ${CLAUDE_PLUGIN_ROOT}`.
|
||||
|
||||
**A step at `fix_rounds` 5 is never built again** (Sub-step F): skip the
|
||||
launch and run only the checks (B, D1, D2, D3, E) on whatever the operator
|
||||
changed. A step at `fix_rounds` 4 on `--resume` was re-planned: launch a fresh
|
||||
`build` helper for the revised step.
|
||||
|
||||
- **Never pass the Agent tool's `name` parameter.** A named spawn becomes an
|
||||
`in_process_teammate`: it does the work, but its result never reaches this
|
||||
|
|
@ -1382,21 +1409,31 @@ present), `start_sha`, `report_path`, `mode: build`, and
|
|||
- **Wait for the helper to finish** before anything else; when it runs in the
|
||||
background, its completion notification is the signal. Never start the next
|
||||
sub-step or step while a helper runs.
|
||||
- `steps.{N}.attempts` = the number of helpers launched for the step (1 here).
|
||||
- `steps.{N}.attempts` += 1 for every helper launched for the step — this
|
||||
one, each fix round's, and the `build` helper after a re-plan. It is never
|
||||
reset, and every launch after a step's first also takes 1 from
|
||||
`iterations_remaining`, so the completion gate's reconciliation
|
||||
(`Σ (attempts − 1)`) holds across re-plans.
|
||||
|
||||
The helper's one-line answer is not evidence. After it returns:
|
||||
1. The report must exist at `report_path` with its `### Ran`, `### Saw` and
|
||||
`### Did NOT check` sections (a missing report is a finding for Sub-step F).
|
||||
2. Register the step's new files so the diff sees them: for each path in
|
||||
`Files:` that `git status --porcelain` shows as `??`, run
|
||||
`git add --intent-to-add {path}` (stages nothing).
|
||||
3. A changed path outside the step's `Files:` is a scope finding (Hard rule 2)
|
||||
for the controller. The executor does not revert it on its own.
|
||||
1. The report at `report_path` must carry a NEW section for this round
|
||||
(`## Step {N} - {round label}`) with its `### Ran`, `### Saw` and
|
||||
`### Did NOT check`; earlier rounds' sections and verdicts stay. A missing
|
||||
section is a finding for Sub-step F (stale sections do not count).
|
||||
2. Register EVERY new file so the diff and every check see it — not only the
|
||||
listed ones: for each path `git ls-files --others --exclude-standard`
|
||||
prints, run `git add --intent-to-add -- {path}` (stages nothing). A new file
|
||||
the step did not list is otherwise invisible to `git diff`, to the floor
|
||||
guard and to the controller. Voyage's own progress file and step reports
|
||||
are the one exception: never registered.
|
||||
3. A changed or new path outside the step's scope (`Files:` + test files) is a
|
||||
scope finding (Hard rule 2) for the controller. The executor does not
|
||||
revert it on its own.
|
||||
|
||||
**Rules the helper is held to** (its agent file carries them; the controller
|
||||
checks them in the diff): follow `Changes:` exactly — no improvising, no added
|
||||
scope; touch only the files in `Files:`; read `Reuses:` first; never weaken a
|
||||
test; never write a stub; never commit.
|
||||
scope; touch only the step's scope (`Files:` + test files); read `Reuses:`
|
||||
first; never weaken a test; never write a stub; never commit.
|
||||
|
||||
#### Sub-step D1 — Command verification
|
||||
|
||||
|
|
@ -1513,7 +1550,10 @@ every one with its ruling.
|
|||
— whatever they said — launch one controller with the Agent tool:
|
||||
`subagent_type: "general-purpose"`, `model: "opus"`, no `name` (Sub-step C),
|
||||
no `isolation`. It is a general helper with the prompt below, not a second
|
||||
Voyage agent. It reads the diff, not the helper's summary: it is never given
|
||||
Voyage agent. The Agent tool sets no reasoning effort, so the controller runs
|
||||
at the session's effort: run `/trekexecute` at effort `high` or above (the
|
||||
standard). A session below `high` is a decision-log entry ("controller ran at
|
||||
effort {level}"), never silent. It reads the diff, not the helper's summary: it is never given
|
||||
the step report, and it judges what changed, not what the helper says changed.
|
||||
|
||||
The prompt (fill in every `{…}`; pass the mechanical results verbatim as JSON):
|
||||
|
|
@ -1554,41 +1594,50 @@ parseable block is a failed control (a finding for Sub-step F), never a pass.
|
|||
Append both verdicts, the findings and `not_judged` to the step report.
|
||||
|
||||
**The round passes** only when ALL hold: `red_proof` is `"proven"` (or, below
|
||||
plan_version 1.8, `"not-run"` because the step has no `Test first:`), Verify
|
||||
PASS, manifest pass, every floor-guard finding ruled `justified` (and the guard
|
||||
plan_version 1.8, the tolerated `"not-run"` of Sub-step B), Verify PASS,
|
||||
manifest pass, every floor-guard finding ruled `justified` (and the guard
|
||||
ran), `"spec": "pass"` and `"quality": "pass"`. Then go to Sub-step G.
|
||||
Otherwise go to Sub-step F with every finding.
|
||||
|
||||
**Re-control after a fix** checks only what was fixed. A fresh controller gets
|
||||
the previous round's findings and the round's own change — before the fix
|
||||
round the executor saves `git diff {start_sha}` to
|
||||
`{report dir}/step-{N}-round-{R}.diff`, and afterwards hands over
|
||||
`diff -u` of that file against the new `git diff {start_sha}` — and answers,
|
||||
per previous finding, fixed or not, plus whether the round changed anything
|
||||
the findings did not name (that is a new finding). It does not re-judge the
|
||||
rest of the step.
|
||||
the previous round's verdicts, findings and floor-guard rulings, and the
|
||||
round's own change — before the fix round the executor saves
|
||||
`git diff {start_sha}` to `{report dir}/step-{N}-round-{R}.diff`, and afterwards
|
||||
hands over `diff -u` of that file against the new `git diff {start_sha}`. It
|
||||
answers in the SAME JSON block as above:
|
||||
- per previous finding, fixed or not — a finding not fixed stays in
|
||||
`findings`; whether the round changed anything the findings did not name
|
||||
(that is a new finding);
|
||||
- `"spec"` and `"quality"` for the step as a whole: the previous verdict
|
||||
carries over for every part the round did not touch, re-judged only where
|
||||
the round changed something;
|
||||
- `floor_guard`: a finding identical (same kind, file and text) to one the
|
||||
previous round already ruled keeps that ruling; only new floor-guard
|
||||
findings are ruled.
|
||||
It does not re-judge the rest of the step.
|
||||
|
||||
#### Sub-step F — Fix rounds
|
||||
|
||||
A failed round never loops sideways. **The fix rounds are capped at maximum 5 fix rounds per step, and they escalate upward** — to a fresh helper, then to
|
||||
the plan, then to the operator. The executor does not improvise a new design,
|
||||
and there is no ladder of cheaper models: every helper and every controller
|
||||
runs `opus` (Voyage runs no helper below Opus 5.5 High).
|
||||
runs `opus` (the step helper pins effort `high`; the controller follows the
|
||||
session's effort, Sub-step E).
|
||||
|
||||
| `fix_rounds` | Goes to | What happens |
|
||||
|---|---|---|
|
||||
| 1, 2, 3 | a fresh helper | Launch `voyage:step-implementer` again (no `name`, no `isolation`), `mode: fix`, with the findings (file, line, problem, fix) and the mechanical failures. It fixes only those; the tests stay locked. Then B (both ways), D1, D2, D3 run again and a fresh controller re-controls only what was fixed (Sub-step E). `attempts` +1; `iterations_remaining` −1 |
|
||||
| 1, 2, 3 | a fresh helper | Launch `voyage:step-implementer` again (no `name`, no `isolation`), `mode: fix`, with the findings (file, line, problem, fix) and the mechanical failures. It fixes only those; the tests stay locked (except in a `not-red-at-base` round, Sub-step B). Then B (both ways), D1, D2, D3 run again and a fresh controller re-controls only what was fixed (Sub-step E). `attempts` +1; `iterations_remaining` −1 |
|
||||
| 4 | the plan | **After three failed fix rounds the question is the architecture, not a fourth fix.** The step goes back to planning (below). Stop class 4 |
|
||||
| 5 | the operator | A step already sent to the plan (`fix_rounds` 4) that fails its control again after re-planning goes to the operator. The executor never builds this step again: on a later `--resume` it only runs the checks (B, D1, D2, D3, E) on whatever the operator changed. Stop class 4 |
|
||||
| 5 | the operator | A step already sent to the plan (`fix_rounds` 4) that fails its control again after re-planning goes to the operator. The executor never builds this step again: on a later `--resume` it only runs the checks (B, D1, D2, D3, E) on whatever the operator changed. If they fail again the step stays at 5 — there is no sixth round, `fix_rounds` never exceeds 5 — and the run stops again. Stop class 4 |
|
||||
|
||||
**Back to planning** (`fix_rounds = 4`): save `git diff {start_sha}` to
|
||||
`{report dir}/step-{N}-upward.diff` and write `{report dir}/step-{N}-upward.md`:
|
||||
the step verbatim, every round's findings and verdicts, and the question for
|
||||
the planner — which assumption in the step (its Changes, its Interface, its
|
||||
test) makes it fail. Then stop: step `failed`, run `stopped`, and tell the
|
||||
test) makes it fail. Then stop: step `failed`, run `failed`, and tell the
|
||||
operator to re-plan the step (`/trekplan` revises it) and run
|
||||
`/trekexecute --resume`, which launches a fresh `build` helper for the revised
|
||||
step and keeps `fix_rounds` at 4. A step whose proof of red cannot run on a
|
||||
step (`attempts` +1, `iterations_remaining` −1) and keeps `fix_rounds` at 4. A step whose proof of red cannot run on a
|
||||
≥ 1.8 plan because its `Verify:` is not one test-runner invocation goes here
|
||||
directly (Sub-step B).
|
||||
|
||||
|
|
@ -1600,9 +1649,13 @@ directly (Sub-step B).
|
|||
- When the step goes up (round 4 or 5), first save the upward files, then
|
||||
revert this step's changes (nothing of the step is committed yet):
|
||||
```bash
|
||||
git checkout -- {files from Files: field}
|
||||
git checkout -- {the step's tracked files changed since start_sha}
|
||||
git rm --cached -q -- {the step's new files: registered with --intent-to-add since start_sha}
|
||||
```
|
||||
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.
|
||||
The new files stay on disk, untracked, and `step-{N}-upward.md` names them —
|
||||
nothing the step made is deleted, and none of it is left registered.
|
||||
Record failure. **Do NOT proceed to next step.** Jump to Phase 7.5 (Phase 7
|
||||
runs only when every step passed).
|
||||
|
||||
**`On failure: skip`**
|
||||
- Mark step as skipped at its first failed round, without fix rounds. Continue
|
||||
|
|
@ -1624,38 +1677,46 @@ directly (Sub-step B).
|
|||
```bash
|
||||
git add {files from completed steps' Files: fields} && git commit -m "wip: trekexecute stopped at step {N} — escalation needed"
|
||||
```
|
||||
Jump to Phase 7.
|
||||
Jump to Phase 7.5 (Phase 7 runs only when every step passed).
|
||||
|
||||
The same wip commit of completed work runs whenever a step goes up (round 4
|
||||
or 5). Release the step's test lock before stopping
|
||||
(`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock`, `test_lock` =
|
||||
`"released"`), so the operator and the planner can edit the test.
|
||||
|
||||
#### Sub-step G — Done line, unlock, checkpoint
|
||||
#### Sub-step G — Checkpoint, done line, unlock
|
||||
|
||||
1. **The done line.** A step is done only when a script says so, and the script
|
||||
writes its line only when the test command exits 0:
|
||||
1. **Checkpoint.** Stage the step's scope — its `Files:` and its test files,
|
||||
and only those, never `git add -A` — then run the `Checkpoint:` git commit
|
||||
command exactly as written in the plan. (The test files are listed apart
|
||||
from `Files:` in the plan; staging only `Files:` would leave the step's
|
||||
test uncommitted and charge it to the next step's diff.)
|
||||
If the commit fails (nothing to commit, etc.): warn but do NOT fail the
|
||||
step. The step's verification already passed — the commit is bookkeeping.
|
||||
2. **The done line**, at the step's own commit. A step is done only when a
|
||||
script says so, and the script writes its line only when the test command
|
||||
exits 0:
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --step {N} --cmd "{Verify command}" --report {report_path}
|
||||
```
|
||||
Exit 0 = `DONE step {N}: …` appended to the step report. Exit 1 = the test
|
||||
command failed after all: back to Sub-step F. Exit 2 = it could not run:
|
||||
never green — on a ≥ 1.8 plan that is the plan's `Verify:` (up to the plan,
|
||||
as in Sub-step B); below 1.8, a step whose `Verify:` is not a test-runner
|
||||
command records "no done line (Verify is not a test-runner command)" in its
|
||||
report and rests on Sub-step D1's exit code. Confirm with
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path}`
|
||||
(exit 0 = present). Neither the helper nor the executor ever types a DONE
|
||||
line.
|
||||
2. **Release the test lock**:
|
||||
Exit 0 = `DONE step {N}: … exit 0 at {the checkpoint sha} …` appended to the
|
||||
step report. Confirm with
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path} --commit {hash}`
|
||||
(exit 0 = the line names this step's commit). Neither the helper nor the
|
||||
executor ever types a DONE line.
|
||||
- Exit 1 = the test went red after the checkpoint (it passed in Sub-step E
|
||||
moments before: a flaky test or an order dependency). A finding for
|
||||
Sub-step F; the fix lands as a second commit with the same message, and
|
||||
Phase 7.5's commit count reports that extra commit as drift — said, not
|
||||
hidden.
|
||||
- Exit 2 = it could not run: never green — on a ≥ 1.8 plan that is the
|
||||
plan's `Verify:` (up to the plan, as in Sub-step B); below 1.8, a step
|
||||
whose `Verify:` is not a test-runner command records "no done line
|
||||
(Verify is not a test-runner command)" in its report and rests on
|
||||
Sub-step D1's exit code.
|
||||
3. **Release the test lock**:
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs unlock` → `test_lock` =
|
||||
`"released"` (a step that was `"not-locked"` stays so).
|
||||
3. **Checkpoint.** Stage the step's `Files:` — and only those, never
|
||||
`git add -A` — then run the `Checkpoint:` git commit command exactly as
|
||||
written in the plan.
|
||||
|
||||
If the commit fails (nothing to commit, etc.): warn but do NOT fail the step.
|
||||
The step's verification already passed — the commit is bookkeeping.
|
||||
|
||||
```
|
||||
Step {N}: PASS (committed: {hash})
|
||||
|
|
@ -1821,22 +1882,23 @@ produces an independent truth based on objective state, not self-narrative.
|
|||
2. **Filesystem check:** for each expected_path, confirm the file exists.
|
||||
|
||||
3. **Commit-count check:** count commits since session-start
|
||||
(`git rev-list --count {start_sha}..HEAD`). Expected count =
|
||||
(`git rev-list --count {session_start_sha}..HEAD`). Expected count =
|
||||
number of steps with `status=completed` (excludes Step 0 if sandbox_preflight).
|
||||
|
||||
4. **Commit-message pattern sweep:** walk `git log {start_sha}..HEAD` and
|
||||
4. **Commit-message pattern sweep:** walk `git log {session_start_sha}..HEAD` and
|
||||
confirm each commit matches one of the declared
|
||||
`commit_message_patterns` (in any order — order is advisory).
|
||||
|
||||
5. **Bash syntax sweep:** for every `.sh` file in
|
||||
`git diff --name-only {start_sha} HEAD`, run `bash -n`. Collect failures.
|
||||
`git diff --name-only {session_start_sha} HEAD`, run `bash -n`. Collect failures.
|
||||
|
||||
6. **Forbidden-path sweep:** for each `scope_forbidden` entry, check
|
||||
`git diff --name-only {start_sha} HEAD -- {path}` is empty.
|
||||
`git diff --name-only {session_start_sha} HEAD -- {path}` is empty.
|
||||
|
||||
7. **Done-line sweep:** for every step with `status=completed` that carries a
|
||||
`report_path`, run
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path}`.
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/verification/done-line.mjs --check --step {N} --report {report_path} --commit {steps.N.commit}`
|
||||
(the line must name the step's own checkpoint commit).
|
||||
Exit 1 on a plan ≥ 1.8 = drift (`{check: "done-line", expected: "DONE step N", actual: "absent"}`);
|
||||
below 1.8 it is drift only when the step's report does not say why it has
|
||||
none (Sub-step G).
|
||||
|
|
@ -2213,8 +2275,8 @@ To continue (multi-session waves): re-run the same /trekexecute command (not --r
|
|||
Phase 7.6 recovery incomplete, OR the run is not verified end to end
|
||||
(Phase 7, gate clause 4)
|
||||
- `blocked` — Step 0 sandbox pre-flight exited 77; no real work attempted
|
||||
- `failed` — a step failed and On failure was revert/retry (retry cap hit)
|
||||
- `stopped` — On failure: escalate triggered
|
||||
- `failed` — a step failed and went up the fix ladder (On failure revert/retry: round 4 to the plan, or 5 to the operator)
|
||||
- `stopped` — On failure: escalate triggered, or another stop on the closed stop list (Phase 5.5)
|
||||
|
||||
**JSON summary block** (always at the end, machine-parseable):
|
||||
|
||||
|
|
|
|||
|
|
@ -9,9 +9,11 @@
|
|||
// --step, a command outside the test-runner allowlist, a command that cannot
|
||||
// start, a working tree that is not a git repo.
|
||||
//
|
||||
// `--check --step <N> --report <file>`: exit 0 when the report carries a
|
||||
// well-formed line for step N, 1 when it does not. A hand-typed "DONE" is not
|
||||
// the script's line.
|
||||
// `--check --step <N> --report <file> [--commit <sha>]`: exit 0 when the report
|
||||
// carries a well-formed line for step N (at exactly that commit, with
|
||||
// --commit), 1 when it does not. /trekexecute writes the line right after the
|
||||
// step's checkpoint and checks it with --commit, so the line is bound to the
|
||||
// commit the test passed on.
|
||||
//
|
||||
// Adapted from obra/superpowers@8ca22db (MIT, (c) 2025 Jesse Vincent),
|
||||
// skills/executing-plans/scripts/task-done: the completion line is appended
|
||||
|
|
@ -24,13 +26,20 @@ import { fileURLToPath } from 'node:url';
|
|||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { commandEnv, DEFAULT_TIMEOUT_MS, repoRoot } from './red-proof.mjs';
|
||||
|
||||
export function doneLineRe(step) {
|
||||
return new RegExp(`^DONE step ${step}: \`.+\` exit 0 at [0-9a-f]{40} \\d{4}-\\d{2}-\\d{2}T\\S+$`, 'm');
|
||||
export function doneLineRe(step, commit) {
|
||||
const sha = commit ? String(commit).replace(/[^0-9a-f]/g, '') : '[0-9a-f]{40}';
|
||||
return new RegExp(`^DONE step ${step}: \`.+\` exit 0 at ${sha} \\d{4}-\\d{2}-\\d{2}T\\S+$`, 'm');
|
||||
}
|
||||
|
||||
export function hasDoneLine(reportPath, step) {
|
||||
/**
|
||||
* With `commit` (the step's checkpoint sha, full 40 hex), the line must name
|
||||
* exactly that commit. A regex cannot tell a typed line from a written one; the
|
||||
* commit binds the line to the state the test ran on (/trekexecute writes the
|
||||
* line right after the checkpoint, so HEAD is the step's own commit).
|
||||
*/
|
||||
export function hasDoneLine(reportPath, step, commit) {
|
||||
if (!existsSync(reportPath)) return false;
|
||||
return doneLineRe(step).test(readFileSync(reportPath, 'utf8'));
|
||||
return doneLineRe(step, commit).test(readFileSync(reportPath, 'utf8'));
|
||||
}
|
||||
|
||||
/** @returns {{status: 'written'|'failed'|'not-run', reason: string, line?: string, exit?: number|null}} */
|
||||
|
|
@ -58,7 +67,7 @@ export function writeDoneLine({ cwd, step, cmd, report, timeoutMs = DEFAULT_TIME
|
|||
|
||||
const USAGE =
|
||||
'usage: done-line.mjs --step <N> --cmd "<test command>" --report <file> [--cwd <dir>] [--json]\n' +
|
||||
' done-line.mjs --check --step <N> --report <file>';
|
||||
' done-line.mjs --check --step <N> --report <file> [--commit <sha>]';
|
||||
|
||||
export function main(argv) {
|
||||
const o = {};
|
||||
|
|
@ -72,12 +81,13 @@ export function main(argv) {
|
|||
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||
else if (a === '--check') check = true;
|
||||
else if (a === '--commit' && argv[i + 1]) o.commit = argv[++i];
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`done-line: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if (check) {
|
||||
if (!/^\d+$/.test(String(o.step ?? '')) || !o.report) { process.stderr.write(`done-line: --check needs --step and --report\n${USAGE}\n`); return 2; }
|
||||
const ok = hasDoneLine(o.report, o.step);
|
||||
const ok = hasDoneLine(o.report, o.step, o.commit);
|
||||
process.stdout.write(json ? `${JSON.stringify({ step: Number(o.step), done: ok })}\n` : `step ${o.step}: ${ok ? 'DONE line present' : 'no DONE line - not done'}\n`);
|
||||
return ok ? 0 : 1;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,25 +1,38 @@
|
|||
// lib/verification/red-proof.mjs
|
||||
// Proof of red (P9 (1)). A step's test is run by a command twice:
|
||||
// Proof of red (P9 (1)). A step's test is run by a command:
|
||||
//
|
||||
// 1. against the START POINT (the step's recorded start_sha, with the step's
|
||||
// test files copied in): it must FAIL - otherwise the test proves nothing
|
||||
// about the change;
|
||||
// 2. against the RESULT (the working tree as the step left it): it must PASS.
|
||||
// 2. against the RESULT (the working tree as the step left it): it must PASS;
|
||||
// 3. CONTROL: the start-point copy with the step's whole change laid on top
|
||||
// must PASS too. A copy that cannot go green even with the change failed
|
||||
// step 1 for an environment reason (a missing dependency, a path, a tool),
|
||||
// not because the change was missing - that is "not-run", never proven.
|
||||
//
|
||||
// `--red-only` runs step 1 alone: the gate the step helper passes BEFORE it
|
||||
// builds (and locks the test). The executor's own check afterwards is the full
|
||||
// two-sided run.
|
||||
// run, and only the full run is authoritative.
|
||||
//
|
||||
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),
|
||||
// 2 = could not run (`not-run`: no test files, an unknown start point, a command
|
||||
// outside the test-runner allowlist, a command that cannot start). 2 is never a
|
||||
// pass and never "proven".
|
||||
// outside the test-runner allowlist, a command that cannot start, a control run
|
||||
// that stays red). 2 is never a pass and never "proven".
|
||||
//
|
||||
// The start point is materialized with `git archive <start_sha>` into a temp
|
||||
// directory, never by checking anything out in the working tree: the step's
|
||||
// work is not touched, and no worktree is registered in the repo. Tracked files
|
||||
// only - pass `--link node_modules` (comma list) to symlink an untracked
|
||||
// dependency directory from the working tree into the start-point copy.
|
||||
// The start point is a temp `git clone --shared` of the repo checked out at
|
||||
// start_sha - a real git checkout, so a git-reading command behaves as it does
|
||||
// in the repo, and nothing in the working tree or the repo's worktree list is
|
||||
// touched. Every gitignored top-level entry of the working tree (node_modules,
|
||||
// .venv, a build cache) is symlinked in, so the copy has the same dependencies;
|
||||
// `--link a,b` adds deeper ones. Measured 2026-10-06 (review): with a bare
|
||||
// `git archive` copy, a test that needs node_modules "proved red" with no change.
|
||||
// Known limit, failing safe: a dependency that points back INTO the working
|
||||
// tree (an editable `pip install -e .`, a workspace symlink) makes the start
|
||||
// point see the new code, and a correct step reads `not-red-at-base`.
|
||||
//
|
||||
// Everything runs at the root of --cwd's repository: plan paths and commands
|
||||
// are root-relative, and `git archive`/`clone` paths from a subdirectory are
|
||||
// not (measured 2026-10-06).
|
||||
//
|
||||
// The command must be ONE test-runner invocation without shell operators
|
||||
// (criteria-runner.mjs `allowedCommand`), and it is screened through the
|
||||
|
|
@ -27,8 +40,8 @@
|
|||
// node never passes through the Bash tool, so the hook cannot fire by itself.
|
||||
//
|
||||
// When the output is TAP (`--test-reporter=tap`, pytest-tap, ...), the counts
|
||||
// `# tests N` / `# fail N` are reported for both runs. They show WHAT the red
|
||||
// was: "fail 1 of 1" on a node test file is usually a load error (an import the
|
||||
// `# tests N` / `# fail N` are reported per run. They show WHAT the red was:
|
||||
// "fail 1 of 1" on a node test file is usually a load error (an import the
|
||||
// start point lacks), not named failing tests - see
|
||||
// tests/fixtures/red-first/02243c6-always-allow-shim.mjs, re-measured through
|
||||
// this tool in tests/lib/red-proof.test.mjs.
|
||||
|
|
@ -36,7 +49,7 @@
|
|||
// Borrowed: GitHub Spec Kit (MIT, github/spec-kit@2e941fe) - its test-first
|
||||
// rule is an instruction there; here a command proves it.
|
||||
|
||||
import { cpSync, existsSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
||||
|
|
@ -61,6 +74,11 @@ export function commandEnv(env = process.env) {
|
|||
return rest;
|
||||
}
|
||||
|
||||
export function repoRoot(dir) {
|
||||
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: dir, encoding: 'utf8' });
|
||||
return top.status === 0 ? top.stdout.trim() : null;
|
||||
}
|
||||
|
||||
function run(cmd, cwd, timeoutMs) {
|
||||
const r = spawnSync('/bin/sh', ['-c', cmd], { cwd, env: commandEnv(), encoding: 'utf8', timeout: timeoutMs, maxBuffer: 64 * 1024 * 1024 });
|
||||
const output = `${r.stdout ?? ''}${r.stderr ?? ''}`;
|
||||
|
|
@ -78,35 +96,71 @@ function unrunnable(r) {
|
|||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], timeoutMs?: number}} o
|
||||
* @returns {{status: 'proven'|'rejected'|'not-run', reason: string, base?: object, result?: object}}
|
||||
*/
|
||||
export function repoRoot(dir) {
|
||||
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: dir, encoding: 'utf8' });
|
||||
return top.status === 0 ? top.stdout.trim() : null;
|
||||
const git = (cwd, args) => spawnSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
|
||||
const lines = (s) => String(s ?? '').split('\n').filter(Boolean);
|
||||
|
||||
function safeRel(p) {
|
||||
return p && !isAbsolute(p) && !p.split('/').includes('..');
|
||||
}
|
||||
|
||||
function copyIn(from, to, rel) {
|
||||
mkdirSync(dirname(join(to, rel)), { recursive: true });
|
||||
rmSync(join(to, rel), { recursive: true, force: true });
|
||||
cpSync(join(from, rel), join(to, rel), { recursive: true, verbatimSymlinks: true });
|
||||
}
|
||||
|
||||
/** Symlink every gitignored top-level entry of the working tree, plus `extra`, into `tree`. */
|
||||
function linkDependencies(root, tree, extra) {
|
||||
const linked = [];
|
||||
const ignored = lines(git(root, ['ls-files', '--others', '--ignored', '--exclude-standard', '--directory']).stdout)
|
||||
.map((p) => p.replace(/\/$/, '').split('/')[0]);
|
||||
for (const l of new Set([...ignored, ...extra])) {
|
||||
if (!safeRel(l) || l === '.git') continue;
|
||||
if (existsSync(join(root, l)) && !existsSync(join(tree, l))) {
|
||||
mkdirSync(dirname(join(tree, l)), { recursive: true });
|
||||
symlinkSync(join(root, l), join(tree, l));
|
||||
linked.push(l);
|
||||
}
|
||||
}
|
||||
return linked;
|
||||
}
|
||||
|
||||
/** Lay the working tree's change since `base` onto `tree` (tracked edits, deletions, untracked files). */
|
||||
function applyResult(root, tree, base) {
|
||||
const status = lines(git(root, ['diff', '--name-status', '--no-renames', base]).stdout);
|
||||
for (const s of status) {
|
||||
const [code, rel] = s.split('\t');
|
||||
if (!safeRel(rel)) continue;
|
||||
if (code.startsWith('D')) rmSync(join(tree, rel), { recursive: true, force: true });
|
||||
else if (existsSync(join(root, rel)) || lstatSync(join(root, rel), { throwIfNoEntry: false })) copyIn(root, tree, rel);
|
||||
}
|
||||
for (const rel of lines(git(root, ['ls-files', '--others', '--exclude-standard']).stdout)) {
|
||||
if (safeRel(rel)) copyIn(root, tree, rel);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], redOnly?: boolean, timeoutMs?: number}} o
|
||||
* @returns {{status: 'proven'|'rejected'|'not-run', reason: string, base?: object, result?: object, control?: object}}
|
||||
*/
|
||||
export function proveRed(o) {
|
||||
const given = resolve(o.cwd ?? process.cwd());
|
||||
const timeoutMs = o.timeoutMs ?? DEFAULT_TIMEOUT_MS;
|
||||
const tests = (o.tests ?? []).filter(Boolean);
|
||||
if (!existsSync(given)) return notRun(`--cwd ${given} does not exist`);
|
||||
// Plan paths and commands are relative to the repo root, and `git archive`
|
||||
// run from a subdirectory packs only that subdirectory (measured
|
||||
// 2026-10-06). So everything below runs at the root of --cwd's repo.
|
||||
const cwd = repoRoot(given);
|
||||
if (!cwd) return notRun(`--cwd ${given} is not inside a git repository`);
|
||||
if (tests.length === 0) return notRun('no test files given: a step without its test cannot prove red');
|
||||
for (const t of tests) {
|
||||
if (isAbsolute(t) || t.split('/').includes('..')) return notRun(`test path must be repo-relative: ${t}`);
|
||||
if (!safeRel(t)) return notRun(`test path must be repo-relative: ${t}`);
|
||||
if (!existsSync(join(cwd, t))) return notRun(`test file not found in the working tree: ${t}`);
|
||||
}
|
||||
for (const l of o.link ?? []) if (!safeRel(l)) return notRun(`--link must be repo-relative: ${l}`);
|
||||
const allowed = allowedCommand(o.cmd, { cwd });
|
||||
if (!allowed.allowed) return notRun(`command not run: ${allowed.reason}`);
|
||||
const screen = screenCommand(o.cmd);
|
||||
if (!screen.allowed) return notRun(`command blocked: ${screen.rule}`);
|
||||
const sha = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${o.base}^{commit}`], { cwd, encoding: 'utf8' });
|
||||
const sha = git(cwd, ['rev-parse', '--verify', '--quiet', `${o.base}^{commit}`]);
|
||||
if (sha.status !== 0) return notRun(`cannot resolve the start point ${o.base} in ${cwd}`);
|
||||
const baseSha = sha.stdout.trim();
|
||||
|
||||
|
|
@ -122,32 +176,29 @@ export function proveRed(o) {
|
|||
const tmp = mkdtempSync(join(tmpdir(), 'voyage-red-proof-'));
|
||||
try {
|
||||
const tree = join(tmp, 'tree');
|
||||
mkdirSync(tree);
|
||||
const tar = join(tmp, 'start.tar');
|
||||
const a = spawnSync('git', ['archive', '--format=tar', '-o', tar, baseSha], { cwd, encoding: 'utf8' });
|
||||
if (a.status !== 0) return notRun(`git archive ${baseSha} failed: ${String(a.stderr).trim()}`, { result });
|
||||
const x = spawnSync('tar', ['-xf', tar, '-C', tree], { encoding: 'utf8' });
|
||||
if (x.status !== 0) return notRun(`tar could not unpack the start point: ${String(x.stderr).trim()}`, { result });
|
||||
for (const t of tests) {
|
||||
mkdirSync(dirname(join(tree, t)), { recursive: true });
|
||||
cpSync(join(cwd, t), join(tree, t));
|
||||
}
|
||||
for (const l of o.link ?? []) {
|
||||
if (!l || isAbsolute(l) || l.split('/').includes('..')) return notRun(`--link must be repo-relative: ${l}`, { result });
|
||||
if (existsSync(join(cwd, l)) && !existsSync(join(tree, l))) {
|
||||
mkdirSync(dirname(join(tree, l)), { recursive: true });
|
||||
symlinkSync(join(cwd, l), join(tree, l));
|
||||
}
|
||||
}
|
||||
const c = git(tmp, ['clone', '--quiet', '--shared', '--no-checkout', cwd, tree]);
|
||||
if (c.status !== 0) return notRun(`could not clone the start point: ${String(c.stderr).trim()}`, { result });
|
||||
const co = git(tree, ['checkout', '--quiet', '--detach', baseSha]);
|
||||
if (co.status !== 0) return notRun(`could not check out ${baseSha}: ${String(co.stderr).trim()}`, { result });
|
||||
for (const t of tests) copyIn(cwd, tree, t);
|
||||
const linked = linkDependencies(cwd, tree, o.link ?? []);
|
||||
|
||||
const base = run(o.cmd, tree, timeoutMs);
|
||||
const baseBroken = unrunnable(base);
|
||||
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result });
|
||||
|
||||
const out = { base_sha: baseSha, base, result };
|
||||
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result, linked });
|
||||
const out = { base_sha: baseSha, linked, base, result };
|
||||
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...out };
|
||||
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', base_sha: baseSha, base };
|
||||
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', base_sha: baseSha, linked, base };
|
||||
if (result.exit !== 0) return { status: 'rejected', reason: 'not-green-at-result', ...out };
|
||||
return { status: 'proven', reason: 'red at the start point, green at the result', ...out };
|
||||
|
||||
// Control: the same copy with the whole change laid on. Red here means the
|
||||
// copy itself cannot run the test green, so its red proved nothing.
|
||||
applyResult(cwd, tree, baseSha);
|
||||
const control = run(o.cmd, tree, timeoutMs);
|
||||
if (control.exit !== 0) {
|
||||
return notRun('the control run (start-point copy + the change) stayed red: the start point failed for an environment reason, not for the missing change', { ...out, control });
|
||||
}
|
||||
return { status: 'proven', reason: 'red at the start point, green at the result, green in the control', ...out, control };
|
||||
} finally {
|
||||
rmSync(tmp, { recursive: true, force: true });
|
||||
}
|
||||
|
|
@ -173,7 +224,7 @@ export function main(argv) {
|
|||
else if (a === '--red-only') o.redOnly = true;
|
||||
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`red-proof: unknown argument ${a}\n${USAGE}\n`); return 2; }
|
||||
else { process.stderr.write(`red-proof: unknown argument ${a} (--tests takes ONE comma-separated list)\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if (!o.base || !o.cmd) { process.stderr.write(`red-proof: --base and --cmd are required\n${USAGE}\n`); return 2; }
|
||||
if (o.timeoutMs !== undefined && !Number.isFinite(o.timeoutMs)) { process.stderr.write(`red-proof: --timeout must be a number\n${USAGE}\n`); return 2; }
|
||||
|
|
@ -186,6 +237,8 @@ export function main(argv) {
|
|||
process.stdout.write(`red proof: ${r.status.toUpperCase()} - ${r.reason}\n`);
|
||||
if (r.base) process.stdout.write(` start point ${r.base_sha ?? o.base}: exit ${r.base.exit}${counts(r.base)}\n`);
|
||||
if (r.result) process.stdout.write(` result: exit ${r.result.exit}${counts(r.result)}\n`);
|
||||
if (r.control) process.stdout.write(` control: exit ${r.control.exit}${counts(r.control)}\n`);
|
||||
if (r.linked?.length) process.stdout.write(` linked into the start point: ${r.linked.join(', ')}\n`);
|
||||
}
|
||||
return r.status === 'proven' ? 0 : r.status === 'rejected' ? 1 : 2;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -45,6 +45,7 @@ export function isTestPath(p) {
|
|||
}
|
||||
|
||||
const isCheckPath = (p) => isTestPath(p) || CHECK_CONFIG_RE.test(p);
|
||||
const PROSE_RE = /\.(md|markdown|mdx|txt|rst|adoc)$/i;
|
||||
|
||||
// --- the patterns -------------------------------------------------------------
|
||||
|
||||
|
|
@ -55,7 +56,7 @@ const TEST_DECL_RE =
|
|||
/^\s*(test|it|describe|suite)(\.\w+)?\s*\(|^\s*(async\s+)?def\s+test_\w*|^\s*func\s+Test\w*|^\s*#\[test\]|@Test\b/;
|
||||
|
||||
const SKIP_RE =
|
||||
/\b(test|it|describe|suite)\.(skip|todo|only)\s*\(|\bx(it|describe|test)\s*\(|\bf(it|describe)\s*\(|\{\s*(skip|todo)\s*:\s*(true|['"`])|@pytest\.mark\.(skip|skipif|xfail)\b|\bpytest\.skip\s*\(|@unittest\.skip|\bt\.Skip(Now|f)?\s*\(|#\[ignore\]|@(Disabled|Ignore)\b/;
|
||||
/\b(test|it|describe|suite)\.(skip|skipIf|todo|only)\s*\(|\bt\.(skip|todo)\s*\(|\bx(it|describe|test)\s*\(|\bf(it|describe)\s*\(|\{\s*(skip|todo)\s*:\s*(true|['"`])|@pytest\.mark\.(skip|skipif|xfail)\b|\bpytest\.skip\s*\(|@unittest\.skip|\bt\.Skip(Now|f)?\s*\(|#\[ignore\]|@(Disabled|Ignore)\b/;
|
||||
|
||||
// Inline suppressions silence a check wherever they appear.
|
||||
const SUPPRESS_RE =
|
||||
|
|
@ -183,6 +184,9 @@ export function analyzeDiff(text) {
|
|||
const removed = all.filter((l) => l.type === '-');
|
||||
const added = all.filter((l) => l.type === '+');
|
||||
const test = isTestPath(f.path);
|
||||
// Prose (docs, notes) describes code; a "not implemented" or an
|
||||
// eslint-disable written in it is text, not a stub or a muted check.
|
||||
const prose = PROSE_RE.test(f.path);
|
||||
|
||||
if (test) {
|
||||
const lostAsserts = count(removed, ASSERT_RE) - count(added, ASSERT_RE);
|
||||
|
|
@ -196,7 +200,7 @@ export function analyzeDiff(text) {
|
|||
for (const l of addedNotRemoved(removed, added, SKIP_RE)) findings.push(finding('skipped-test', f.path, l));
|
||||
}
|
||||
|
||||
for (const l of addedNotRemoved(removed, added, SUPPRESS_RE)) findings.push(finding('muted-check', f.path, l));
|
||||
if (!prose) for (const l of addedNotRemoved(removed, added, SUPPRESS_RE)) findings.push(finding('muted-check', f.path, l));
|
||||
if (isCheckPath(f.path)) {
|
||||
for (const l of addedNotRemoved(removed, added, MUTE_RE)) findings.push(finding('muted-check', f.path, l));
|
||||
for (const h of f.hunks) {
|
||||
|
|
@ -211,7 +215,7 @@ export function analyzeDiff(text) {
|
|||
}
|
||||
}
|
||||
|
||||
if (!test) for (const l of addedNotRemoved(removed, added, STUB_RE)) findings.push(finding('stub', f.path, l));
|
||||
if (!test && !prose) for (const l of addedNotRemoved(removed, added, STUB_RE)) findings.push(finding('stub', f.path, l));
|
||||
}
|
||||
return { status: findings.length ? 'findings' : 'clean', findings };
|
||||
}
|
||||
|
|
@ -227,7 +231,10 @@ function diffFromGit(cwd, base, head) {
|
|||
const okHead = spawnSync('git', ['rev-parse', '--verify', '--quiet', `${head}^{commit}`], { cwd, encoding: 'utf8' });
|
||||
if (okHead.status !== 0) return { error: `cannot resolve --head ${head} in ${cwd}` };
|
||||
}
|
||||
const args = ['diff', '--no-color', '--no-ext-diff', '--find-renames', base, ...(head ? [head] : [])];
|
||||
// --no-renames: a rename is read as a deletion plus an addition. Paired as a
|
||||
// rename, `foo.test.js -> foo.test.js.bak` showed no removed line at all and
|
||||
// switched a test off silently (review, 2026-10-06).
|
||||
const args = ['diff', '--no-color', '--no-ext-diff', '--no-renames', base, ...(head ? [head] : [])];
|
||||
const r = spawnSync('git', args, { cwd, encoding: 'utf8', maxBuffer: 256 * 1024 * 1024 });
|
||||
if (r.status !== 0) return { error: `git diff failed: ${String(r.stderr).trim()}` };
|
||||
return { text: r.stdout };
|
||||
|
|
|
|||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -22,7 +22,7 @@
|
|||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:961-976 | profile, phase_models, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2263-2285 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2325-2347 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | commands/trekreview.md (orchestrator-emit Phase 8) | trekreview.md:255 | profile, phase_models, profile_source | none |
|
||||
|
|
|
|||
|
|
@ -81,8 +81,13 @@ test('done line: --check finds the line for that step only', () => {
|
|||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'no report file = not done');
|
||||
writeFileSync(report, '# Step 3 report\nDONE step 3 (typed by hand)\n');
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 1, 'a hand-typed line is not the script line');
|
||||
// A well-formed line typed by hand, at a sha that is not the step's commit.
|
||||
writeFileSync(report, `# Step 3 report\nDONE step 3: \`node --test tests/ok.test.mjs\` exit 0 at ${'0'.repeat(40)} 2026-10-06T12:00:00.000Z\n`);
|
||||
const head = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report, '--commit', head]).status, 1, 'the line must be at the step commit');
|
||||
run(['--cwd', dir, '--step', '3', '--cmd', 'node --test tests/ok.test.mjs', '--report', report]);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report]).status, 0);
|
||||
assert.equal(run(['--check', '--step', '3', '--report', report, '--commit', head]).status, 0);
|
||||
assert.equal(run(['--check', '--step', '4', '--report', report]).status, 1);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
|
|
|||
|
|
@ -141,16 +141,61 @@ test('K8: the test lock is used through its CLI, and a missing CLI is said out l
|
|||
});
|
||||
|
||||
test('P9 (2): the floor guard runs on the step diff and exit 2 is never green', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /lib\/verification\/test-integrity\.mjs --base \{start_sha\}/);
|
||||
assert.match(text, /exit 2[^\n]*never (green|a pass)/i);
|
||||
const d3 = section(read(CMD), '#### Sub-step D3 — Floor guard');
|
||||
assert.match(d3, /lib\/verification\/test-integrity\.mjs --base \{start_sha\}/);
|
||||
assert.match(d3, /exit 2 is never green/i);
|
||||
assert.match(d3, /justified/);
|
||||
});
|
||||
|
||||
test('P9 (4): the done line is written by a script, only when the test command exits 0', () => {
|
||||
test('P9 (4): the done line is written by a script, only when the test command exits 0, at the step commit', () => {
|
||||
const g = section(read(CMD), '#### Sub-step G — Checkpoint, done line, unlock');
|
||||
assert.match(g, /lib\/verification\/done-line\.mjs --step \{N\}/);
|
||||
assert.match(g, /done-line\.mjs --check --step \{N\} --report \{report_path\} --commit \{hash\}/);
|
||||
assert.match(g, /writes its line only when the test command\s+exits 0/i);
|
||||
// The checkpoint comes first, so the line names the step's own commit.
|
||||
assert.ok(g.indexOf('**Checkpoint.**') < g.indexOf('**The done line**'), 'checkpoint before the done line');
|
||||
});
|
||||
|
||||
// --- the independent review of 2026-10-06 (15 findings): what the fixes must keep
|
||||
|
||||
test('review: the step test files are part of the scope and committed at the checkpoint', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(text, /lib\/verification\/done-line\.mjs/);
|
||||
assert.match(text, /done-line\.mjs --check/);
|
||||
assert.match(text, /only (when|if) the test command exits 0/i);
|
||||
assert.match(section(text, '#### Sub-step B — Test first and proof of red'), /scope = its `Files:` \+ its test files/);
|
||||
assert.match(section(text, '#### Sub-step G — Checkpoint, done line, unlock'), /its `Files:` and its test files/);
|
||||
});
|
||||
|
||||
test('review: every new file is registered (not only listed ones), so no check is blind to it', () => {
|
||||
const c = section(read(CMD), '#### Sub-step C — Implement through one fresh helper');
|
||||
assert.match(c, /git ls-files --others --exclude-standard/);
|
||||
assert.match(c, /git add --intent-to-add -- \{path\}/);
|
||||
});
|
||||
|
||||
test('review: proof of red has a control run and a real git checkout as its start point', () => {
|
||||
const b = section(read(CMD), '#### Sub-step B — Test first and proof of red');
|
||||
assert.match(b, /control run/);
|
||||
assert.match(b, /git clone --shared/);
|
||||
});
|
||||
|
||||
test('review: Phase 7.5 audits the run from session_start_sha, never a step start_sha', () => {
|
||||
const text = read(CMD);
|
||||
const audit = text.slice(text.indexOf('## Phase 7.5'), text.indexOf('### Machine-verifiable completion gate'));
|
||||
assert.match(audit, /\{session_start_sha\}\.\.HEAD/);
|
||||
assert.doesNotMatch(audit, /\{start_sha\}/);
|
||||
});
|
||||
|
||||
test('review: no sixth round, attempts never reset, a step at 5 is never built again', () => {
|
||||
const text = read(CMD);
|
||||
assert.match(section(text, '#### Sub-step F — Fix rounds'), /no sixth round/);
|
||||
const c = section(text, '#### Sub-step C — Implement through one fresh helper');
|
||||
assert.match(c, /never\s+reset/);
|
||||
assert.match(c, /A step at `fix_rounds` 5 is never built again/);
|
||||
});
|
||||
|
||||
test('review: the helper appends its report, and passes tests as one comma-separated list', () => {
|
||||
const agent = read(AGENT);
|
||||
assert.match(agent, /APPEND it to `report_path`/);
|
||||
assert.match(agent, /ONE comma-separated argument/);
|
||||
assert.match(agent, /A step with no `Test first:`/);
|
||||
});
|
||||
|
||||
test('P9 (6): without a run-the-thing check the result is "not verified end to end", never green', () => {
|
||||
|
|
|
|||
|
|
@ -143,6 +143,76 @@ test('red proof: --cwd in a subdirectory still runs at the repo root (plan paths
|
|||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
// Review finding 2 (2026-10-06): a start-point copy that cannot run the test at
|
||||
// all failed "red" for the wrong reason and was called proven. Three guards:
|
||||
// the copy is a real git checkout, gitignored dependency dirs are linked in, and
|
||||
// the same copy WITH the change must go green (the control run).
|
||||
|
||||
test('red proof: a gitignored dependency the test needs is linked into the start point - no false red', () => {
|
||||
const { dir, base } = repo('a + b');
|
||||
try {
|
||||
writeFileSync(join(dir, '.gitignore'), 'node_modules/\n');
|
||||
sh(dir, 'git', ['add', '.gitignore']);
|
||||
sh(dir, 'git', ['commit', '-q', '-m', 'ignore']);
|
||||
const base2 = sh(dir, 'git', ['rev-parse', 'HEAD']);
|
||||
mkdirSync(join(dir, 'node_modules', 'dep'), { recursive: true });
|
||||
writeFileSync(join(dir, 'node_modules', 'dep', 'index.mjs'), 'export const one = 1;\n');
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'dep.test.mjs'), [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"import { one } from '../node_modules/dep/index.mjs';",
|
||||
"import { sum } from '../lib/sum.mjs';",
|
||||
"test('dep', () => { assert.equal(sum(one, one), 2); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
// The test already passes without any change: it proves nothing.
|
||||
const r = proveRed({ cwd: dir, base: base2, cmd: 'node --test tests/dep.test.mjs', tests: ['tests/dep.test.mjs'] });
|
||||
assert.equal(r.status, 'rejected', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
assert.equal(r.reason, 'not-red-at-base');
|
||||
void base;
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: the start point is a git checkout - a git-reading command does not fail there for lack of .git', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
const r = proveRed({ cwd: dir, base, cmd: 'git status', tests: ['tests/sum.test.mjs'] });
|
||||
assert.notEqual(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: a copy that cannot go green even with the change is "not-run" (environment), never proven', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
// Red anywhere outside the real working tree, whatever the code says.
|
||||
writeFileSync(join(dir, 'tests', 'env.test.mjs'), [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"test('env', () => { assert.ok(!process.cwd().includes('voyage-red-proof-')); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
const r = proveRed({ cwd: dir, base, cmd: 'node --test tests/env.test.mjs', tests: ['tests/env.test.mjs'] });
|
||||
assert.equal(r.status, 'not-run', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
assert.match(r.reason, /control/);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: an untracked file the change adds (not a listed test) is part of the result, so the control run goes green', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
addTest(dir);
|
||||
// The fix lives in a NEW untracked module the step adds.
|
||||
writeFileSync(join(dir, 'lib', 'add.mjs'), 'export const add = (a, b) => a + b;\n');
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), "export { add as sum } from './add.mjs';\n");
|
||||
const r = proveRed({ cwd: dir, base, cmd: TEST_CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(r.status, 'proven', JSON.stringify({ status: r.status, reason: r.reason }));
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('red proof: the start-point run leaves the working tree untouched', () => {
|
||||
const { dir, base } = repo('a - b');
|
||||
try {
|
||||
|
|
|
|||
|
|
@ -82,6 +82,54 @@ test('floor guard: hunk line counts decide what is content - a removed "-- x" li
|
|||
assert.deepEqual(r.findings.map((f) => [f.kind, f.file]), [['removed-assert', 'tests/a.test.mjs']]);
|
||||
});
|
||||
|
||||
test('floor guard: skipIf and node\'s lowercase t.skip() / t.todo() are skips too', () => {
|
||||
const diff = (line) => [
|
||||
'diff --git a/tests/a.test.mjs b/tests/a.test.mjs',
|
||||
'--- a/tests/a.test.mjs',
|
||||
'+++ b/tests/a.test.mjs',
|
||||
'@@ -1,1 +1,2 @@',
|
||||
" test('a', (t) => {",
|
||||
`+${line}`,
|
||||
].join('\n');
|
||||
for (const l of [" it.skipIf(process.env.CI)('x', () => {});", ' t.skip();', " t.todo('later');", "test('b', { todo: true }, () => {});"]) {
|
||||
assert.deepEqual(analyzeDiff(diff(l)).findings.map((f) => f.kind), ['skipped-test'], l);
|
||||
}
|
||||
});
|
||||
|
||||
test('floor guard: renaming a test out of the runner\'s pattern is a deleted test (--base never pairs renames)', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'floor-guard-rename-'));
|
||||
const git = (...a) => {
|
||||
const r = spawnSync('git', a, { cwd: dir, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `git ${a.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
};
|
||||
try {
|
||||
git('init', '-q');
|
||||
git('config', 'user.email', 't@example.invalid');
|
||||
git('config', 'user.name', 'test');
|
||||
writeFileSync(join(dir, 'foo.test.js'), "test('a', () => {\n assert.equal(1, 1);\n});\n");
|
||||
git('add', 'foo.test.js');
|
||||
git('commit', '-q', '-m', 'base');
|
||||
const base = git('rev-parse', 'HEAD');
|
||||
git('mv', 'foo.test.js', 'foo.test.js.bak');
|
||||
const r = cli(['--base', base, '--cwd', dir, '--json']);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.ok(JSON.parse(r.stdout).findings.some((f) => f.kind === 'deleted-test'), r.stdout);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('floor guard: prose files are not code - "not implemented" in a .md is not a stub', () => {
|
||||
const diff = [
|
||||
'diff --git a/docs/guide.md b/docs/guide.md',
|
||||
'--- a/docs/guide.md',
|
||||
'+++ b/docs/guide.md',
|
||||
'@@ -1,1 +1,2 @@',
|
||||
' # Guide',
|
||||
"+Never write `throw new Error('not implemented')` or an eslint-disable as a fix.",
|
||||
].join('\n');
|
||||
assert.deepEqual(analyzeDiff(diff).findings, []);
|
||||
});
|
||||
|
||||
test('floor guard: an empty diff is clean (nothing changed weakens nothing)', () => {
|
||||
assert.equal(analyzeDiff('').status, 'clean');
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue