merge: order 10, the fix round before 6.3.0
A step of only tests can never be proven red: the plan validator warns (PLAN_STEP_TESTS_ONLY), trekplan, the template and plan-critic say the test belongs in the step that builds the behaviour, and the executor sends such a step straight up to the plan. Also: timing tests tolerate one slow run of three, the example plan uses the K11 words, trekreview knows FINDING_PROSE_LINE_OUT_OF_RANGE and checks the whole reviewer text, the criteria runner drops NODE_TEST_CONTEXT, and the resolver text says when effort is absent. Branch suite 2128 (2126/0/2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
commit
f669b8775f
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
26 changed files with 716 additions and 117 deletions
|
|
@ -136,6 +136,14 @@ questions, which defeats the purpose.
|
|||
- Are the verification commands correct and runnable?
|
||||
- `plan_version: 1.8`: does every step's `Test first:` name a test that would
|
||||
FAIL before the step and pass after it (not a test that already passes)?
|
||||
- Does any step consist of only tests — every file in its `Files:` a test —
|
||||
for behaviour an earlier step delivered? Its tests pass at its start point,
|
||||
so it can never be proven red and `/trekexecute` stops on it
|
||||
(`not-red-at-base`; the validator's warning `PLAN_STEP_TESTS_ONLY`). That is
|
||||
a **major** finding: the test for a behaviour belongs in the step that
|
||||
builds it — move it there. A test meant to pass from the start (a golden or
|
||||
characterization test) is the `Regression:` of the step that changes the
|
||||
code, not a step of its own.
|
||||
- Is there a "run the thing" step (`End-to-end:`), and does its command run
|
||||
the product the way a user does — not a unit test renamed? A plan whose only
|
||||
end-to-end check is a unit test is a **blocker**: its result could only be
|
||||
|
|
|
|||
|
|
@ -38,10 +38,14 @@ and never touch a file the step does not list.
|
|||
Then register every new file you created so the executor's diff sees it:
|
||||
`git add --intent-to-add -- <path>` (this stages nothing).
|
||||
2. **Prove it red before you build.** Run, from the repository root:
|
||||
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files as ONE comma-separated argument: a,b>`
|
||||
`node {voyage_root}/lib/verification/red-proof.mjs --red-only --base {start_sha} --cmd "<the step's Verify command>" --tests <test files as ONE comma-separated argument: a,b> --plan "{plan}" --step {step}`
|
||||
Exit 0 = the test fails at the start point: continue. Exit 1 = the test
|
||||
already passes at the start point, so it proves nothing: rewrite the test
|
||||
until it fails for the reason the step exists, and run the check again.
|
||||
But when the answer carries `tests_only: true` (it prints that line),
|
||||
every file in the step's `Files:` is a test: the behaviour already exists
|
||||
at the start point, and no test can be red here. Do not rewrite the test and do not build; say so in
|
||||
the report and stop. The executor sends the step up to the plan.
|
||||
Exit 2 = it could not run (the Verify command is not one test-runner
|
||||
invocation, or a file is missing): on `plan_version` ≥ 1.8 do not build;
|
||||
say so in the report and stop. Below 1.8, write "red proof not run:
|
||||
|
|
|
|||
|
|
@ -1459,12 +1459,14 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
|
|||
|
||||
1. **The helper proves red before it builds** (Sub-step C): it writes the test
|
||||
from `Test first:`, runs the gate
|
||||
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {test files, ONE comma-separated argument}`
|
||||
`red-proof.mjs --red-only --base {start_sha} --cmd "{Verify}" --tests {test files, ONE comma-separated argument} --plan "{plan_path}" --step {N}`
|
||||
(the test must FAIL at the start point), locks the test files, and only then
|
||||
implements.
|
||||
implements. A `tests_only: true` answer stops the helper before it builds
|
||||
(it does not rewrite the test); the executor's own run below then sends the
|
||||
step up to the plan.
|
||||
2. **The executor proves it both ways after the helper returns:**
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {test files, ONE comma-separated argument} --json
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/verification/red-proof.mjs --base {start_sha} --cmd "{Verify command}" --tests {test files, ONE comma-separated argument} --plan "{plan_path}" --step {N} --json
|
||||
```
|
||||
The test must FAIL against the start point and PASS against the result
|
||||
(the working tree), and a **control run** must PASS: the start-point copy
|
||||
|
|
@ -1478,7 +1480,8 @@ already rejects such a step (`PLAN_STEP_VERIFY_NOT_RUNNABLE`).
|
|||
| Exit | `steps.{N}.red_proof` | Executor |
|
||||
|------|-----------------------|----------|
|
||||
| 0 | `"proven"` | continue |
|
||||
| 1, `not-red-at-base` | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again |
|
||||
| 1, `not-red-at-base`, `tests_only: true` | `"rejected"` | every file in the step's `Files:` is a test (the plan validator's `PLAN_STEP_TESTS_ONLY`, the same reading): the behaviour its tests cover already exists at the start point (an earlier step built it, or it was there before the plan), so the step can never be red and no rewrite of the test can fix that. The plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4), with no fix round — the planner moves the test into the step that builds the behaviour, or makes a test meant to pass from the start the `Regression:` of the step that changes the code |
|
||||
| 1, `not-red-at-base`, otherwise | `"rejected"` | the test passes without the change, so the TEST is wrong: a finding for Sub-step F, and that fix round is the one exception to the lock — the executor releases the step's test lock for it, the fix helper rewrites the test, passes the `--red-only` gate and locks again. `tests_only: null` (the plan or the step could not be read) counts here too |
|
||||
| 1, `not-green-at-result` | `"rejected"` | the change does not make the test pass: a finding for Sub-step F |
|
||||
| 2 | `"not-run"` | never a pass and never proven. **plan_version ≥ 1.8:** when the reason is the command (the `Verify:` is not one test-runner invocation), the plan is what failed: the step goes straight up to the plan (Sub-step F, `fix_rounds = 4`, stop class 4); any other reason (a missing test file, an unresolvable start point, a control run that stays red) is a finding for Sub-step F. **Below 1.8:** a step with no `Test first:`, or whose `Verify:` is not a test-runner command, is tolerated as `"not-run"` — counted in the final report among the steps without proof of red, never called proven; any other reason is a finding |
|
||||
|
||||
|
|
@ -1823,7 +1826,7 @@ session's effort, Sub-step E).
|
|||
| `fix_rounds` | Goes to | What happens |
|
||||
|---|---|---|
|
||||
| 1, 2, 3 | a fresh helper | First `node ${CLAUDE_PLUGIN_ROOT}/lib/util/test-lock.mjs pending` (the step is no longer seen green; D1 marks it again). Launch `voyage:step-implementer` again (no `name`, no `isolation`), `mode: fix`, with the findings (file, line, problem, fix) and the mechanical failures. It fixes only those; the tests stay locked (except in a `not-red-at-base` round, Sub-step B). Then B (both ways), D1, D2, D3, D4 run again and a fresh controller re-controls only what was fixed (Sub-step E). `attempts` +1; `iterations_remaining` −1 |
|
||||
| 4 | the plan | **After three failed fix rounds the question is the architecture, not a fourth fix.** The step goes back to planning (below). Stop class 4 |
|
||||
| 4 | the plan | **After three failed fix rounds the question is the architecture, not a fourth fix.** The step goes back to planning (below). Sub-step B sends two reds here directly, with no fix round: a `"not-run"` caused by the `Verify:` command, and a `not-red-at-base` with `tests_only: true` (a step of only tests). Stop class 4 |
|
||||
| 5 | the operator | A step already sent to the plan (`fix_rounds` 4) that fails its control again after re-planning goes to the operator. The executor never builds this step again: on a later `--resume` it only runs the checks (B, D1, D2, D3, D4, E) on whatever the operator changed. If they fail again the step stays at 5 — there is no sixth round, `fix_rounds` never exceeds 5 — and the run stops again. Stop class 4 |
|
||||
|
||||
**Back to planning** (`fix_rounds = 4`): save `git diff {start_sha}` to
|
||||
|
|
@ -2922,13 +2925,14 @@ invoked in Phase 2.4 alongside the sequencing-gate brief-validator call:
|
|||
|
||||
```bash
|
||||
# v5.9 — composed phase-model resolution (brief > profile > default) for the
|
||||
# execute phase. ONE call returns {effort, model, source}; captured as
|
||||
# phase_signal_result. Append --profile {profile} when the operator passed
|
||||
# --profile.
|
||||
# execute phase. ONE call returns {model, source}, plus
|
||||
# effort only when the brief's phase signal sets one; without it the phase
|
||||
# runs at effort `standard` (no change). Captured as phase_signal_result.
|
||||
# Append --profile {profile} when the operator passed --profile.
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase execute --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||
```
|
||||
|
||||
`phase_signal_result.effort` is consumed when picking the execution
|
||||
`phase_signal_result.effort` (absent = `standard`) is consumed when picking the execution
|
||||
strategy (gates auto-escalation, parallel-wave choice — see High-effort
|
||||
behavior below). The resolver does NOT control the orchestrator's own
|
||||
model — that is fixed at invocation time (command frontmatter omits
|
||||
|
|
|
|||
|
|
@ -75,7 +75,9 @@ Parse `$ARGUMENTS` for mode flags. Order of precedence:
|
|||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
||||
|
||||
# v5.9 — composed phase-model resolution (brief > profile > default) for
|
||||
# the plan phase. ONE call returns {effort, model, source}; captured as
|
||||
# the plan phase. ONE call returns {model, source}, plus
|
||||
# effort only when the brief's phase signal sets one; without it the
|
||||
# phase runs at effort `standard` (no change). Captured as
|
||||
# phase_signal_result and injected at Agent-spawn sites below.
|
||||
# Append --profile {profile} when the operator passed --profile.
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase plan --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||
|
|
@ -676,7 +678,9 @@ Do NOT write this synthesis to disk. It is internal working context only.
|
|||
> run is `PLAN_STEP_REGRESSION_NOT_RUNNABLE`. And `On failure:` (K11) opens with
|
||||
> one of `revert`, `retry`, `skip`, `escalate` — the words `/trekexecute`
|
||||
> reads — then the specifics; any other opening is the warning
|
||||
> `PLAN_STEP_ON_FAILURE_UNKNOWN`.
|
||||
> `PLAN_STEP_ON_FAILURE_UNKNOWN`. A step whose `Files:` are all tests is the
|
||||
> warning `PLAN_STEP_TESTS_ONLY`: the test for a behaviour belongs in the step
|
||||
> that builds it (see "Steps are vertical slices" below).
|
||||
>
|
||||
> **Validator self-check (mandatory after writing `plan.md`):** run
|
||||
> `node ${CLAUDE_PLUGIN_ROOT}/lib/validators/plan-validator.mjs --strict --json {plan_path}`
|
||||
|
|
@ -712,7 +716,8 @@ Write the plan following the template structure. The plan MUST include:
|
|||
- Which existing code to reuse
|
||||
- Dependencies on other steps
|
||||
- Whether the step is based on codebase analysis or external research
|
||||
- **Test first:** — the test file and what it verifies, on EVERY step
|
||||
- **Test first:** — the test file and what it verifies, on EVERY step,
|
||||
written in the step that builds the behaviour it tests
|
||||
- **Interface in:** / **Interface out:** — what the step consumes (and which
|
||||
earlier step or existing code produces it) and what it produces for later
|
||||
steps. Write them whenever a step hands something to another step; a
|
||||
|
|
@ -731,6 +736,16 @@ Write the plan following the template structure. The plan MUST include:
|
|||
all the code". Preparatory refactoring comes first as its own steps; a wide
|
||||
change goes expand → migrate → contract.
|
||||
|
||||
**The test for a behaviour belongs in the step that builds it.** A step
|
||||
cannot consist of only tests for behaviour an earlier step delivered: at its
|
||||
start point that behaviour already exists, so its tests pass there and it
|
||||
can never be proven red — `/trekexecute` stops on it (`not-red-at-base`,
|
||||
stop class 4). The voyage-lekerepo trial (2026-10-09) lost a run that way to
|
||||
a step that only added `tests/e2e.sh` and a guard script for step 1's
|
||||
behaviour. Put such tests in the step that builds the behaviour; the
|
||||
validator warns on a step whose `Files:` are all tests
|
||||
(`PLAN_STEP_TESTS_ONLY`).
|
||||
|
||||
**The "run the thing" step.** At least one step carries
|
||||
`End-to-end: \`<command>\` → expected: <what the user sees>`: the product run
|
||||
the way a user runs it (the CLI, the server and a request, the app; for iOS,
|
||||
|
|
@ -1239,8 +1254,10 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
|||
The brief signal wins per-phase when present; the profile fills any
|
||||
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||
invoked in Phase 1; the resolved JSON `{effort, model, source}` is captured
|
||||
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
invoked in Phase 1; the resolved JSON is `{model, source}`, plus
|
||||
`effort` only when the brief's phase signal sets one (without it the
|
||||
phase runs at effort `standard`: no change). It is captured as
|
||||
`phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||
omits `model:`, so it follows the session model) and cannot be switched
|
||||
|
|
|
|||
|
|
@ -55,15 +55,19 @@ Supported flags:
|
|||
|
||||
When `{dir}/brief.md` exists, ALWAYS run the brief-validator (soft mode)
|
||||
AND the composed phase-model resolver for this command's phase before
|
||||
continuing. The resolver's JSON output `{effort, model, source}`
|
||||
(brief signal > profile > default) is captured as `phase_signal_result`
|
||||
and used at Agent-spawn sites in Phase 4 to inject the resolved model:
|
||||
continuing. The resolver's JSON output (brief signal > profile > default)
|
||||
is `{model, source}`, plus
|
||||
`effort` only when the brief's phase signal sets one (without it the
|
||||
phase runs at effort `standard`: no change). It is captured as
|
||||
`phase_signal_result` and used at Agent-spawn sites in Phase 4 to inject
|
||||
the resolved model:
|
||||
|
||||
```bash
|
||||
# When --min-brief-version was passed, append --min-version {min_brief_version}
|
||||
# so an older brief raises BRIEF_VERSION_BELOW_MINIMUM (warn, never block).
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json [--min-version {min_brief_version}] "{dir}/brief.md"
|
||||
# v5.9 — composed resolver: ONE call returns {effort, model, source}.
|
||||
# v5.9 — composed resolver: ONE call returns {model, source}, plus
|
||||
# effort only when the brief's phase signal sets one (without it: standard).
|
||||
# Append --profile {profile} when the operator passed --profile.
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model --phase research --brief-path "{dir}/brief.md" [--profile {profile}] --json
|
||||
```
|
||||
|
|
@ -881,8 +885,10 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
|||
The brief signal wins per-phase when present; the profile fills any
|
||||
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||
invoked in Phase 1; the resolved JSON `{effort, model, source}` is captured
|
||||
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
invoked in Phase 1; the resolved JSON is `{model, source}`, plus
|
||||
`effort` only when the brief's phase signal sets one (without it the
|
||||
phase runs at effort `standard`: no change). It is captured as
|
||||
`phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||
omits `model:`, so it follows the session model) and cannot be switched
|
||||
|
|
|
|||
|
|
@ -105,7 +105,9 @@ as the file is parseable:
|
|||
node ${CLAUDE_PLUGIN_ROOT}/lib/validators/brief-validator.mjs --soft --json "{brief_path}"
|
||||
|
||||
# v5.9 — composed phase-model resolution (brief > profile > default) for the
|
||||
# review phase. ONE call returns {effort, model, source}; captured as
|
||||
# review phase. ONE call returns {model, source}, plus
|
||||
# effort only when the brief's phase signal sets one; without it the phase
|
||||
# runs at effort `standard` (no change). Captured as
|
||||
# phase_signal_result and used in Phase 7 at the reviewer-launch site to
|
||||
# inject the resolved model. Append --profile {profile} when the operator
|
||||
# passed --profile.
|
||||
|
|
@ -397,11 +399,15 @@ Phase 4.5, pasted verbatim — the command, exit code and first line of output f
|
|||
every Success Criterion. It is the ONLY evidence that agent has about whether a
|
||||
criterion's verification passes, because it cannot run one.
|
||||
|
||||
Collect each reviewer's trailing JSON block and **validate it against the
|
||||
reviewer-output schema** rather than merely parsing it. Run:
|
||||
Save each reviewer's **whole answer**, verbatim — its prose and its trailing
|
||||
JSON block — to `{project_dir}/.review-out-{reviewer}.md`, not only the JSON
|
||||
block: the check reads the prose around the block too (wave 2, order 9: the
|
||||
saved file held the block alone, 76 bytes, so no prose was ever checked).
|
||||
Then **validate it against the reviewer-output schema** rather than merely
|
||||
parsing it. Run:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/findings-schema.mjs --json --repo-root "$(git rev-parse --show-toplevel)" <reviewer-output-file>
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/findings-schema.mjs --json --repo-root "$(git rev-parse --show-toplevel)" "{project_dir}/.review-out-{reviewer}.md"
|
||||
```
|
||||
|
||||
`validateReviewerOutput` in `lib/review/findings-schema.mjs` extracts the
|
||||
|
|
@ -414,6 +420,14 @@ voyage-lekerepo trial, 2026-10-06, lost its only brief-anchored MAJOR that way).
|
|||
same stable error codes (`FINDINGS_NO_JSON_BLOCK`, `FINDINGS_PARSE_ERROR`,
|
||||
`FINDING_*`).
|
||||
|
||||
`FINDING_PROSE_LINE_OUT_OF_RANGE` is the same check on a `path:line` written
|
||||
in prose. In a finding's own fields (`title`, `detail`, `recommended_action`,
|
||||
`brief_ref`) it is an **error**: re-ask that reviewer like any other schema
|
||||
failure. In the prose around the JSON block it is a **warning**: the re-ask
|
||||
asks for the block only and cannot correct that prose, so do not re-ask —
|
||||
report each such warning (reviewer, citation, the file's line count) in the
|
||||
Coverage section of `review.md`.
|
||||
|
||||
On any failure, re-ask **that reviewer** to re-emit a conforming JSON
|
||||
block only — quote the reported error codes/locations so the fix is
|
||||
targeted. **Bounded retries: N=2.** If the output still fails after 2
|
||||
|
|
@ -826,8 +840,10 @@ model_for_phase = brief.phase_signals[<phase>]?.model ?? profile.phase_models[
|
|||
The brief signal wins per-phase when present; the profile fills any
|
||||
gaps. Both fields are mechanically resolved by the single composed CLI
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/profiles/resolver.mjs --resolve-phase-model`
|
||||
invoked in Phase 2; the resolved JSON `{effort, model, source}` is captured
|
||||
as `phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
invoked in Phase 2; the resolved JSON is `{model, source}`, plus
|
||||
`effort` only when the brief's phase signal sets one (without it the
|
||||
phase runs at effort `standard`: no change). It is captured as
|
||||
`phase_signal_result` and passed to `Agent` tool calls explicitly. The
|
||||
resolver controls the `model` parameter at Agent-spawn sites only — the
|
||||
orchestrator's own model is fixed at invocation time (command frontmatter
|
||||
omits `model:`, so it follows the session model) and cannot be switched
|
||||
|
|
|
|||
|
|
@ -269,6 +269,7 @@ prose form (relaxed in S26 so the template's prose emission parses).
|
|||
7. **Every plan_version, no version bump** (a plan without the field validates unchanged; voyage-lekerepo trial 2, 2026-10-08):
|
||||
- **K10 — `Regression:`** (optional step field): ONE backticked command the criteria runner's allowlist runs, with no pipe and no `&&` (first backticked span, as for `Verify:`). The executor runs it green before the step's checkpoint; a step without it runs nothing new. `Verify:` stays the step's own red-first test. One that can never run is `PLAN_STEP_REGRESSION_NOT_RUNNABLE` (an error; a warning under `--soft`).
|
||||
- **K11 — `On failure:`** opens with one of `revert`, `retry`, `skip`, `escalate` (case-insensitive; a leading `**`, `_`, backtick or `-` is ignored, other leading characters are not; `onFailureAction` in `lib/parsers/plan-schema.mjs`), the words `/trekexecute` reads; any other opening is the warning `PLAN_STEP_ON_FAILURE_UNKNOWN`. A missing `On failure:` is unchanged (the executor defaults to `escalate`).
|
||||
- **A step of only tests** (order 10, voyage-lekerepo trial 3, 2026-10-09): when every file in a step's `Files:` is a test (its `Test first:` file, or a path `lib/util/test-path.mjs` reads as a test: under `tests/`, `test/`, `__tests__/`, `spec/`, `specs/`, or named `*.test.*`, `*.spec.*`, `*_test.*`, `*_spec.*`, `test_*.py`, but never a document such as `docs/specs/api.md`; `stepTestsOnly` in `lib/parsers/plan-schema.mjs`), the warning is `PLAN_STEP_TESTS_ONLY`: the behaviour its tests cover exists at its start point, so it can never be proven red. A step without `Files:` is not judged. At run time `red-proof.mjs --plan <plan.md> --step <N>` puts the same reading on a `not-red-at-base` as `tests_only`, and `/trekexecute` sends a `tests_only: true` step straight up to the plan.
|
||||
|
||||
**Validation strategy:**
|
||||
|
||||
|
|
@ -300,6 +301,7 @@ The strongest validation in the entire pipeline. Phase 5.5 (planning-orchestrato
|
|||
| `PLAN_ARCH_RULE_NO_ENFORCER` / `PLAN_ARCH_RULE_NO_PROOF` | 1.8 architectural: a rule without an enforcer, or a checked rule without an existing proving step | Name the enforcer; name the step |
|
||||
| `PLAN_STEP_REGRESSION_NOT_RUNNABLE` | K10, any version: a step's `Regression:` is not one command the allowlist runs (a pipe, `&&`, no backticks, an unknown runner) | One backticked runner call (`npm test`); a wider check goes in `bash tests/<script>.sh` |
|
||||
| `PLAN_STEP_ON_FAILURE_UNKNOWN` | K11, any version, warning: `On failure:` does not open with revert / retry / skip / escalate | Open with the word, then the specifics |
|
||||
| `PLAN_STEP_TESTS_ONLY` | Any version, warning: every file in the step's `Files:` is a test, so the step can never be proven red | Move the test into the step that builds the behaviour |
|
||||
|
||||
**The plan gate (K3).** `node lib/validators/plan-gate.mjs <plan.md> [--json]` decides whether a plan may go on: **0** = it validates strict, its `## Critic Verdict` matches the current text, no BLOCKER is open, and every open MAJOR was accepted by the operator; **1** = stop, with `reasons` (`PLAN_GATE_INVALID_PLAN`, `PLAN_GATE_VERSION` — below plan_version 1.8, `PLAN_GATE_NO_VERDICT`, `PLAN_GATE_VERDICT_MALFORMED`, `PLAN_GATE_VERDICT_STALE`, `PLAN_GATE_OPEN_BLOCKER`, `PLAN_GATE_ROUNDS_EXHAUSTED`, `PLAN_GATE_OPEN_MAJOR` — rounds left: revise, `PLAN_GATE_OPEN_MAJOR_FINAL` — after the last round: ask the operator) and `warnings` (`PLAN_GATE_OPEN_MAJOR` for majors the operator accepted); **2** = could not read the plan. `--hash <plan.md>` prints the operative hash; Phase 9 takes it before the reviewers read the plan. `--record <plan.md> --reviewed <hash>` reads plan-review-dedup output on stdin (which lists the reviewers that delivered, `sources`) and writes the next round; exit 2 when a reviewer is missing or the plan changed since `--hash`; it refuses round 4 (exit 1, stop and ask) unless `--extra-round` follows the operator's yes, and refuses that flag before the cap is spent. Unrated findings count as blockers. The review log is excluded from the hash only up to its first other heading, so a step placed under it is hashed. `--accept-open <plan.md>` records the operator's "build as is" for the current round's open MAJORs (`accepted_open` in the verdict block, bound to the round and the plan hash; only after the last round; never for a BLOCKER; exit 1 when rounds are left, the verdict is stale, a blocker is open or no major is). Operator decision 2026-10-08: open majors used to be warnings, and a weakness the critics named was built and came back from `/trekreview` as MAJOR. plan-critic and scope-guardian rate on the review catalogue's tiers. The verdict is a trace of the review, not a signature.
|
||||
|
||||
|
|
|
|||
|
|
@ -91,7 +91,7 @@ or stability harness comes before behavior change.
|
|||
- **Changes:** Three subtests, one per golden file. Each runs `node src/cli.mjs <cmd> ...` and asserts stdout `===` `readFileSync('tests/golden/<cmd>.stdout')`. The test should PASS today (no behavior change yet) — it's the canary for step 5 onwards.
|
||||
- **Verify:** `node --test tests/cli-no-verbose-stability.test.mjs` → expected: 3 pass
|
||||
- **Checkpoint:** `git commit -m "test(small-auth): stdout stability harness for verbose-flag work"`
|
||||
- **On failure:** if subtests fail, the goldens are wrong — re-run step 1.
|
||||
- **On failure:** escalate — if subtests fail, the goldens are wrong: re-run step 1.
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
|
|
@ -133,7 +133,7 @@ or stability harness comes before behavior change.
|
|||
- **Changes:** At `src/cli.mjs:62` (ctx construction) add `verbose: globalFlags.verbose` to the ctx literal. No handler changes yet.
|
||||
- **Verify:** `node --test tests/cli-no-verbose-stability.test.mjs` → expected: 3 pass (handlers ignore the new field for now)
|
||||
- **Checkpoint:** `git commit -m "feat(cli): thread verbose into command handler ctx"`
|
||||
- **On failure:** stability tests fail → ctx mutation broke something. Bisect by reverting and adding back one line at a time.
|
||||
- **On failure:** revert — stability tests fail → ctx mutation broke something. Bisect by reverting and adding back one line at a time.
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
|
|
@ -150,16 +150,17 @@ or stability harness comes before behavior change.
|
|||
|
||||
### Step 5: Wire verbose output in `login`, `token-refresh`, `users-list`, `users-create`, `logout`
|
||||
|
||||
- **Files:** `src/commands/login.mjs`, `src/commands/token-refresh.mjs`, `src/commands/users-list.mjs`, `src/commands/users-create.mjs`, `src/commands/logout.mjs`
|
||||
- **Changes:** At each internal step (3 for login, 4 for token-refresh, 2 for users-list, 3 for users-create, 2 for logout — 14 call sites total), add `if (ctx.verbose) ctx.stderr.write(\`[verbose] <step description>\\n\`);`. Step descriptions per file:
|
||||
- **Files:** `src/commands/login.mjs`, `src/commands/token-refresh.mjs`, `src/commands/users-list.mjs`, `src/commands/users-create.mjs`, `src/commands/logout.mjs`, `tests/cli-verbose-flag.test.mjs` (new file)
|
||||
- **Changes:** First the test, `tests/cli-verbose-flag.test.mjs`: spawn `node src/cli.mjs login --verbose alice`, capture stderr, assert exit 0, assert stderr contains all three expected verbose lines: "[verbose] parsing argv", "[verbose] credential lookup", "[verbose] issuing session token", in that order. It fails before the wiring below (no verbose lines yet): the test for a behaviour lands in the step that builds it. Then, at each internal step (3 for login, 4 for token-refresh, 2 for users-list, 3 for users-create, 2 for logout — 14 call sites total), add `if (ctx.verbose) ctx.stderr.write(\`[verbose] <step description>\\n\`);`. Step descriptions per file:
|
||||
- login: "parsing argv", "credential lookup", "issuing session token"
|
||||
- token-refresh: "parsing argv", "validating refresh token", "rotating session token", "persisting new token"
|
||||
- users-list: "parsing argv", "querying user store"
|
||||
- users-create: "parsing argv", "validating input", "writing user record"
|
||||
- logout: "parsing argv", "invalidating session token"
|
||||
- **Verify:** `node --test tests/cli-no-verbose-stability.test.mjs` → expected: 3 pass (stdout unchanged when flag absent)
|
||||
- **Verify:** `node --test tests/cli-verbose-flag.test.mjs` → expected: 1 pass
|
||||
- **Regression:** `node --test tests/cli-no-verbose-stability.test.mjs` → expected: 3 pass (stdout unchanged when flag absent)
|
||||
- **Checkpoint:** `git commit -m "feat(commands): emit verbose stderr trace for 5 commands"`
|
||||
- **On failure:** stability tests fail → likely a stray `console.log` or `ctx.stdout.write` instead of `ctx.stderr.write`. Re-grep all five files for `stdout` mentions added in this step.
|
||||
- **On failure:** retry — stability tests fail → likely a stray `console.log` or `ctx.stdout.write` instead of `ctx.stderr.write`. Re-grep all five files for `stdout` mentions added in this step. If the verbose test misses a line, check the `[verbose]` strings for typos; if its exit code != 0, check that login still works without verbose.
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
|
|
@ -169,7 +170,8 @@ or stability harness comes before behavior change.
|
|||
- src/commands/users-list.mjs
|
||||
- src/commands/users-create.mjs
|
||||
- src/commands/logout.mjs
|
||||
min_file_count: 5
|
||||
- tests/cli-verbose-flag.test.mjs
|
||||
min_file_count: 6
|
||||
commit_message_pattern: "^feat\\(commands\\): emit verbose"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
|
|
@ -178,30 +180,11 @@ or stability harness comes before behavior change.
|
|||
pattern: "ctx\\.verbose"
|
||||
- path: src/commands/token-refresh.mjs
|
||||
pattern: "ctx\\.verbose"
|
||||
```
|
||||
|
||||
### Step 6: Add verbose-content test for `login`
|
||||
|
||||
- **Files:** `tests/cli-verbose-flag.test.mjs` (new file)
|
||||
- **Changes:** Single test: spawn `node src/cli.mjs login --verbose alice`, capture stderr, assert exit 0, assert stderr contains all three expected verbose lines: "[verbose] parsing argv", "[verbose] credential lookup", "[verbose] issuing session token", in that order.
|
||||
- **Verify:** `node --test tests/cli-verbose-flag.test.mjs` → expected: 1 pass
|
||||
- **Checkpoint:** `git commit -m "test(small-auth): assert --verbose emits expected stderr trace"`
|
||||
- **On failure:** if assertion misses a line, check step 5 for typos in the `[verbose]` strings; if exit code != 0, check that login still works without verbose (regression).
|
||||
- **Manifest:**
|
||||
```yaml
|
||||
manifest:
|
||||
expected_paths:
|
||||
- tests/cli-verbose-flag.test.mjs
|
||||
min_file_count: 1
|
||||
commit_message_pattern: "^test\\(small-auth\\): assert --verbose"
|
||||
bash_syntax_check: []
|
||||
forbidden_paths: []
|
||||
must_contain:
|
||||
- path: tests/cli-verbose-flag.test.mjs
|
||||
pattern: "\\[verbose\\] credential lookup"
|
||||
```
|
||||
|
||||
### Step 7: Update `--help` text
|
||||
### Step 6: Update `--help` text
|
||||
|
||||
- **Files:** `src/cli.mjs`
|
||||
- **Changes:** At the help-text constant (`src/cli.mjs:78`), add a line under "Global flags": ` -v, --verbose emit per-step trace to stderr (single level only)`.
|
||||
|
|
@ -224,7 +207,7 @@ or stability harness comes before behavior change.
|
|||
|
||||
## Verification
|
||||
|
||||
Final acceptance run after step 7:
|
||||
Final acceptance run after step 6:
|
||||
|
||||
```bash
|
||||
node --test tests/ # all 26 tests pass (24 + 2 new)
|
||||
|
|
@ -239,13 +222,14 @@ node src/cli.mjs --help | grep -c "\-v, --verbose" # 1
|
|||
|
||||
- No deferred decisions: every step names its files, lines, and exact
|
||||
string changes.
|
||||
- TDD: stability harness (step 2) precedes behavior changes (steps 3-5).
|
||||
- TDD: stability harness (step 2) precedes behavior changes (steps 3-5),
|
||||
and the verbose-content test lands in step 5, with the code it tests.
|
||||
- Verify commands are runnable, not "test it works".
|
||||
- Steps 5 wires 5 files in one commit; this is over the 1–2 file
|
||||
- Step 5 wires 5 files (and their test) in one commit; this is over the 1–2 file
|
||||
guideline but is justified by symmetry — the change is mechanical
|
||||
and atomic across the five files; splitting would create five tiny
|
||||
commits with no test value between them.
|
||||
|
||||
## Execution Strategy
|
||||
|
||||
Single session, 7 steps, ~15-20 minutes. No parallel decomposition needed.
|
||||
Single session, 6 steps, ~15-20 minutes. No parallel decomposition needed.
|
||||
|
|
|
|||
|
|
@ -9,8 +9,8 @@
|
|||
"started_at": "2026-05-01T10:14:32Z",
|
||||
"updated_at": "2026-05-01T10:31:08Z",
|
||||
"status": "completed",
|
||||
"current_step": 7,
|
||||
"total_steps": 7,
|
||||
"current_step": 6,
|
||||
"total_steps": 6,
|
||||
"steps": [
|
||||
{
|
||||
"n": 1,
|
||||
|
|
@ -67,31 +67,20 @@
|
|||
"title": "Wire verbose output in login, token-refresh, users-list, users-create, logout",
|
||||
"status": "completed",
|
||||
"started_at": "2026-05-01T10:22:13Z",
|
||||
"completed_at": "2026-05-01T10:27:34Z",
|
||||
"completed_at": "2026-05-01T10:29:51Z",
|
||||
"commit_sha": "a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4",
|
||||
"files_changed": [
|
||||
"src/commands/login.mjs",
|
||||
"src/commands/token-refresh.mjs",
|
||||
"src/commands/users-list.mjs",
|
||||
"src/commands/users-create.mjs",
|
||||
"src/commands/logout.mjs"
|
||||
],
|
||||
"verify_passed": true
|
||||
},
|
||||
{
|
||||
"n": 6,
|
||||
"title": "Add verbose-content test for login",
|
||||
"status": "completed",
|
||||
"started_at": "2026-05-01T10:27:34Z",
|
||||
"completed_at": "2026-05-01T10:29:51Z",
|
||||
"commit_sha": "b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5",
|
||||
"files_changed": [
|
||||
"src/commands/logout.mjs",
|
||||
"tests/cli-verbose-flag.test.mjs"
|
||||
],
|
||||
"verify_passed": true
|
||||
},
|
||||
{
|
||||
"n": 7,
|
||||
"n": 6,
|
||||
"title": "Update --help text",
|
||||
"status": "completed",
|
||||
"started_at": "2026-05-01T10:29:51Z",
|
||||
|
|
|
|||
|
|
@ -12,6 +12,7 @@
|
|||
|
||||
import { ok, fail, issue } from '../util/result.mjs';
|
||||
import { parseDocument } from '../util/frontmatter.mjs';
|
||||
import { isTestPath } from '../util/test-path.mjs';
|
||||
|
||||
export const STEP_HEADING_REGEX = /^### Step (\d+):\s+(.+?)\s*$/m;
|
||||
export const STEP_HEADING_GLOBAL = /^### Step (\d+):\s+(.+?)\s*$/gm;
|
||||
|
|
@ -281,6 +282,52 @@ export function onFailureAction(text) {
|
|||
return m && ON_FAILURE_ACTIONS.includes(m[1]) ? m[1] : null;
|
||||
}
|
||||
|
||||
// ---- a step that holds nothing but tests -----------------------------------
|
||||
|
||||
/**
|
||||
* The paths a field names. Each entry (comma- or line-separated, a sub-bullet's
|
||||
* `-` dropped) gives its backticked spans, or else its first word: a plain
|
||||
* path beside a backticked one is still a path (independent review, order 10).
|
||||
*/
|
||||
function fieldPaths(text) {
|
||||
const out = [];
|
||||
for (const raw of String(text ?? '').split(/[,\n]/)) {
|
||||
const entry = raw.trim().replace(/^[-*]\s+/, '');
|
||||
if (entry === '') continue;
|
||||
const spans = [...entry.matchAll(/`([^`\n]+)`/g)].map((m) => m[1].trim());
|
||||
if (spans.length > 0) out.push(...spans);
|
||||
else out.push(entry.split(/\s+/)[0]);
|
||||
}
|
||||
return out.filter(Boolean);
|
||||
}
|
||||
|
||||
// A document is never the test here: `docs/specs/api.md` sits under a `specs/`
|
||||
// folder, which lib/util/test-path.mjs reads as tests for the floor guard.
|
||||
const DOC_RE = /\.(md|markdown|mdx|txt|rst|adoc)$/i;
|
||||
|
||||
/**
|
||||
* Whether every file in a step's `Files:` is a test: the step's own `Test
|
||||
* first:` file, or a path that is clearly a test (lib/util/test-path.mjs). Such
|
||||
* a step can never be proven red - the behaviour its tests cover already
|
||||
* exists at its start point (an earlier step built it, or it was there before
|
||||
* the plan), so they already pass there (voyage-lekerepo
|
||||
* trial 3, 2026-10-09: `not-red-at-base`, stop class 4). The plan validator
|
||||
* warns on it (PLAN_STEP_TESTS_ONLY) and red-proof.mjs reports it
|
||||
* (`tests_only`), so the executor sends the step up to the plan. A step
|
||||
* without `Files:` is not such a step.
|
||||
* @param {string} body - the step's section (sliceSteps)
|
||||
* @returns {{testsOnly: boolean, files: string[]}}
|
||||
*/
|
||||
export function stepTestsOnly(body) {
|
||||
const f = parseStepFields(body);
|
||||
const files = f.has('Files') ? fieldPaths(f.get('Files').text) : [];
|
||||
const testFirst = f.has('Test first')
|
||||
? [...f.get('Test first').text.matchAll(/\bFile:\**[ \t]*`([^`\n]+)`/gi)].map((m) => m[1].trim())
|
||||
: [];
|
||||
const isTest = (p) => testFirst.includes(p) || (isTestPath(p) && !DOC_RE.test(p));
|
||||
return { testsOnly: files.length > 0 && files.every(isTest), files };
|
||||
}
|
||||
|
||||
// ---- sections (fence-aware) ------------------------------------------------
|
||||
|
||||
/**
|
||||
|
|
|
|||
12
lib/util/command-env.mjs
Normal file
12
lib/util/command-env.mjs
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
// lib/util/command-env.mjs
|
||||
// The environment a check command runs in. The command's exit code must be its
|
||||
// own: under `node --test` a child inherits NODE_TEST_CONTEXT, and a nested
|
||||
// `node --test` then reports to that parent and exits 0 whatever its tests did
|
||||
// (measured 2026-10-06: a red start point read as green). So the variable
|
||||
// never reaches the command. Used by red-proof.mjs, regression-check.mjs and
|
||||
// the criteria runner's default exec.
|
||||
|
||||
export function commandEnv(env = process.env) {
|
||||
const { NODE_TEST_CONTEXT, ...rest } = env;
|
||||
return rest;
|
||||
}
|
||||
15
lib/util/test-path.mjs
Normal file
15
lib/util/test-path.mjs
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
// lib/util/test-path.mjs
|
||||
// What counts as a test file, by its path. One reading for every caller: the
|
||||
// floor guard (test-integrity.mjs) reads a diff with it, and the plan reader
|
||||
// (plan-schema.mjs `stepTestsOnly`) asks whether a step holds nothing but tests.
|
||||
|
||||
const TEST_PATH_RE = [
|
||||
/(^|\/)(tests?|__tests__|spec|specs)\//,
|
||||
/\.(test|spec)\.[a-z0-9]+$/i,
|
||||
/_(test|spec)\.[a-z0-9]+$/i,
|
||||
/(^|\/)test_[^/]*\.py$/,
|
||||
];
|
||||
|
||||
export function isTestPath(p) {
|
||||
return TEST_PATH_RE.some((re) => re.test(p));
|
||||
}
|
||||
|
|
@ -30,6 +30,9 @@
|
|||
// can never run is an error (a warning under --soft).
|
||||
// - K11: `On failure:` opens with revert | retry | skip | escalate, the words
|
||||
// /trekexecute reads; any other opening is a warning.
|
||||
// - A step whose `Files:` are all tests (plan-schema `stepTestsOnly`) is the
|
||||
// warning PLAN_STEP_TESTS_ONLY: its tests pass at its start point, so it can
|
||||
// never be proven red (voyage-lekerepo trial 3, 2026-10-09).
|
||||
|
||||
import { readFileSync, existsSync, realpathSync } from 'node:fs';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
|
@ -37,7 +40,7 @@ import {
|
|||
sliceSteps, validatePlanHeadings, extractPlanVersion, extractPlanCeremony,
|
||||
parseStepFields, findSection, splitSections, findModuleCycle, parseMermaid, fencedBlocks,
|
||||
versionAtLeast, PLAN_VERSION_CURRENT, PLAN_VERSIONS_SUPPORTED, CEREMONY_VALUES,
|
||||
ON_FAILURE_ACTIONS, onFailureAction,
|
||||
ON_FAILURE_ACTIONS, onFailureAction, stepTestsOnly,
|
||||
} from '../parsers/plan-schema.mjs';
|
||||
import { validateAllManifests } from '../parsers/manifest-yaml.mjs';
|
||||
import { parseDocument } from '../util/frontmatter.mjs';
|
||||
|
|
@ -160,6 +163,15 @@ function checkRecovery(sections, opts) {
|
|||
`Step ${s.n}: \`On failure:\` does not open with a recovery word the executor reads`,
|
||||
`Open with one of ${ON_FAILURE_ACTIONS.join(' | ')}, then the specifics (\`retry once, then escalate\`).`));
|
||||
}
|
||||
if (stepTestsOnly(s.body).testsOnly) {
|
||||
warnings.push(issue('PLAN_STEP_TESTS_ONLY',
|
||||
`Step ${s.n}: every file in \`Files:\` is a test; a step of only tests can never be proven red`,
|
||||
'The test for a behaviour belongs in the step that builds it: at this step\'s start point the behaviour ' +
|
||||
'already exists (an earlier step built it, or it was there before the plan), so its test passes there and ' +
|
||||
'red proof rejects it (`not-red-at-base`). Move the test into the step that builds the behaviour; a test ' +
|
||||
'meant to pass from the start (a golden or characterization test) is the `Regression:` of the step that ' +
|
||||
'changes the code, not a step of its own.'));
|
||||
}
|
||||
}
|
||||
return { errors, warnings };
|
||||
}
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@ import { resolve, dirname, join } from 'node:path';
|
|||
import { fileURLToPath } from 'node:url';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { sliceSteps, parseStepFields } from '../parsers/plan-schema.mjs';
|
||||
import { commandEnv } from '../util/command-env.mjs';
|
||||
|
||||
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
|
||||
|
|
@ -359,8 +360,11 @@ function screenShape(command, opts, writeTime) {
|
|||
|
||||
// --- running ----------------------------------------------------------------
|
||||
|
||||
// Without NODE_TEST_CONTEXT (lib/util/command-env.mjs): started under
|
||||
// `node --test`, a nested `node --test` criterion reported to the parent and
|
||||
// exited 0 on a red test - a false green (order 10, 2026-10-09).
|
||||
function defaultExec(command, { cwd, timeoutMs }) {
|
||||
const r = spawnSync('/bin/sh', ['-c', command], { cwd, encoding: 'utf8', timeout: timeoutMs });
|
||||
const r = spawnSync('/bin/sh', ['-c', command], { cwd, encoding: 'utf8', timeout: timeoutMs, env: commandEnv() });
|
||||
return { status: r.status, stdout: r.stdout ?? '', stderr: r.stderr ?? '' };
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -14,6 +14,10 @@
|
|||
// builds (and locks the test). The executor's own check afterwards is the full
|
||||
// run, and only the full run is authoritative.
|
||||
//
|
||||
// `--plan <plan.md> --step <N>` adds `tests_only` to a `not-red-at-base`: true
|
||||
// when every file in the step's `Files:` is a test, so the step can never be
|
||||
// red and goes up to the plan (see `testsOnly` below).
|
||||
//
|
||||
// Exit 0 = proven, 1 = rejected (`not-red-at-base` or `not-green-at-result`),
|
||||
// 2 = could not run (`not-run`: no test files, an unknown start point, a command
|
||||
// outside the test-runner allowlist, a command that cannot start, a control run
|
||||
|
|
@ -49,12 +53,14 @@
|
|||
// Borrowed: GitHub Spec Kit (MIT, github/spec-kit@2e941fe) - its test-first
|
||||
// rule is an instruction there; here a command proves it.
|
||||
|
||||
import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { cpSync, existsSync, lstatSync, mkdirSync, mkdtempSync, readFileSync, realpathSync, rmSync, symlinkSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, isAbsolute, join, resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { allowedCommand, screenCommand } from './criteria-runner.mjs';
|
||||
import { commandEnv } from '../util/command-env.mjs';
|
||||
import { sliceSteps, stepTestsOnly } from '../parsers/plan-schema.mjs';
|
||||
|
||||
export const DEFAULT_TIMEOUT_MS = 900_000;
|
||||
const TAIL = 2000;
|
||||
|
|
@ -65,14 +71,9 @@ function tap(output) {
|
|||
return tests && fail ? { tests: Number(tests[1]), fail: Number(fail[1]) } : null;
|
||||
}
|
||||
|
||||
// The command's exit code must be its own. Under `node --test` a child inherits
|
||||
// NODE_TEST_CONTEXT, and a nested `node --test` then reports to that parent and
|
||||
// exits 0 whatever its tests did (measured 2026-10-06: a red start point read as
|
||||
// green). So the variable never reaches the command.
|
||||
export function commandEnv(env = process.env) {
|
||||
const { NODE_TEST_CONTEXT, ...rest } = env;
|
||||
return rest;
|
||||
}
|
||||
// The command's exit code must be its own: NODE_TEST_CONTEXT never reaches it
|
||||
// (lib/util/command-env.mjs).
|
||||
export { commandEnv };
|
||||
|
||||
export function repoRoot(dir) {
|
||||
const top = spawnSync('git', ['rev-parse', '--show-toplevel'], { cwd: dir, encoding: 'utf8' });
|
||||
|
|
@ -100,6 +101,23 @@ const redKind = (base) => (base.loadError ? 'load-error' : 'test-failure');
|
|||
|
||||
const notRun = (reason, extra = {}) => ({ status: 'not-run', reason, ...extra });
|
||||
|
||||
// With `--plan` and `--step`, a `not-red-at-base` also says whether the step
|
||||
// holds nothing but tests (plan-schema `stepTestsOnly`, the plan validator's
|
||||
// PLAN_STEP_TESTS_ONLY). Such a step can never be red: the behaviour its tests
|
||||
// cover already exists at its start point. Rewriting the test cannot fix that, so
|
||||
// the executor sends the step up to the plan instead of into fix rounds
|
||||
// (voyage-lekerepo trial 3, 2026-10-09). `null` = the plan or the step could
|
||||
// not be read; without `--plan` the field is absent and nothing new happens.
|
||||
function testsOnly(o) {
|
||||
if (!o.plan) return {};
|
||||
let text;
|
||||
try { text = readFileSync(o.plan, 'utf8'); }
|
||||
catch (e) { return { tests_only: null, tests_only_note: `cannot read the plan: ${e.message}` }; }
|
||||
const step = sliceSteps(text).find((s) => Number(s.n) === Number(o.step));
|
||||
if (!step) return { tests_only: null, tests_only_note: `no "### Step ${o.step}:" in the plan` };
|
||||
return { tests_only: stepTestsOnly(step.body).testsOnly };
|
||||
}
|
||||
|
||||
// A shell that could not find or execute the command (126/127), a signal (the
|
||||
// timeout), or a spawn error is not a test verdict at all.
|
||||
function unrunnable(r) {
|
||||
|
|
@ -153,7 +171,7 @@ function applyResult(root, tree, base) {
|
|||
}
|
||||
|
||||
/**
|
||||
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], redOnly?: boolean, timeoutMs?: number}} o
|
||||
* @param {{cwd: string, base: string, cmd: string, tests: string[], link?: string[], redOnly?: boolean, timeoutMs?: number, plan?: string, step?: number}} o
|
||||
* @returns {{status: 'proven'|'rejected'|'not-run', reason: string, base?: object, result?: object, control?: object}}
|
||||
*/
|
||||
export function proveRed(o) {
|
||||
|
|
@ -200,7 +218,7 @@ export function proveRed(o) {
|
|||
const baseBroken = unrunnable(base);
|
||||
if (baseBroken) return notRun(`at the start point: ${baseBroken}`, { base, result, linked });
|
||||
const out = { base_sha: baseSha, linked, base, result };
|
||||
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...out };
|
||||
if (base.exit === 0) return { status: 'rejected', reason: 'not-red-at-base', ...testsOnly(o), ...out };
|
||||
if (o.redOnly) return { status: 'proven', reason: 'red at the start point (--red-only: the result was not run)', red_kind: redKind(base), base_sha: baseSha, linked, base };
|
||||
if (result.exit !== 0) return { status: 'rejected', reason: 'not-green-at-result', ...out };
|
||||
|
||||
|
|
@ -220,7 +238,7 @@ export function proveRed(o) {
|
|||
// --- CLI ------------------------------------------------------------------------
|
||||
|
||||
const USAGE =
|
||||
'usage: red-proof.mjs --base <start_sha> --cmd "<test command>" --tests <a,b> [--red-only] [--cwd <dir>] [--link <dir,...>] [--timeout <ms>] [--json]';
|
||||
'usage: red-proof.mjs --base <start_sha> --cmd "<test command>" --tests <a,b> [--plan <plan.md> --step <N>] [--red-only] [--cwd <dir>] [--link <dir,...>] [--timeout <ms>] [--json]';
|
||||
|
||||
const list = (v) => String(v).split(',').map((s) => s.trim()).filter(Boolean);
|
||||
|
||||
|
|
@ -234,12 +252,16 @@ export function main(argv) {
|
|||
else if (a === '--tests' && argv[i + 1]) o.tests = list(argv[++i]);
|
||||
else if (a === '--cwd' && argv[i + 1]) o.cwd = resolve(argv[++i]);
|
||||
else if (a === '--link' && argv[i + 1]) o.link = list(argv[++i]);
|
||||
else if (a === '--plan' && argv[i + 1]) o.plan = resolve(argv[++i]);
|
||||
else if (a === '--step' && argv[i + 1]) o.step = Number(argv[++i]);
|
||||
else if (a === '--red-only') o.redOnly = true;
|
||||
else if (a === '--timeout' && argv[i + 1]) o.timeoutMs = Number(argv[++i]);
|
||||
else if (a === '--json') json = true;
|
||||
else { process.stderr.write(`red-proof: unknown argument ${a} (--tests takes ONE comma-separated list)\n${USAGE}\n`); return 2; }
|
||||
}
|
||||
if (!o.base || !o.cmd) { process.stderr.write(`red-proof: --base and --cmd are required\n${USAGE}\n`); return 2; }
|
||||
if (o.plan && !(Number.isInteger(o.step) && o.step > 0)) { process.stderr.write(`red-proof: --plan needs --step <N>\n${USAGE}\n`); return 2; }
|
||||
if (o.step !== undefined && !o.plan) { process.stderr.write(`red-proof: --step needs --plan <plan.md>\n${USAGE}\n`); return 2; }
|
||||
if (o.timeoutMs !== undefined && !Number.isFinite(o.timeoutMs)) { process.stderr.write(`red-proof: --timeout must be a number\n${USAGE}\n`); return 2; }
|
||||
let r;
|
||||
try { r = proveRed(o); }
|
||||
|
|
@ -248,6 +270,7 @@ export function main(argv) {
|
|||
else {
|
||||
const counts = (x) => (x?.tap ? ` (tap: ${x.tap.fail} of ${x.tap.tests} failed)` : '');
|
||||
process.stdout.write(`red proof: ${r.status.toUpperCase()} - ${r.reason}${r.red_kind ? ` (red: ${r.red_kind})` : ''}\n`);
|
||||
if (r.tests_only === true) process.stdout.write(' tests_only: true - every file in the step\'s Files: is a test, so it can never be red: up to the plan\n');
|
||||
if (r.base) process.stdout.write(` start point ${r.base_sha ?? o.base}: exit ${r.base.exit}${counts(r.base)}\n`);
|
||||
if (r.result) process.stdout.write(` result: exit ${r.result.exit}${counts(r.result)}\n`);
|
||||
if (r.control) process.stdout.write(` control: exit ${r.control.exit}${counts(r.control)}\n`);
|
||||
|
|
|
|||
|
|
@ -27,23 +27,16 @@ import { readFileSync, realpathSync } from 'node:fs';
|
|||
import { spawnSync } from 'node:child_process';
|
||||
import { resolve } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { isTestPath } from '../util/test-path.mjs';
|
||||
|
||||
export { isTestPath };
|
||||
|
||||
// --- what counts as a test / a check config ----------------------------------
|
||||
|
||||
const TEST_PATH_RE = [
|
||||
/(^|\/)(tests?|__tests__|spec|specs)\//,
|
||||
/\.(test|spec)\.[a-z0-9]+$/i,
|
||||
/_(test|spec)\.[a-z0-9]+$/i,
|
||||
/(^|\/)test_[^/]*\.py$/,
|
||||
];
|
||||
// A test path is lib/util/test-path.mjs (shared with the plan reader).
|
||||
|
||||
const CHECK_CONFIG_RE =
|
||||
/(^|\/)(\.github|\.forgejo|\.gitea|\.woodpecker)\/|(^|\/)(\.gitlab-ci\.yml|Makefile|package\.json|pyproject\.toml|setup\.cfg|tox\.ini|\.pre-commit-config\.yaml|tsconfig[^/]*\.json|\.eslintrc[^/]*|eslint\.config\.[a-z]+|(jest|vitest|playwright)\.config\.[a-z]+|\.c8rc[^/]*|\.nycrc[^/]*|codecov\.ya?ml)$/;
|
||||
|
||||
export function isTestPath(p) {
|
||||
return TEST_PATH_RE.some((re) => re.test(p));
|
||||
}
|
||||
|
||||
const isCheckPath = (p) => isTestPath(p) || CHECK_CONFIG_RE.test(p);
|
||||
const PROSE_RE = /\.(md|markdown|mdx|txt|rst|adoc)$/i;
|
||||
|
||||
|
|
|
|||
|
|
@ -72,7 +72,10 @@ Each step targets 1–2 files and one focused change, and is a vertical slice:
|
|||
a narrow but complete path through every layer it touches, verifiable on its
|
||||
own. Preparatory refactoring comes first, as its own steps. Every step has
|
||||
`Test first:` (plan_version 1.8 requires it), and one step runs the thing
|
||||
end to end (`End-to-end:`).
|
||||
end to end (`End-to-end:`). The test for a behaviour belongs in the step that
|
||||
builds it: a step of only tests for behaviour an earlier step delivered passes
|
||||
at its start point and can never be proven red (the validator warns:
|
||||
`PLAN_STEP_TESTS_ONLY`).
|
||||
|
||||
Optional per-step fields: `Interface in:` (what the step consumes, and which
|
||||
earlier step or existing code produces it), `Interface out:` (what it produces
|
||||
|
|
|
|||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -22,7 +22,7 @@
|
|||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1130-1145 | profile, phase_models, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | lib/stats/stats-append.mjs (trekresearch.md Stats tracking) | trekresearch.md:782-808 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | lib/stats/stats-append.mjs (trekplan.md Phase 12) | trekplan.md:1115-1135 | profile, phase_models, parallel_agents, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2855-2877 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2858-2880 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:628-632 | profile, phase_models, profile_source | none |
|
||||
|
|
|
|||
|
|
@ -1,11 +1,20 @@
|
|||
// hook-helper.mjs — Shared test helper for hook scripts.
|
||||
// Spawns a hook as a child process and feeds it JSON via stdin.
|
||||
//
|
||||
// Source: ../../../llm-security/tests/hooks/hook-helper.mjs (verbatim copy)
|
||||
// Source: ../../../llm-security/tests/hooks/hook-helper.mjs (copied; since order 10
|
||||
// the kill limit is HOOK_TIMEOUT_MS and a call can set its own)
|
||||
// Provenance: borrowed within the same marketplace (same author, MIT).
|
||||
|
||||
import { execFile } from 'node:child_process';
|
||||
|
||||
// How long a hook may run before the helper kills it. The tests check a
|
||||
// hook's answer, not its speed (SC #14 times itself, tests/helpers/timing.mjs),
|
||||
// so the limit only has to catch a hook that hangs. 5 s killed P7
|
||||
// (post-edit-check) at load 95-177 in four wave sessions while it was 8/8
|
||||
// green alone (order 10, 2026-10-09). hooks/hooks.json sets no timeout for
|
||||
// these hooks, so 5 s was stricter than anything the hook meets in use.
|
||||
export const HOOK_TIMEOUT_MS = 30_000;
|
||||
|
||||
/**
|
||||
* Run a hook script by spawning `node <scriptPath>` and piping `input` to stdin.
|
||||
*
|
||||
|
|
@ -23,15 +32,16 @@ export function runHook(scriptPath, input) {
|
|||
* @param {string} scriptPath - Absolute path to the hook .mjs file
|
||||
* @param {object|string} input - JSON payload (object will be stringified)
|
||||
* @param {Record<string, string>} envOverrides - Extra env vars to set
|
||||
* @param {{ timeoutMs?: number }} [opts] - kill limit for this call (default HOOK_TIMEOUT_MS)
|
||||
* @returns {Promise<{ code: number, stdout: string, stderr: string }>}
|
||||
*/
|
||||
export function runHookWithEnv(scriptPath, input, envOverrides) {
|
||||
export function runHookWithEnv(scriptPath, input, envOverrides, opts = {}) {
|
||||
return new Promise((resolve) => {
|
||||
const env = { ...process.env, ...envOverrides };
|
||||
const child = execFile(
|
||||
'node',
|
||||
[scriptPath],
|
||||
{ timeout: 5000, env },
|
||||
{ timeout: opts.timeoutMs ?? HOOK_TIMEOUT_MS, env },
|
||||
(err, stdout, stderr) => {
|
||||
resolve({
|
||||
code: child.exitCode ?? (err && err.code === 'ERR_CHILD_PROCESS_STDIO_FINAL' ? 0 : 1),
|
||||
|
|
|
|||
25
tests/helpers/timing.mjs
Normal file
25
tests/helpers/timing.mjs
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
// tests/helpers/timing.mjs
|
||||
// A timing assertion that tolerates a busy machine without loosening its limit.
|
||||
//
|
||||
// Load only ever ADDS time to a run, so the fastest of a few runs is the best
|
||||
// reading of what the code itself costs. `fastestWithin` measures up to
|
||||
// `tries` runs and passes as soon as one is under the limit. A real slowness
|
||||
// is slow every time and still fails; one run slowed by four other test suites
|
||||
// does not (measured S123 and in wave 2, 2026-10-08: SC #14 red under load
|
||||
// 95-177, green alone).
|
||||
|
||||
/**
|
||||
* @param {number} limitMs - the limit; a run must be strictly under it
|
||||
* @param {() => Promise<number>} measure - runs once, resolves to its elapsed ms
|
||||
* @param {number} [tries=3]
|
||||
* @returns {Promise<{ok: boolean, fastest: number, runs: number[]}>}
|
||||
*/
|
||||
export async function fastestWithin(limitMs, measure, tries = 3) {
|
||||
const runs = [];
|
||||
for (let i = 0; i < tries; i++) {
|
||||
runs.push(await measure());
|
||||
if (runs[runs.length - 1] < limitMs) break;
|
||||
}
|
||||
const fastest = Math.min(...runs);
|
||||
return { ok: fastest < limitMs, fastest, runs };
|
||||
}
|
||||
|
|
@ -27,12 +27,16 @@ gitIn(REPO, 'checkout', '-q', '-b', 'feat');
|
|||
const sha = (s) => createHash('sha256').update(s).digest('hex');
|
||||
|
||||
let callNo = 0;
|
||||
// Approval records live in one temp dir per user (hooks/lib/approval-record.mjs),
|
||||
// keyed by session_id + tool_use_id, so two suites running at once (two
|
||||
// worktrees) took each other's records with fixed ids. Unique per run.
|
||||
const RUN = `${process.pid}-${Date.now()}`;
|
||||
|
||||
// A Bash call goes through the same two hooks as in a session: the guard asks
|
||||
// (and records what it asked), the call runs, then the approval is logged.
|
||||
async function post(root, input, env = {}) {
|
||||
const logFile = join(tmp(), 'stub.jsonl');
|
||||
const ids = { session_id: 'sess-log', tool_use_id: `toolu_log_${(callNo += 1)}` };
|
||||
const ids = { session_id: `sess-log-${RUN}`, tool_use_id: `toolu_log_${(callNo += 1)}` };
|
||||
if (input.tool_name === 'Bash') {
|
||||
await runHookWithEnv(join(root, 'hooks', 'scripts', 'pre-bash-executor.mjs'),
|
||||
{ hook_event_name: 'PreToolUse', cwd: REPO, ...ids, tool_name: 'Bash', tool_input: input.tool_input }, env);
|
||||
|
|
|
|||
|
|
@ -16,6 +16,9 @@ const PRE_WRITE = join(ROOT, 'hooks', 'scripts', 'pre-write-executor.mjs');
|
|||
const STOP = join(ROOT, 'hooks', 'scripts', 'stop-test-lock.mjs');
|
||||
const LOCK = join(ROOT, 'lib', 'util', 'test-lock.mjs');
|
||||
const STEP = join(ROOT, 'lib', 'verification', 'step-check.mjs');
|
||||
// Approval records share one temp dir per user, keyed by session_id +
|
||||
// tool_use_id: fixed ids let two suites running at once take each other's.
|
||||
const RUN = `${process.pid}-${Date.now()}`;
|
||||
const { classifyGitApproval } = await import(join(ROOT, 'hooks', 'lib', 'git-approval.mjs'));
|
||||
|
||||
const FEAT = makeRepo('main');
|
||||
|
|
@ -81,7 +84,7 @@ test('review 8: the log records what the person was asked, not a re-reading afte
|
|||
const repo = makeRepo('main');
|
||||
gitIn(repo, 'checkout', '-q', '-b', 'feat');
|
||||
const cmd = 'git checkout main && git merge feat';
|
||||
const base = { session_id: 's-8', tool_use_id: 'toolu_8', cwd: repo, tool_name: 'Bash', tool_input: { command: cmd } };
|
||||
const base = { session_id: `s-8-${RUN}`, tool_use_id: 'toolu_8', cwd: repo, tool_name: 'Bash', tool_input: { command: cmd } };
|
||||
const pre = await runHook(join(plugin, 'hooks', 'scripts', 'pre-bash-executor.mjs'), { ...base, hook_event_name: 'PreToolUse' });
|
||||
assert.match(pre.stdout, /merge into main/);
|
||||
gitIn(repo, 'checkout', '-q', 'main'); // the command ran
|
||||
|
|
@ -97,7 +100,7 @@ test('review 8: a command nobody was asked about is not logged, even if it reads
|
|||
const repo = makeRepo('main');
|
||||
const log = join(tmp(), 'stub.jsonl');
|
||||
await runHookWithEnv(join(plugin, 'hooks', 'scripts', 'post-approval-log.mjs'), {
|
||||
hook_event_name: 'PostToolUse', session_id: 's-9', tool_use_id: 'toolu_never_asked', cwd: repo,
|
||||
hook_event_name: 'PostToolUse', session_id: `s-9-${RUN}`, tool_use_id: 'toolu_never_asked', cwd: repo,
|
||||
tool_name: 'Bash', tool_input: { command: 'git push origin main' },
|
||||
}, { STUB_LOG: log });
|
||||
assert.equal(readStubLog(log).length, 0);
|
||||
|
|
@ -110,7 +113,7 @@ test('review 8: a command nobody was asked about is not logged, even if it reads
|
|||
test('uncertain (no-prompt mode): the push to main is refused, and no approval is logged', async () => {
|
||||
const plugin = pluginCopy({ 'lib/util/approval-log.mjs': 'stub' });
|
||||
const repo = makeRepo('main');
|
||||
const base = { session_id: 's-b', tool_use_id: 'toolu_b', cwd: repo, tool_name: 'Bash', tool_input: { command: 'git push origin main' }, permission_mode: 'bypassPermissions' };
|
||||
const base = { session_id: `s-b-${RUN}`, tool_use_id: 'toolu_b', cwd: repo, tool_name: 'Bash', tool_input: { command: 'git push origin main' }, permission_mode: 'bypassPermissions' };
|
||||
const pre = await runHook(join(plugin, 'hooks', 'scripts', 'pre-bash-executor.mjs'), { ...base, hook_event_name: 'PreToolUse' });
|
||||
assert.equal(pre.code, 2, `refused, not asked: ${JSON.stringify(pre)}`);
|
||||
const log = join(tmp(), 'stub.jsonl');
|
||||
|
|
@ -297,7 +300,7 @@ test('review 2nd round (N4): when the record cannot be written, the prompt says
|
|||
const fileAsDir = join(tmp(), 'not-a-dir');
|
||||
writeFileSync(fileAsDir, 'x');
|
||||
const r = await runHookWithEnv(join(plugin, 'hooks', 'scripts', 'pre-bash-executor.mjs'), {
|
||||
hook_event_name: 'PreToolUse', session_id: 's-n4', tool_use_id: 'toolu_n4', cwd: makeRepo('main'),
|
||||
hook_event_name: 'PreToolUse', session_id: `s-n4-${RUN}`, tool_use_id: 'toolu_n4', cwd: makeRepo('main'),
|
||||
tool_name: 'Bash', tool_input: { command: 'git push origin main' },
|
||||
}, { TMPDIR: fileAsDir });
|
||||
const reason = parseJsonOut(r.stdout).hookSpecificOutput.permissionDecisionReason;
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ import { join, dirname } from 'node:path';
|
|||
import { tmpdir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { runHookWithEnv } from '../helpers/hook-helper.mjs';
|
||||
import { fastestWithin } from '../helpers/timing.mjs';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const HOOK_PATH = join(__dirname, '..', '..', 'hooks', 'scripts', 'otel-export.mjs');
|
||||
|
|
@ -104,16 +105,21 @@ test('SC #14: VOYAGE_EXPORT_MODE=otlp + invalid endpoint → stderr [voyage] war
|
|||
test('SC #14: tail-latency for textfile mode < 200ms (NFR)', async () => {
|
||||
const dataDir = setupDataDir();
|
||||
try {
|
||||
const start = performance.now();
|
||||
const r = await runHookWithEnv(HOOK_PATH, '{}', {
|
||||
VOYAGE_EXPORT_MODE: 'textfile',
|
||||
CLAUDE_PLUGIN_DATA: dataDir,
|
||||
// The fastest of up to three cold runs against the same 800 ms limit
|
||||
// (tests/helpers/timing.mjs): load only adds time, so a real slowness
|
||||
// still fails every run, and one run slowed by a busy machine does not.
|
||||
const t = await fastestWithin(800, async () => {
|
||||
const start = performance.now();
|
||||
const r = await runHookWithEnv(HOOK_PATH, '{}', {
|
||||
VOYAGE_EXPORT_MODE: 'textfile',
|
||||
CLAUDE_PLUGIN_DATA: dataDir,
|
||||
});
|
||||
assert.equal(r.code, 0);
|
||||
return performance.now() - start;
|
||||
});
|
||||
const elapsed = performance.now() - start;
|
||||
assert.equal(r.code, 0);
|
||||
// 200ms NFR with extra headroom for cold-start node spawn (~100ms typical)
|
||||
assert.ok(elapsed < 800,
|
||||
`textfile export tail-latency too slow: ${elapsed.toFixed(0)}ms (NFR <200ms in-process; <800ms allowed for cold spawn)`);
|
||||
assert.ok(t.ok,
|
||||
`textfile export tail-latency too slow: fastest ${t.fastest.toFixed(0)}ms of ${t.runs.map((x) => x.toFixed(0)).join('/')}ms (NFR <200ms in-process; <800ms allowed for cold spawn)`);
|
||||
} finally {
|
||||
rmSync(dataDir, { recursive: true, force: true });
|
||||
}
|
||||
|
|
|
|||
131
tests/lib/retterunde-prose-pins.test.mjs
Normal file
131
tests/lib/retterunde-prose-pins.test.mjs
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
// tests/lib/retterunde-prose-pins.test.mjs
|
||||
// Prose pins for the fix round before 6.3.0 (order 10) whose fix is text a
|
||||
// session reads. Counted as doc pins by lib/util/test-census.mjs (the file name).
|
||||
// 1b A step of only tests can never be proven red: the plan text, the
|
||||
// template and plan-critic say the test belongs in the step that builds
|
||||
// the behaviour (voyage-lekerepo trial 3, 2026-10-09, finding 1).
|
||||
// 1c The executor and the step helper pass `--plan`/`--step` to red-proof
|
||||
// and send a `tests_only` `not-red-at-base` up to the plan, not into fix
|
||||
// rounds that cannot succeed.
|
||||
// 4 /trekreview names FINDING_PROSE_LINE_OUT_OF_RANGE and hands the check
|
||||
// the reviewer's whole answer, not only its JSON block.
|
||||
// 6 The resolver returns `effort` only when the brief's phase signal sets
|
||||
// one; no command says it always does.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { mkdtempSync, mkdirSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { join, dirname } from 'node:path';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { findSection } from '../../lib/parsers/plan-schema.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const read = (p) => readFileSync(join(ROOT, p), 'utf8');
|
||||
|
||||
/** The text of a heading's section (level from its #-count), up to the next one outside a fence. */
|
||||
function section(text, heading) {
|
||||
const level = heading.match(/^#+/)[0].length;
|
||||
const s = findSection(text, heading.replace(/^#+ /, ''), level);
|
||||
assert.ok(s, `missing ${heading}`);
|
||||
return s.text;
|
||||
}
|
||||
|
||||
const RULE = /test\s+for\s+a\s+behaviou?r\s+belongs\s+in\s+the\s+step\s+that\s+builds\s+it/i;
|
||||
|
||||
// --- 1b --------------------------------------------------------------------------
|
||||
|
||||
test('1b: /trekplan says the test belongs in the step that builds the behaviour, and names the warning', () => {
|
||||
const plan = read('commands/trekplan.md');
|
||||
assert.match(plan, RULE);
|
||||
// At the vertical-slice rule and at the step field list.
|
||||
assert.match(plan, /Steps are vertical slices[\s\S]{0,900}only tests[\s\S]{0,300}never be proven red/i);
|
||||
assert.match(plan, /\*\*Test first:\*\* — [^\n]*\n[^\n]*step that builds/i);
|
||||
assert.match(plan, /PLAN_STEP_TESTS_ONLY/);
|
||||
});
|
||||
|
||||
test('1b: the plan template carries the same rule', () => {
|
||||
const t = read('templates/plan-template.md');
|
||||
assert.match(t, RULE);
|
||||
assert.match(t, /only tests[\s\S]{0,200}never be proven red/i);
|
||||
});
|
||||
|
||||
test('1b: plan-critic has it as a finding under Verification gaps, and still 12 dimensions', () => {
|
||||
const critic = read('agents/plan-critic.md');
|
||||
const d8 = section(critic, '### 8. Verification gaps');
|
||||
assert.match(d8, /PLAN_STEP_TESTS_ONLY/);
|
||||
assert.match(d8, /only tests[\s\S]{0,400}\*\*major\*\*/i);
|
||||
assert.equal((critic.match(/^### \d+\. /gm) ?? []).length, 12);
|
||||
});
|
||||
|
||||
// --- 1c --------------------------------------------------------------------------
|
||||
|
||||
test('1c: the executor runs red-proof with --plan/--step and sends a tests_only not-red-at-base up to the plan', () => {
|
||||
const ex = read('commands/trekexecute.md');
|
||||
assert.match(ex, /red-proof\.mjs --base \{start_sha\} --cmd "\{Verify command\}" --tests [^\n]*--plan "\{plan_path\}" --step \{N\}[^\n]*--json/);
|
||||
assert.match(ex, /`not-red-at-base`, `tests_only: true`[^\n]*straight up to the plan \(Sub-step F, `fix_rounds = 4`, stop class 4\)/);
|
||||
// The helper's gate, as the executor describes it, carries the same flags.
|
||||
assert.match(ex, /red-proof\.mjs --red-only --base \{start_sha\} [^\n]*--plan "\{plan_path\}" --step \{N\}/);
|
||||
// The other not-red-at-base keeps its fix round.
|
||||
assert.match(ex, /`not-red-at-base`, otherwise[^\n]*finding for Sub-step F/);
|
||||
});
|
||||
|
||||
test('1c: the step helper passes --plan/--step to its red gate and does not rewrite a tests_only test', () => {
|
||||
const h = read('agents/step-implementer.md');
|
||||
assert.match(h, /red-proof\.mjs --red-only [^\n]*--plan "\{plan\}" --step \{step\}/);
|
||||
assert.match(h, /`tests_only: true`[\s\S]{0,300}do not rewrite/i);
|
||||
});
|
||||
|
||||
// --- 4 ---------------------------------------------------------------------------
|
||||
|
||||
test('4: /trekreview names FINDING_PROSE_LINE_OUT_OF_RANGE: re-ask on the error, report the warning', () => {
|
||||
const r = read('commands/trekreview.md');
|
||||
assert.match(r, /FINDING_PROSE_LINE_OUT_OF_RANGE/);
|
||||
assert.match(r, /FINDING_PROSE_LINE_OUT_OF_RANGE[\s\S]{0,600}re-ask/i);
|
||||
assert.match(r, /FINDING_PROSE_LINE_OUT_OF_RANGE[\s\S]{0,900}warning[\s\S]{0,300}report/i);
|
||||
});
|
||||
|
||||
test('4: /trekreview saves the reviewer\'s whole answer for the check, not only the JSON block', () => {
|
||||
const r = read('commands/trekreview.md');
|
||||
assert.match(r, /\{project_dir\}\/\.review-out-\{reviewer\}\.md/);
|
||||
assert.match(r, /whole answer[\s\S]{0,300}not only the\s+(JSON\s+)?block/i);
|
||||
assert.match(r, /findings-schema\.mjs --json --repo-root [^\n]*"\{project_dir\}\/\.review-out-\{reviewer\}\.md"/);
|
||||
assert.doesNotMatch(r, /Collect each reviewer's trailing JSON block/);
|
||||
});
|
||||
|
||||
test('4: what the check is given decides: the whole answer warns on a prose citation, the block alone cannot', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'retterunde-review-'));
|
||||
try {
|
||||
mkdirSync(join(dir, 'src'));
|
||||
writeFileSync(join(dir, 'src', 'a.mjs'), 'export const a = 1;\n');
|
||||
const block = '```json\n{"reviewer":"code-correctness-reviewer","findings":[]}\n```\n';
|
||||
writeFileSync(join(dir, 'whole.md'), `The bug is at \`src/a.mjs:40\`.\n\n${block}`);
|
||||
writeFileSync(join(dir, 'block.md'), block);
|
||||
const cli = (f) => spawnSync(process.execPath,
|
||||
[join(ROOT, 'lib', 'review', 'findings-schema.mjs'), '--json', '--repo-root', dir, join(dir, f)], { encoding: 'utf8' });
|
||||
const whole = JSON.parse(cli('whole.md').stdout);
|
||||
assert.deepEqual(whole.warnings.map((w) => w.code), ['FINDING_PROSE_LINE_OUT_OF_RANGE']);
|
||||
assert.deepEqual(JSON.parse(cli('block.md').stdout).warnings, []);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
// --- 6 ---------------------------------------------------------------------------
|
||||
|
||||
const RESOLVER_COMMANDS = ['trekplan', 'trekexecute', 'trekresearch', 'trekreview'];
|
||||
|
||||
test('6: no command says the resolver always returns {effort, model, source}', () => {
|
||||
for (const f of readdirSync(join(ROOT, 'commands')).filter((x) => x.endsWith('.md'))) {
|
||||
const t = read(`commands/${f}`);
|
||||
assert.doesNotMatch(t, /\{effort, model, source\}/, f);
|
||||
}
|
||||
});
|
||||
|
||||
test('6: each resolver-using command says `effort` comes only from the brief signal, else standard', () => {
|
||||
for (const c of RESOLVER_COMMANDS) {
|
||||
const t = read(`commands/${c}.md`);
|
||||
const hits = t.match(/\{model, source\}[^\n]*\n?[^\n]*effort[^\n]*only when the brief's phase signal sets one/g) ?? [];
|
||||
assert.ok(hits.length >= 1, `${c}: no statement that effort is only present with a brief signal`);
|
||||
assert.match(t, /without it[^\n]*\n?[^\n]*`?standard`?/i, c);
|
||||
}
|
||||
});
|
||||
218
tests/lib/retterunde.test.mjs
Normal file
218
tests/lib/retterunde.test.mjs
Normal file
|
|
@ -0,0 +1,218 @@
|
|||
// tests/lib/retterunde.test.mjs
|
||||
// The fix round before 6.3.0 (order 10), the points that are behaviour, not
|
||||
// prose. Each test was red before its fix.
|
||||
// 1 A step whose `Files:` are all tests can never be proven red: the
|
||||
// behaviour it tests is already at its start point (voyage-lekerepo
|
||||
// trial 3, 2026-10-09, finding 1). (a) the validator warns
|
||||
// PLAN_STEP_TESTS_ONLY; (c) red-proof says `tests_only` on its
|
||||
// `not-red-at-base`, so the executor sends the step up to the plan instead
|
||||
// of into fix rounds that cannot succeed. One reading: stepTestsOnly.
|
||||
// 3 The example plan's `On failure:` lines open with a K11 word.
|
||||
// 5 The criteria runner's commands never see NODE_TEST_CONTEXT: a nested
|
||||
// `node --test` would report to the parent and exit 0 on a red test.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { mkdirSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { validatePlan, validatePlanContent } from '../../lib/validators/plan-validator.mjs';
|
||||
import { parseStepFields, sliceSteps, stepTestsOnly } from '../../lib/parsers/plan-schema.mjs';
|
||||
import { proveRed } from '../../lib/verification/red-proof.mjs';
|
||||
import { runCriteria } from '../../lib/verification/criteria-runner.mjs';
|
||||
import { buildPlan } from '../helpers/plan-v18-builder.mjs';
|
||||
|
||||
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||
const ROOT = join(HERE, '..', '..');
|
||||
const codes = (list) => list.map((e) => e.code);
|
||||
|
||||
// --- 1 (a): the validator -----------------------------------------------------
|
||||
|
||||
// Step 2 changes only tests for behaviour step 1 delivered, as in the trial.
|
||||
const TESTS_ONLY_PLAN = buildPlan({
|
||||
steps: [{}, { file: 'tests/e2e.sh', test: 'tests/e2e-guard.test.mjs', endToEnd: 'bash tests/e2e.sh' }],
|
||||
});
|
||||
|
||||
test('1a: a step whose Files are all tests gives PLAN_STEP_TESTS_ONLY, a warning on that step', () => {
|
||||
const r = validatePlanContent(TESTS_ONLY_PLAN);
|
||||
const w = r.warnings.filter((x) => x.code === 'PLAN_STEP_TESTS_ONLY');
|
||||
assert.equal(w.length, 1, codes(r.warnings).join(','));
|
||||
assert.match(w[0].message, /Step 2/);
|
||||
assert.ok(!codes(r.errors).includes('PLAN_STEP_TESTS_ONLY'), 'a warning, never an error');
|
||||
assert.equal(r.valid, true, codes(r.errors).join(','));
|
||||
});
|
||||
|
||||
test('1a: a step with code and its test gives no PLAN_STEP_TESTS_ONLY (as before)', () => {
|
||||
const r = validatePlanContent(buildPlan());
|
||||
assert.ok(!codes(r.warnings).includes('PLAN_STEP_TESTS_ONLY'), codes(r.warnings).join(','));
|
||||
});
|
||||
|
||||
test('1a: the reading - Files all tests (by path or as the Test first file) or not; no Files = no warning', () => {
|
||||
const body = (files, tf = 'tests/x.test.mjs') => [
|
||||
'### Step 1: x', '',
|
||||
...(files === null ? [] : [`- **Files:** ${files}`]),
|
||||
'- **Test first:**', ` - File: \`${tf}\` (new)`, ' - Verifies: x',
|
||||
].join('\n');
|
||||
assert.equal(stepTestsOnly(body('`tests/e2e.sh`, `tests/e2e-guard.sh`')).testsOnly, true);
|
||||
assert.equal(stepTestsOnly(body('`src/a.test.ts` (new file), `lib/b.spec.js`, `pkg/c_test.go`')).testsOnly, true);
|
||||
assert.equal(stepTestsOnly(body('`__tests__/a.js`, `test/b.js`')).testsOnly, true);
|
||||
// The Test first file counts as a test wherever it lives.
|
||||
assert.equal(stepTestsOnly(body('`scripts/check.sh`', 'scripts/check.sh')).testsOnly, true);
|
||||
assert.equal(stepTestsOnly(body('`src/cli.mjs`, `tests/cli.test.mjs`')).testsOnly, false);
|
||||
assert.equal(stepTestsOnly(body('`src/cli.mjs`')).testsOnly, false);
|
||||
assert.equal(stepTestsOnly(body(null)).testsOnly, false);
|
||||
assert.deepEqual(stepTestsOnly(body('`tests/e2e.sh`, `tests/e2e-guard.sh`')).files, ['tests/e2e.sh', 'tests/e2e-guard.sh']);
|
||||
});
|
||||
|
||||
test('1a: a plain path beside a backticked one is still read - a step that also changes code is not tests-only', () => {
|
||||
const body = (files) => ['### Step 1: x', '', `- **Files:** ${files}`].join('\n');
|
||||
const mixed = stepTestsOnly(body('src/cli.mjs (modify), `tests/x.test.mjs` (new)'));
|
||||
assert.deepEqual(mixed.files, ['src/cli.mjs', 'tests/x.test.mjs']);
|
||||
assert.equal(mixed.testsOnly, false);
|
||||
const list = stepTestsOnly(['### Step 1: x', '', '- **Files:**', ' - `tests/x.test.mjs` (new)', ' - src/cli.mjs'].join('\n'));
|
||||
assert.equal(list.testsOnly, false, JSON.stringify(list.files));
|
||||
});
|
||||
|
||||
test('1a: a document under spec/ or specs/ is not a test for this reading', () => {
|
||||
const body = (files) => ['### Step 1: x', '', `- **Files:** ${files}`].join('\n');
|
||||
assert.equal(stepTestsOnly(body('`docs/specs/api.md`, `specs/001-auth/plan.md`')).testsOnly, false);
|
||||
assert.equal(stepTestsOnly(body('`spec/cli_spec.rb`')).testsOnly, true);
|
||||
});
|
||||
|
||||
// --- 1 (c): red-proof says tests_only on not-red-at-base ----------------------
|
||||
|
||||
function sh(cwd, cmd, args) {
|
||||
const r = spawnSync(cmd, args, { cwd, encoding: 'utf8' });
|
||||
assert.equal(r.status, 0, `${cmd} ${args.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
const SUM_TEST = [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"import { sum } from '../lib/sum.mjs';",
|
||||
"test('sum adds', () => { assert.equal(sum(1, 2), 3); });",
|
||||
'',
|
||||
].join('\n');
|
||||
|
||||
/** The behaviour is already at the start point; the step adds only its test. */
|
||||
function greenAtBase() {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'retterunde-red-'));
|
||||
sh(dir, 'git', ['init', '-q']);
|
||||
sh(dir, 'git', ['config', 'user.email', 't@example.invalid']);
|
||||
sh(dir, 'git', ['config', 'user.name', 'test']);
|
||||
mkdirSync(join(dir, 'lib'));
|
||||
writeFileSync(join(dir, 'lib', 'sum.mjs'), 'export function sum(a, b) { return a + b; }\n');
|
||||
sh(dir, 'git', ['add', 'lib/sum.mjs']);
|
||||
sh(dir, 'git', ['commit', '-q', '-m', 'base']);
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'sum.test.mjs'), SUM_TEST);
|
||||
return { dir, base: sh(dir, 'git', ['rev-parse', 'HEAD']) };
|
||||
}
|
||||
|
||||
function planWithStep2Files(dir, files) {
|
||||
const p = join(dir, 'plan.local.md');
|
||||
writeFileSync(p, buildPlan({ steps: [{}, { file: files, test: 'tests/sum.test.mjs', endToEnd: 'bash tests/e2e.sh' }] }));
|
||||
return p;
|
||||
}
|
||||
|
||||
const CMD = 'node --test tests/sum.test.mjs';
|
||||
|
||||
test('1c: not-red-at-base on a step whose Files are all tests says tests_only: true (red-only and full)', () => {
|
||||
const { dir, base } = greenAtBase();
|
||||
try {
|
||||
const plan = planWithStep2Files(dir, 'tests/sum.test.mjs');
|
||||
for (const redOnly of [true, false]) {
|
||||
const r = proveRed({ cwd: dir, base, cmd: CMD, tests: ['tests/sum.test.mjs'], redOnly, plan, step: 2 });
|
||||
assert.equal(r.reason, 'not-red-at-base', JSON.stringify(r));
|
||||
assert.equal(r.tests_only, true, JSON.stringify(r));
|
||||
}
|
||||
const cli = spawnSync(process.execPath, [join(ROOT, 'lib', 'verification', 'red-proof.mjs'),
|
||||
'--cwd', dir, '--base', base, '--cmd', CMD, '--tests', 'tests/sum.test.mjs', '--plan', plan, '--step', '2', '--json'],
|
||||
{ encoding: 'utf8' });
|
||||
assert.equal(cli.status, 1, cli.stdout + cli.stderr);
|
||||
assert.equal(JSON.parse(cli.stdout).tests_only, true);
|
||||
// Text mode names the field too (the step helper reads it there).
|
||||
const text = spawnSync(process.execPath, [join(ROOT, 'lib', 'verification', 'red-proof.mjs'),
|
||||
'--red-only', '--cwd', dir, '--base', base, '--cmd', CMD, '--tests', 'tests/sum.test.mjs', '--plan', plan, '--step', '2'],
|
||||
{ encoding: 'utf8' });
|
||||
assert.match(text.stdout, /tests_only: true/);
|
||||
// --step without --plan is a usage error, never silently dropped.
|
||||
const lone = spawnSync(process.execPath, [join(ROOT, 'lib', 'verification', 'red-proof.mjs'),
|
||||
'--cwd', dir, '--base', base, '--cmd', CMD, '--tests', 'tests/sum.test.mjs', '--step', '2'], { encoding: 'utf8' });
|
||||
assert.equal(lone.status, 2, lone.stderr);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
test('1c: not-red-at-base on a step that also changes code says tests_only: false (fix rounds, as before)', () => {
|
||||
const { dir, base } = greenAtBase();
|
||||
try {
|
||||
const plan = planWithStep2Files(dir, 'lib/sum.mjs');
|
||||
const r = proveRed({ cwd: dir, base, cmd: CMD, tests: ['tests/sum.test.mjs'], plan, step: 2 });
|
||||
assert.equal(r.reason, 'not-red-at-base');
|
||||
assert.equal(r.tests_only, false, JSON.stringify(r));
|
||||
// Without --plan nothing new is said: the field is absent.
|
||||
const bare = proveRed({ cwd: dir, base, cmd: CMD, tests: ['tests/sum.test.mjs'] });
|
||||
assert.equal(bare.reason, 'not-red-at-base');
|
||||
assert.equal('tests_only' in bare, false);
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
|
||||
// --- 3: the example plan reads with the K11 words -----------------------------
|
||||
|
||||
test('3: the example plan gives no PLAN_STEP_ON_FAILURE_UNKNOWN', () => {
|
||||
const r = validatePlan(join(ROOT, 'examples', '01-add-verbose-flag', 'plan.md'));
|
||||
assert.deepEqual(r.warnings.filter((w) => w.code === 'PLAN_STEP_ON_FAILURE_UNKNOWN').map((w) => w.message), []);
|
||||
});
|
||||
|
||||
// --- 5: the criteria runner strips NODE_TEST_CONTEXT ---------------------------
|
||||
|
||||
test('5: a nested `node --test` on a failing test is red even when NODE_TEST_CONTEXT is set', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'retterunde-ctx-'));
|
||||
const saved = process.env.NODE_TEST_CONTEXT;
|
||||
try {
|
||||
mkdirSync(join(dir, 'tests'));
|
||||
writeFileSync(join(dir, 'tests', 'fail.test.mjs'), [
|
||||
"import { test } from 'node:test';",
|
||||
"import { strict as assert } from 'node:assert';",
|
||||
"test('fails', () => { assert.equal(1, 2); });",
|
||||
'',
|
||||
].join('\n'));
|
||||
process.env.NODE_TEST_CONTEXT = 'child-v8';
|
||||
const [r] = runCriteria([{ label: 'C1', text: 'x', command: 'node --test tests/fail.test.mjs' }], { cwd: dir });
|
||||
assert.equal(r.status, 'failed', JSON.stringify(r));
|
||||
assert.notEqual(r.exitCode, 0);
|
||||
} finally {
|
||||
if (saved === undefined) delete process.env.NODE_TEST_CONTEXT;
|
||||
else process.env.NODE_TEST_CONTEXT = saved;
|
||||
rmSync(dir, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
// Operator 2026-10-09: the example's verbose-content test moves into the step
|
||||
// that wires the verbose output; steps 1-2 (golden capture and the stability
|
||||
// harness: tests of today's behaviour, never red) keep the warning, and a step
|
||||
// type for them is a wave 3 question.
|
||||
test('1: the example plan warns only on its two golden/stability steps; the verbose test sits with the code', () => {
|
||||
const r = validatePlan(join(ROOT, 'examples', '01-add-verbose-flag', 'plan.md'));
|
||||
const w = r.warnings.filter((x) => x.code === 'PLAN_STEP_TESTS_ONLY').map((x) => x.message.match(/Step (\d+)/)[1]);
|
||||
assert.deepEqual(w, ['1', '2']);
|
||||
const text = readFileSync(join(ROOT, 'examples', '01-add-verbose-flag', 'plan.md'), 'utf8');
|
||||
const step = sliceSteps(text).find((s) => /tests\/cli-verbose-flag\.test\.mjs/.test(parseStepFields(s.body).get('Files')?.text ?? ''));
|
||||
assert.ok(step, 'a step lists tests/cli-verbose-flag.test.mjs in Files');
|
||||
assert.match(parseStepFields(step.body).get('Files').text, /src\/commands\/login\.mjs/);
|
||||
// The run record follows the plan: one entry per step.
|
||||
const progress = JSON.parse(readFileSync(join(ROOT, 'examples', '01-add-verbose-flag', 'progress.json'), 'utf8'));
|
||||
const n = r.parsed.steps.length;
|
||||
assert.equal(progress.total_steps, n);
|
||||
assert.equal(progress.current_step, n);
|
||||
assert.deepEqual(progress.steps.map((s) => s.n), Array.from({ length: n }, (_, i) => i + 1));
|
||||
});
|
||||
|
||||
// Guard against a reading of the example that counts nothing: it has steps.
|
||||
test('3: the example plan is read (six or more steps)', () => {
|
||||
const text = readFileSync(join(ROOT, 'examples', '01-add-verbose-flag', 'plan.md'), 'utf8');
|
||||
assert.ok(validatePlanContent(text).parsed.steps.length >= 6);
|
||||
});
|
||||
63
tests/lib/timing-under-load.test.mjs
Normal file
63
tests/lib/timing-under-load.test.mjs
Normal file
|
|
@ -0,0 +1,63 @@
|
|||
// tests/lib/timing-under-load.test.mjs
|
||||
// The two timing tests that went red under load in the wave sessions (order 10,
|
||||
// point 2; operator 2026-10-09): P7 (post-edit-check) was killed by the hook
|
||||
// helper's 5 s limit at load 95-177 and was 8/8 green alone; SC #14
|
||||
// (otel-export) demands < 800 ms for ONE cold run.
|
||||
// - The helper's kill limit is longer (HOOK_TIMEOUT_MS) and overridable per
|
||||
// call. P7 checks the hook's answer, not its speed, so that weakens nothing.
|
||||
// - SC #14 keeps its 800 ms limit and measures up to three runs, passing on
|
||||
// the fastest. Load only ever ADDS time, so the fastest run is the best
|
||||
// reading of the real cost: a real slowness is slow every time and still
|
||||
// fails; one run slowed by a busy machine does not.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { join } from 'node:path';
|
||||
import { fastestWithin } from '../helpers/timing.mjs';
|
||||
import { HOOK_TIMEOUT_MS, runHookWithEnv } from '../helpers/hook-helper.mjs';
|
||||
|
||||
/** A measure that returns the given durations in turn (no real waiting). */
|
||||
const stub = (durations) => {
|
||||
let i = 0;
|
||||
return async () => durations[i++];
|
||||
};
|
||||
|
||||
test('timing: a stub that is slow every time fails the limit (a real slowness is felled)', async () => {
|
||||
const r = await fastestWithin(800, stub([900, 950, 1200]));
|
||||
assert.equal(r.ok, false);
|
||||
assert.deepEqual(r.runs, [900, 950, 1200]);
|
||||
assert.equal(r.fastest, 900);
|
||||
});
|
||||
|
||||
test('timing: one slow run of three is tolerated - the next run under the limit passes', async () => {
|
||||
const r = await fastestWithin(800, stub([1500, 300, 2000]));
|
||||
assert.equal(r.ok, true);
|
||||
assert.deepEqual(r.runs, [1500, 300], 'stops at the first run under the limit');
|
||||
});
|
||||
|
||||
test('timing: the limit itself is not loosened - a run AT the limit is not under it', async () => {
|
||||
const r = await fastestWithin(800, stub([800, 800, 800]));
|
||||
assert.equal(r.ok, false);
|
||||
});
|
||||
|
||||
test('timing: a first run under the limit passes at once (one run on a quiet machine)', async () => {
|
||||
const r = await fastestWithin(800, stub([120]));
|
||||
assert.equal(r.ok, true);
|
||||
assert.deepEqual(r.runs, [120]);
|
||||
});
|
||||
|
||||
test('hook helper: the kill limit is far above 5 s by default, and a call can set its own', async () => {
|
||||
assert.ok(HOOK_TIMEOUT_MS >= 30_000, `HOOK_TIMEOUT_MS = ${HOOK_TIMEOUT_MS}`);
|
||||
const dir = mkdtempSync(join(tmpdir(), 'hook-helper-'));
|
||||
try {
|
||||
const slow = join(dir, 'slow-hook.mjs');
|
||||
writeFileSync(slow, "setTimeout(() => { process.stdout.write('done'); process.exit(0); }, 400);\n");
|
||||
const killed = await runHookWithEnv(slow, '{}', {}, { timeoutMs: 100 });
|
||||
assert.notEqual(killed.code, 0, 'a per-call limit below the hook\'s run time kills it');
|
||||
const done = await runHookWithEnv(slow, '{}', {});
|
||||
assert.equal(done.code, 0);
|
||||
assert.equal(done.stdout, 'done');
|
||||
} finally { rmSync(dir, { recursive: true, force: true }); }
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue