test(red-proof): fix round 4.5b2 part C red proof and changelog
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
ff3e20919e
commit
6c8f7377bb
5 changed files with 510 additions and 1 deletions
17
CHANGELOG.md
17
CHANGELOG.md
|
|
@ -35,7 +35,7 @@ Part B makes a WARN open the merge lock and the release check only when the oper
|
|||
- **The operator answers each MAJOR, and "fixed" runs the fix loop** (FXL). `/trekreview` Phase 8.8 lists the open MAJORs with `review-answers.mjs pending` and asks about each with `AskUserQuestion`, at most four per call and one finding per question: Fix it, or Accept with a reason, never offered for a security finding. `/trekreview --answers` runs only this step and opens no new round. A "fixed" answer gets its `### Step R<n>`; `/trekexecute` enters its fix loop on `review-unanswered: fixed-pending` and closes the merge with the decline reason `review-unanswered` otherwise.
|
||||
- **A `REVIEW.md` policy caps and skips MINOR and SUGGESTION findings only** (POL). `lib/review/review-policy.mjs` reads one `voyage-review-policy` block (`max-minor`, `skip`) from `REVIEW.md` at the repo root, from git at `scope_sha_start`, as data. A capped or skipped finding goes to Declined to Judge (`policy:cap`, `policy:skip`); a BLOCKER, a MAJOR or a security finding is never moved, and the gate exits 2 on a review that moved one. A malformed, symlinked or in-range changed file applies no policy; the review records the file's sha256.
|
||||
- **The standard path puts back engine findings the coordinator dropped** (ENG). `lib/review/engine-compare.mjs`, run by finalize, puts an engine MAJOR or BLOCKER that `review.md` does not hold back at its own severity, as the workflow path does; the verdict is only raised (a BLOCKER gives BLOCK). A finding under a trimmed title is not put back twice, an engine finding in `out_of_scope` stays declined, and its row rule is pinned equal to the workflow script's.
|
||||
- **The code-correctness reviewer checks the repo's tracked instruction files** (CMD). Its eighth dimension checks the diff against the set `lib/review/instruction-scope.mjs --set` gives: the root and on-path `CLAUDE.md` and `AGENTS.md` and `.claude/rules/*.md`, tracked at the reviewed commit. An untracked file, a file outside the repo or a tracked symlink that resolves outside it is left out; a finding that cites a file outside the set goes to Declined to Judge (`instruction-out-of-set`).
|
||||
- **The code-correctness reviewer checks the repo's tracked instruction files** (CMD). Its eighth dimension checks the diff against the set `lib/review/instruction-scope.mjs --set` gives: the root and on-path `CLAUDE.md` and `AGENTS.md` and the `.md` files under `.claude/rules/` at any depth (a changed path's own directory included), tracked at the reviewed commit. An untracked file, a file outside the repo or a tracked symlink that resolves outside it is left out, and a tracked link to a file inside the repo counts under its own name and its target's; a finding that cites a file outside the set goes to Declined to Judge (`instruction-out-of-set`).
|
||||
- **A changed instruction file gets deep review at any depth** (DPR). A changed `CLAUDE.md`, `AGENTS.md` or `.claude/rules/*.md`, `docs/**` included, gets deep review (`instruction-scope.mjs --triage`); the triage table's rows are unchanged.
|
||||
- **Two instruction-file rule keys at MINOR** (CAT). The rule catalogue gains `CLAUDE_MD_DEVIATION` and `CLAUDE_MD_STALE`, both MINOR, category conformance, with no memory route; it holds 17 keys, and the workflow script's tier table holds both.
|
||||
- **The merge question says how the run was tested and how it is undone** (MEV). Before the merge, `/trekexecute` Phase 8 runs `lib/review/merge-evidence.mjs write`, which derives an internal evidence file from the run's artifacts under the git common dir, never tracked. The Bash guard reads it into a "Tested by" and an "Undone by" line of its merge question; its answer stays "ask", and a missing, stale or unreadable file reads "no run evidence" or "not measured". Files older than 30 days are removed.
|
||||
|
|
@ -129,6 +129,19 @@ Part A fixes four of the six MAJOR findings of part 4's review in the Bash guard
|
|||
- **A `<<` whose opener a backslash-newline splits is never skipped** (SIS). A `<<` whose opener a backslash-newline splits is never skipped, as text in a string or as text in the body of a quoted heredoc that passes, because the regex reading cannot see it; such an opener inside a `bash -c` string with a body was let through, also inside a quoted heredoc fed to `bash`, and the inner shell expanded the body. A harmless one with a line after the command may now be refused. Decision 126 widened it to the body skip. It is decision 123.
|
||||
- **A test header names its red base** (HDR). The header of `tests/hooks/bash-guard-false-stops.test.mjs` named the wrong red base; it now names part 3's end point `591f3c4`. It names review finding `86488cd6`.
|
||||
|
||||
**Fix round 4.5b2, part C (decisions 135 and 137)**
|
||||
|
||||
Part C fixes the eight code MAJOR findings of part B's review, round 1, on the same branch before that review's round 2 (decision 135). The guard's reading of a quoted value is left out and listed under Known issues (decision 137). Each fix has a test seen failing first at part B's head `f1f5ab5` (`tests/fix-round-4-5b2-c-red.sh` proves it).
|
||||
|
||||
- **One answer can no longer clear two MAJORs that share an id** (DUP). `lib/review/verdict-gate.mjs` exits 2 (not measured) when the frontmatter `findings:` list holds one id twice or two findings of the trailing JSON block share one id, and a BLOCK review still exits 1. `lib/review/review-answers.mjs` refuses such a review too: its pending list is not measured, and `record` appends no answer to it. It names review finding `e57259bf`.
|
||||
- **An answer recorded after a stopped round-2 prepare no longer locks the rounds** (RSP). `lib/review/review-rounds.mjs prepare` knows review.md's own archive by its content (its round number, its trailing JSON block and its `scope_sha_end`), not only by its bytes, and refreshes it with review.md, so an answer or its R-step written after the stop is not counted as another round. An archive whose findings or `scope_sha_end` differ still counts as an earlier round. It names review finding `de594275`.
|
||||
- **Nested rules files are in the instruction set** (NRL). `lib/review/instruction-scope.mjs --set` holds every tracked `.md` file under `.claude/rules/` at any depth, at the root and in each directory on a changed path, as the classifier for deep review already read them. It names review finding `fd5a6e18`.
|
||||
- **A linked instruction file keeps its own name** (SYM). A tracked symlink whose target is a tracked regular file inside the repo adds both its own path and its target's to the set, so a finding that cites the link's name is not declined as out of set; a link that resolves outside the repo still adds nothing. It names review finding `d81f3c45`.
|
||||
- **The evidence writer takes a verdict only for the branch head** (WHD). `lib/review/merge-evidence.mjs write` asks the verdict gate only when `git rev-parse --verify HEAD` in `--repo` is the branch head; from any other HEAD the review verdict is null and the merge question reads "review not measured". It names review finding `561d6f11`.
|
||||
- **No run evidence for a merged name git reads as a tag or another ref** (AMB). The guard's reader (`hooks/lib/merge-evidence.mjs`) gives "no run evidence" when the merged name is also `refs/<name>` or `refs/tags/<name>`, loose or in `packed-refs`, or an entry in the git directory, and when `packed-refs` cannot be read; the writer refuses such a name (`"written": false`). It names review finding `df9f9870`.
|
||||
- **The reader knows that `--cleanup` takes a value** (CLU). The reader skips the value of `--cleanup`, so `git merge --cleanup strip --no-ff feat` shows feat's run evidence. A quoted `-m` value with spaces before the branch still gives "no run evidence", the fail-safe side; its fix goes with the guard's quoted-value reading (decision 137). It names review finding `b5ae9798`.
|
||||
- **A command with several merges says its run evidence is not shown** (MMG). The Bash guard hands every merge of the command to `evidenceForMerges` in `hooks/lib/merge-evidence.mjs`: one merge keeps its two lines, and two or more give a "Tested by: not shown" line that names the count and says to merge one at a time, followed by "Undone by: not shown". Before, only the first merge got run evidence. The answer stays "ask". It names review finding `73e1d174`.
|
||||
|
||||
Fix round 4.5a (decisions 57, 67 and 68): the round's 30 findings, where a Voyage control could be bypassed or was too lax, or where the text gave a guarantee no code held. Each is fixed, and each has a test that fails on the code before it, except DJ, whose sentences already stood and whose pins fail when they are deleted (`tests/fix-round-4-5a-red.sh` proves every other id red at the round's start point, and DJ under its two mutations).
|
||||
|
||||
**Execute**
|
||||
|
|
@ -242,6 +255,7 @@ Fix round 4.5a (decisions 57, 67 and 68): the round's 30 findings, where a Voyag
|
|||
- **`/security-review` re-raises run-range findings in round 2** (part B). The engines stay on the run range from round 2, because `/security-review` takes no range, so it raises its findings from the whole run again. Carry folds each into the previous finding it matches: it is carried, not counted as new, and it stays open while the engine raises it.
|
||||
- **Review answers cannot be recorded with the installed 6.5.0 plugin** (part B). Its `/trekreview` has no answers step, and its review stays at 1.1. This delivery's own review runs on 6.5.0, so before its release check the operator answers its MAJORs with this repo's `lib/review/review-answers.mjs` (`pending`, then `record`), the path `docs/release-runbook.md` gives.
|
||||
- **The fork point is read from a ref the session can move** (part B). `lib/review/pre-existing.mjs` reads the fork point from `refs/heads/main` (or `master`), and that ref can be moved with `git update-ref`, `git branch -f` or a plain commit on main before the branch is made; the Bash guard does not ask about any of them, so the binding is a trace, not a signature.
|
||||
- **The Bash guard lets some commands with a quoted value through unasked** (part C, decision 137). The guard's view of a command loses a quoted value that holds a space or is empty, so the option before it takes the next word as its value, and a merge into main, a push, a tag or a publish is not asked about. Measured with the guard's classifier (`classifyGitApproval`), each of these gives no question: `git tag -m "Release one" v1.0`, `git -C "/tmp/a b" push origin main`, `git -c "x.y=a b" push origin main`, `echo "a b"#; git push origin main`, `git tag -m '' v1.0`, `git -C ~/"My Repos"/x merge --no-ff feat`, `npm --workspace "$WS" publish`, and `git -C "<path with a space>" merge --no-ff feat`, the form `/trekexecute` Phase 8 merges with when the repo path holds a space. The hole is on main already, and part B's merge does not widen it. Under decision 137 the fix gets its own round with its own brief, where the mechanism is chosen on purpose.
|
||||
|
||||
### Doors
|
||||
|
||||
|
|
@ -256,6 +270,7 @@ Fix round 4.5a (decisions 57, 67 and 68): the round's 30 findings, where a Voyag
|
|||
- two-way: fix round 4.5b part 4 (decision 90) — a revert restores part 3's literal-delimiter gate and its refusal of a heredoc operator inside a string or after a backslash; the catch-branch test and the TOK wording go with it.
|
||||
- two-way: fix round 4.5b2 part A (decision 91, decision 107, decision 123) — a revert restores part 4's whole-command escape limits, the exact-word check for a split delimiter, the string and body skips of a split opener and the unchecked twice-escaped operator; the corpus, the end-to-end script, the keep-assertion test and the corrected test header go with it.
|
||||
- two-way: review gate 4.5b2 part B — `review_version` 1.2 and the answered-MAJOR merge lock (decisions 93-105, 128); a revert restores the 1.1 gate, 1.2 files still read under the old validator (unknown keys tolerated), and K2 `review` records keep the record shape.
|
||||
- two-way: fix round 4.5b2 part C (decisions 135, 137) — a revert restores part B's behaviour.
|
||||
|
||||
## v6.5.0 — 2026-10-18 — Three skills for brief, criteria and review, English labels, and three trial-5 fixes
|
||||
|
||||
|
|
|
|||
12
tests/fix-round-4-5b2-c-mutations.mjs
Normal file
12
tests/fix-round-4-5b2-c-mutations.mjs
Normal file
|
|
@ -0,0 +1,12 @@
|
|||
// tests/fix-round-4-5b2-c-mutations.mjs
|
||||
// The mutations of fix round 4.5b2, part C (plan
|
||||
// .claude/plans/trekplan-2026-11-03-review-gate-4-5b2-fixes.md, Step 7;
|
||||
// decisions 135 and 137), for the shared red script
|
||||
// (tests/helpers/red-round.sh, through tests/helpers/red-mutate.mjs).
|
||||
// The list is empty: no case of this round reads a file under tests/, so
|
||||
// every id of the round is new behaviour whose tests fail on an assertion at
|
||||
// the round's start point f1f5ab5, and none needs a mutation. Kept behaviour
|
||||
// is an untagged keep: test or a Control: assertion inside a red tagged test.
|
||||
// Data only; never matched by `npm test`.
|
||||
|
||||
export const MUTATIONS = [];
|
||||
77
tests/fix-round-4-5b2-c-red.sh
Normal file
77
tests/fix-round-4-5b2-c-red.sh
Normal file
|
|
@ -0,0 +1,77 @@
|
|||
#!/usr/bin/env bash
|
||||
# tests/fix-round-4-5b2-c-red.sh
|
||||
# The red proof of fix round 4.5b2, part C (plan
|
||||
# .claude/plans/trekplan-2026-11-03-review-gate-4-5b2-fixes.md, Step 7;
|
||||
# decisions 135 and 137), as a wrapper over the shared red script
|
||||
# tests/helpers/red-round.sh. Every id of the round has tests whose names
|
||||
# start with "[id]"; each must fail on an assertion at the round's start point
|
||||
# f1f5ab5 (part B's head, where all eight defects exist). Kept behaviour is an
|
||||
# untagged keep: test or a Control: assertion inside a red tagged test.
|
||||
# This wrapper passes the shared script:
|
||||
# --base BASE, the full sha of f1f5ab5;
|
||||
# --ids IDS (the round's nine ids);
|
||||
# --other-ids: the ids of fix round 4-5a, of 4-5b part 1, of part 2, of
|
||||
# part 3, of part 4, of 4.5b2 part A and of 4.5b2 part B, read with sed
|
||||
# from the IDS= lines of tests/fix-round-4-5a-red.sh,
|
||||
# tests/fix-round-4-5b-red.sh, tests/fix-round-4-5b-part-2-red.sh,
|
||||
# tests/fix-round-4-5b-part-3-red.sh, tests/fix-round-4-5b-part-4-red.sh,
|
||||
# tests/fix-round-4-5b2-a-red.sh and tests/fix-round-4-5b2-b-red.sh and
|
||||
# passed as ONE value, in that order (the shared script keeps only the
|
||||
# last --other-ids it is given); a tag on that list is not judged;
|
||||
# --mutations tests/fix-round-4-5b2-c-mutations.mjs (no mutation: no case
|
||||
# of this round reads a file under tests/);
|
||||
# the round's FILES: the seven new tagged test files. The scripts test is
|
||||
# untagged and not listed.
|
||||
# No --require-sha256: the round has no SHA-256 test.
|
||||
# Exit 0 all red, 1 a rule broken (the judge names it), 2 a setup failure
|
||||
# (also: no ids on one of the seven earlier IDS= lines); never 0 on a setup
|
||||
# failure.
|
||||
# bash 3.2 compatible, ASCII only, no arrays.
|
||||
|
||||
set -u
|
||||
set -f
|
||||
|
||||
cd "$(dirname "$0")/.." || { echo "red script: setup failure: cannot cd to the repo root" >&2; exit 2; }
|
||||
ROOT="$(pwd -P)"
|
||||
|
||||
BASE=f1f5ab5528a376050dd03b7f7ac43f887596c9b4
|
||||
IDS='DUP RSP NRL SYM WHD AMB CLU MMG PRC'
|
||||
FILES='tests/lib/unique-finding-ids.test.mjs tests/lib/review-rounds-stopped-prepare.test.mjs tests/lib/instruction-scope-depth.test.mjs tests/lib/merge-evidence-writer-head.test.mjs tests/hooks/merge-evidence-ambiguous.test.mjs tests/hooks/merge-evidence-many.test.mjs tests/lib/fix-round-4-5b2-c-prose.test.mjs'
|
||||
|
||||
OTHER_A=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5a-red.sh")
|
||||
if [ -z "$OTHER_A" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5a-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_B=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-red.sh")
|
||||
if [ -z "$OTHER_B" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_C=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-2-red.sh")
|
||||
if [ -z "$OTHER_C" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-2-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_D=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-3-red.sh")
|
||||
if [ -z "$OTHER_D" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-3-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_E=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-4-red.sh")
|
||||
if [ -z "$OTHER_E" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-4-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_F=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b2-a-red.sh")
|
||||
if [ -z "$OTHER_F" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b2-a-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_G=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b2-b-red.sh")
|
||||
if [ -z "$OTHER_G" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b2-b-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
exec /bin/bash "$ROOT/tests/helpers/red-round.sh" --base "$BASE" --ids "$IDS" --other-ids "$OTHER_A $OTHER_B $OTHER_C $OTHER_D $OTHER_E $OTHER_F $OTHER_G" --mutations "$ROOT/tests/fix-round-4-5b2-c-mutations.mjs" $FILES
|
||||
193
tests/lib/fix-round-4-5b2-c-prose.test.mjs
Normal file
193
tests/lib/fix-round-4-5b2-c-prose.test.mjs
Normal file
|
|
@ -0,0 +1,193 @@
|
|||
// tests/lib/fix-round-4-5b2-c-prose.test.mjs
|
||||
// Prose pins for fix round 4.5b2, part C (plan
|
||||
// .claude/plans/trekplan-2026-11-03-review-gate-4-5b2-fixes.md, Step 7): the
|
||||
// round's CHANGELOG entry. The tag in brackets is the part-C id from the
|
||||
// round's plan; each tagged test fails on an assertion at the round's red base
|
||||
// f1f5ab5, where the CHANGELOG holds no part C entry.
|
||||
// PRC the CHANGELOG lists part C: under ### Fixed one group names each of
|
||||
// the eight behaviour tags exactly once, one bullet per tag, each
|
||||
// ending with the review finding it fixes; ### Doors holds the part C
|
||||
// two-way line; ### Known issues holds one decision-137 bullet that
|
||||
// names the guard's quoted-value hole with each of the eight commands
|
||||
// measured in the plan's Context, "decision 137" and "its own round".
|
||||
// No bullet of this round may hold a string an earlier prose test counts.
|
||||
// The helpers below are copied, never imported from an earlier prose test.
|
||||
// Fix the SOURCE a pin guards, not the pin.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { existsSync, readFileSync } from 'node:fs';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
|
||||
/** A repo file's text; the file must exist. */
|
||||
function read(rel) {
|
||||
assert.ok(existsSync(join(ROOT, rel)), `missing file ${rel}`);
|
||||
return readFileSync(join(ROOT, rel), 'utf8');
|
||||
}
|
||||
|
||||
/** Whitespace (line breaks too) collapsed to one space. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-prose.test.mjs (flat).
|
||||
const flat = (s) => s.replace(/\s+/g, ' ').trim();
|
||||
|
||||
// Part C stands in the same Unreleased section as the earlier parts. The
|
||||
// section is found by its title: `## Unreleased — <title>` before the release,
|
||||
// `## vX.Y.Z — <date> — <title>` after it, exactly once, up to the next `## `.
|
||||
const CHANGELOG_TITLE = "The annotating skill, the run coupling and P10's model-free part";
|
||||
const CHANGELOG_HEADING = new RegExp(`^## (?:Unreleased|v\\d+\\.\\d+\\.\\d+ — \\d{4}-\\d{2}-\\d{2}) — ${CHANGELOG_TITLE.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}$`);
|
||||
|
||||
/** The lines of the round's CHANGELOG section, heading excluded. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-prose.test.mjs (changelogSection).
|
||||
function changelogSection() {
|
||||
const lines = read('CHANGELOG.md').split('\n');
|
||||
const at = [];
|
||||
lines.forEach((l, i) => { if (CHANGELOG_HEADING.test(l.replace(/\s+$/, ''))) at.push(i); });
|
||||
assert.equal(at.length, 1, `CHANGELOG.md must hold the "${CHANGELOG_TITLE}" section exactly once (found ${at.length})`);
|
||||
let end = lines.length;
|
||||
for (let i = at[0] + 1; i < lines.length; i++) if (/^## /.test(lines[i])) { end = i; break; }
|
||||
return lines.slice(at[0] + 1, end);
|
||||
}
|
||||
|
||||
/** The lines of one `### ` subsection of the round's CHANGELOG section, heading excluded. */
|
||||
function subsection(heading) {
|
||||
const lines = changelogSection();
|
||||
const s = lines.findIndex((l) => l.replace(/\s+$/, '') === heading);
|
||||
assert.ok(s !== -1, `the "${CHANGELOG_TITLE}" section of CHANGELOG.md must have "${heading}"`);
|
||||
let end = lines.length;
|
||||
for (let i = s + 1; i < lines.length; i++) if (/^### /.test(lines[i])) { end = i; break; }
|
||||
return lines.slice(s + 1, end);
|
||||
}
|
||||
|
||||
/** The bullets of one `### ` subsection of the round's CHANGELOG section, each flattened to one line. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-prose.test.mjs (changelogBullets).
|
||||
function changelogBullets(heading) {
|
||||
const out = [];
|
||||
for (const l of subsection(heading)) {
|
||||
if (/^- /.test(l)) out.push(l);
|
||||
else if (out.length && l.trim() !== '' && /^\s/.test(l)) out[out.length - 1] += ` ${l}`;
|
||||
else if (l.trim() !== '') out.push('');
|
||||
}
|
||||
return out.filter((b) => b !== '').map(flat);
|
||||
}
|
||||
|
||||
const PRC_GROUP = '**Fix round 4.5b2, part C (decisions 135 and 137)**';
|
||||
// The round's behaviour tags with the review finding each fixes (first 8 hex of
|
||||
// the finding id, written out here from the plan's Context table). PRC itself,
|
||||
// the CHANGELOG step's own tag, is in no count.
|
||||
const PRC_FINDINGS = [
|
||||
['DUP', 'e57259bf'],
|
||||
['RSP', 'de594275'],
|
||||
['NRL', 'fd5a6e18'],
|
||||
['SYM', 'd81f3c45'],
|
||||
['WHD', '561d6f11'],
|
||||
['AMB', 'df9f9870'],
|
||||
['CLU', 'b5ae9798'],
|
||||
['MMG', '73e1d174'],
|
||||
];
|
||||
// Strings the earlier prose tests count in this section: none may stand in a
|
||||
// bullet of this round.
|
||||
const COUNTED = ['decision 88', 'decision 89', 'decision 90', 'decision 91', 'decision 107', 'decision 122',
|
||||
'decision 123', 'decision 124', "'90s", "bash -c $'", "<<$'\\t'", 'VOYAGE_ARGS_EOF_7F3C9E', '6.5.0',
|
||||
'`review_version` 1.2', 'ref the session can move', 'unchanged caller',
|
||||
'(TOK)', '(HDG)', '(CKI)', '(DLC)', '(CQW)', '(PQO)', '(CSG)', '(SLT)', '(ESC)', '(CTB)',
|
||||
'(RXT)', '(CNT)', '(SFX)', '(ESR)', '(EVL)', '(SIS)', '(HDR)',
|
||||
'(ANS)', '(RLS)', '(RAP)', '(V12)', '(PRX)', '(RND)', '(CAP)', '(CRY)', '(FXL)', '(POL)', '(ENG)', '(CMD)',
|
||||
'(DPR)', '(CAT)', '(MEV)', '(RMP)', '(PXB)', '(FN1)'];
|
||||
// The eight commands the plan's Context measured as giving no question (decision
|
||||
// 137): seven from critic rounds 1-2 and the risk assessor's Phase 8 form.
|
||||
const MEASURED = [
|
||||
'`git tag -m "Release one" v1.0`',
|
||||
'`git -C "/tmp/a b" push origin main`',
|
||||
'`git -c "x.y=a b" push origin main`',
|
||||
'`echo "a b"#; git push origin main`',
|
||||
"`git tag -m '' v1.0`",
|
||||
'`git -C ~/"My Repos"/x merge --no-ff feat`',
|
||||
'`npm --workspace "$WS" publish`',
|
||||
'`git -C "<path with a space>" merge --no-ff feat`',
|
||||
];
|
||||
const DOOR = "- two-way: fix round 4.5b2 part C (decisions 135, 137) — a revert restores part B's behaviour.";
|
||||
|
||||
/** Part C's group under `### Fixed`: its intro text and its bullets, each flattened. */
|
||||
function fixedGroup() {
|
||||
const body = subsection('### Fixed');
|
||||
const at = body.filter((l) => l.replace(/\s+$/, '') === PRC_GROUP).length;
|
||||
assert.equal(at, 1, `### Fixed must hold the group ${PRC_GROUP} exactly once (found ${at})`);
|
||||
const start = body.findIndex((l) => l.replace(/\s+$/, '') === PRC_GROUP);
|
||||
const intro = [];
|
||||
const bullets = [];
|
||||
for (const l of body.slice(start + 1)) {
|
||||
if (/^\*\*[^*].*\*\*\s*$/.test(l)) break;
|
||||
if (/^- /.test(l)) bullets.push(l);
|
||||
else if (bullets.length && l.trim() !== '' && /^\s/.test(l)) bullets[bullets.length - 1] += ` ${l}`;
|
||||
else if (!bullets.length && l.trim() !== '') intro.push(l);
|
||||
else if (bullets.length && l.trim() !== '') break;
|
||||
}
|
||||
return { intro: flat(intro.join(' ')), bullets: bullets.map(flat) };
|
||||
}
|
||||
|
||||
/** How often `needle` stands in `text`. */
|
||||
const occurrences = (text, needle) => text.split(needle).length - 1;
|
||||
|
||||
test('[PRC] the part C group names each of the eight behaviour tags exactly once', () => {
|
||||
const { intro, bullets } = fixedGroup();
|
||||
assert.ok(intro.includes('`f1f5ab5`'), `the group's intro names the red base f1f5ab5: ${intro}`);
|
||||
assert.ok(intro.includes('`tests/fix-round-4-5b2-c-red.sh`'), `the group's intro names tests/fix-round-4-5b2-c-red.sh: ${intro}`);
|
||||
assert.equal(PRC_FINDINGS.length, 8, 'Control: eight behaviour tags');
|
||||
const text = [intro, ...bullets].join('\n');
|
||||
for (const [tag] of PRC_FINDINGS) {
|
||||
assert.equal(occurrences(text, `(${tag})`), 1, `the group must name (${tag}) exactly once; found ${occurrences(text, `(${tag})`)}`);
|
||||
assert.equal(bullets.filter((b) => b.includes(`(${tag})`)).length, 1, `exactly one bullet of the group is marked (${tag})`);
|
||||
}
|
||||
assert.equal(bullets.length, PRC_FINDINGS.length, `the group holds one bullet per tag: found ${bullets.length} bullets`);
|
||||
assert.equal(occurrences(text, '(PRC)'), 0, 'the group does not name PRC, the CHANGELOG step\'s own tag');
|
||||
for (const b of bullets) {
|
||||
assert.match(b, /^- \*\*.+?\*\* \([A-Z0-9]{3}\)\./, `each bullet opens with a bold headline and its tag: ${b}`);
|
||||
for (const s of COUNTED) assert.ok(!b.includes(s), `no bullet of the group may hold ${JSON.stringify(s)}: ${b}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('[PRC] each part C bullet names its review finding', () => {
|
||||
const { bullets } = fixedGroup();
|
||||
for (const [tag, id] of PRC_FINDINGS) {
|
||||
const hits = bullets.filter((b) => b.includes(`(${tag})`));
|
||||
assert.equal(hits.length, 1, `exactly one bullet of the group is marked (${tag}); found ${hits.length}`);
|
||||
assert.ok(hits[0].endsWith(`It names review finding \`${id}\`.`),
|
||||
`the (${tag}) bullet must end "It names review finding \`${id}\`.": ${hits[0]}`);
|
||||
assert.equal(bullets.filter((b) => b.includes(id)).length, 1, `review finding ${id} stands in exactly one bullet of the group`);
|
||||
}
|
||||
// The CLU bullet says its quoted -m half stays "no run evidence" (decision 137).
|
||||
const clu = bullets.filter((b) => b.includes('(CLU)'))[0];
|
||||
assert.ok(clu.includes('`-m`'), `the (CLU) bullet names the quoted \`-m\` value: ${clu}`);
|
||||
assert.ok(clu.includes('no run evidence'), `the (CLU) bullet says the quoted -m half stays "no run evidence": ${clu}`);
|
||||
assert.ok(clu.includes('decision 137'), `the (CLU) bullet names decision 137: ${clu}`);
|
||||
// Control: no fixed finding is listed as open.
|
||||
const known = changelogBullets('### Known issues');
|
||||
for (const [, id] of PRC_FINDINGS) {
|
||||
assert.deepEqual(known.filter((b) => b.includes(id)), [], `Control: the fixed finding ${id} must not stand under ### Known issues`);
|
||||
}
|
||||
});
|
||||
|
||||
test('[PRC] the Doors section has the part C two-way line', () => {
|
||||
const doors = changelogBullets('### Doors');
|
||||
const hits = doors.filter((b) => b.startsWith('- two-way: fix round 4.5b2 part C'));
|
||||
assert.deepEqual(hits, [DOOR], `### Doors must hold exactly the one part C line ${DOOR}`);
|
||||
for (const s of COUNTED) assert.ok(!hits[0].includes(s), `the door must not hold ${JSON.stringify(s)}, a string an earlier prose test counts`);
|
||||
// Control: every door bullet of the section is a door line.
|
||||
for (const b of doors) assert.match(b, /^- (one-way|two-way): \S/, `Control: every ### Doors bullet is a door line: ${b}`);
|
||||
});
|
||||
|
||||
test('[PRC] the known issues hold one decision-137 guard bullet with each of the eight measured commands, "decision 137" and "its own round"', () => {
|
||||
const known = changelogBullets('### Known issues');
|
||||
const hits = known.filter((b) => b.includes('(part C, decision 137)'));
|
||||
assert.equal(hits.length, 1, `### Known issues must hold exactly one bullet marked (part C, decision 137); found ${hits.length}`);
|
||||
const b = hits[0];
|
||||
assert.equal(MEASURED.length, 8, 'Control: eight measured commands');
|
||||
for (const cmd of MEASURED) {
|
||||
assert.ok(b.includes(cmd), `the decision-137 bullet must name the measured command ${cmd}: ${b}`);
|
||||
}
|
||||
assert.ok(b.includes('decision 137'), `the decision-137 bullet must hold "decision 137": ${b}`);
|
||||
assert.ok(b.includes('its own round'), `the decision-137 bullet must hold "its own round": ${b}`);
|
||||
for (const s of COUNTED) assert.ok(!b.includes(s), `the decision-137 bullet must not hold ${JSON.stringify(s)}: ${b}`);
|
||||
});
|
||||
212
tests/lib/fix-round-4-5b2-c-scripts.test.mjs
Normal file
212
tests/lib/fix-round-4-5b2-c-scripts.test.mjs
Normal file
|
|
@ -0,0 +1,212 @@
|
|||
// tests/lib/fix-round-4-5b2-c-scripts.test.mjs
|
||||
// The form of fix round 4.5b2 part C's shell scripts and mutation module (plan
|
||||
// .claude/plans/trekplan-2026-11-03-review-gate-4-5b2-fixes.md, Step 7): the
|
||||
// red wrapper, the mutations and the round's end-to-end run.
|
||||
// Untagged on purpose and listed in no red wrapper's FILES: these checks are
|
||||
// not a part-C fix.
|
||||
// - each script exists, is ASCII only, passes /bin/bash -n and uses no
|
||||
// bash 4 form; the mutation module exists and is ASCII only;
|
||||
// - the red wrapper hands the shared red script the base f1f5ab5, exactly
|
||||
// one --other-ids, whose words are the IDS= words of the seven earlier
|
||||
// wrappers in order (4-5a, 4-5b, part 2, part 3, part 4, 4-5b2-a,
|
||||
// 4-5b2-b), the round's nine ids, the part-C mutation module and exactly
|
||||
// the seven tagged test files of this round (run from a temp root next to
|
||||
// a stub red-round.sh that prints its arguments, the method of
|
||||
// tests/lib/fix-round-4-5b2-b-scripts.test.mjs);
|
||||
// - with any earlier wrapper lacking its IDS= line, the red wrapper exits 2
|
||||
// with a setup failure and never runs the shared script;
|
||||
// - red-mutate.mjs list exits 0 and prints no mutation (no case of this
|
||||
// round reads a file under tests/, so none needs one);
|
||||
// - every test file under tests/ that holds one of the round's tags is one
|
||||
// of the seven, and none of the seven is in an earlier wrapper's FILES.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join, relative } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { childEnv } from '../helpers/child-env.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
|
||||
const RED = 'tests/fix-round-4-5b2-c-red.sh';
|
||||
const E2E = 'tests/e2e-fix-round-4-5b2-c.sh';
|
||||
const MUTATIONS = 'tests/fix-round-4-5b2-c-mutations.mjs';
|
||||
const SELF = 'tests/lib/fix-round-4-5b2-c-scripts.test.mjs';
|
||||
const BASE = 'f1f5ab5528a376050dd03b7f7ac43f887596c9b4';
|
||||
const OLDER = [
|
||||
'tests/fix-round-4-5a-red.sh',
|
||||
'tests/fix-round-4-5b-red.sh',
|
||||
'tests/fix-round-4-5b-part-2-red.sh',
|
||||
'tests/fix-round-4-5b-part-3-red.sh',
|
||||
'tests/fix-round-4-5b-part-4-red.sh',
|
||||
'tests/fix-round-4-5b2-a-red.sh',
|
||||
'tests/fix-round-4-5b2-b-red.sh',
|
||||
];
|
||||
const PARTC_IDS = ['DUP', 'RSP', 'NRL', 'SYM', 'WHD', 'AMB', 'CLU', 'MMG', 'PRC'];
|
||||
const PARTC_FILES = [
|
||||
'tests/lib/unique-finding-ids.test.mjs',
|
||||
'tests/lib/review-rounds-stopped-prepare.test.mjs',
|
||||
'tests/lib/instruction-scope-depth.test.mjs',
|
||||
'tests/lib/merge-evidence-writer-head.test.mjs',
|
||||
'tests/hooks/merge-evidence-ambiguous.test.mjs',
|
||||
'tests/hooks/merge-evidence-many.test.mjs',
|
||||
'tests/lib/fix-round-4-5b2-c-prose.test.mjs',
|
||||
];
|
||||
/** The shared script's options that take a value; every other argument after them is a test file. */
|
||||
const VALUE_OPTIONS = ['--base', '--ids', '--other-ids', '--mutations'];
|
||||
|
||||
/** A repo file's bytes; the file must exist. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-scripts.test.mjs (bytes).
|
||||
function bytes(rel) {
|
||||
assert.ok(existsSync(join(ROOT, rel)), `missing file ${rel}`);
|
||||
return readFileSync(join(ROOT, rel));
|
||||
}
|
||||
|
||||
/** The single-quoted words of a wrapper's one <name>='...' line. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-scripts.test.mjs (wordsOf).
|
||||
function wordsOf(rel, name) {
|
||||
const lines = bytes(rel).toString('utf8').split('\n').filter((l) => l.startsWith(`${name}=`));
|
||||
assert.equal(lines.length, 1, `expected one ${name}= line in ${rel}; found ${lines.length}`);
|
||||
const m = lines[0].match(new RegExp(`^${name}='([^']*)'$`));
|
||||
assert.ok(m, `the ${name}= line of ${rel} is not one single-quoted list: ${lines[0]}`);
|
||||
return m[1].split(/\s+/).filter(Boolean);
|
||||
}
|
||||
|
||||
test('part-C scripts: each exists, is ASCII only, passes /bin/bash -n and uses no bash 4 form', () => {
|
||||
for (const rel of [RED, E2E]) {
|
||||
const buf = bytes(rel);
|
||||
assert.equal(buf.findIndex((b) => b > 127), -1, `${rel} is not ASCII only`);
|
||||
const r = spawnSync('/bin/bash', ['-n', join(ROOT, rel)], { encoding: 'utf8', env: childEnv() });
|
||||
assert.equal(r.status, 0, `${rel} does not pass /bin/bash -n: ${r.stderr}`);
|
||||
const text = buf.toString('utf8');
|
||||
for (const bash4 of [/\bdeclare -A\b/, /\bmapfile\b/, /\breadarray\b/, /\|&/]) {
|
||||
assert.doesNotMatch(text, bash4, `${rel} uses ${bash4}, which bash 3.2 lacks`);
|
||||
}
|
||||
}
|
||||
const mod = bytes(MUTATIONS);
|
||||
assert.equal(mod.findIndex((b) => b > 127), -1, `${MUTATIONS} is not ASCII only`);
|
||||
});
|
||||
|
||||
/** A stand-in for the shared script: one ARG: line per argument, exit 0. */
|
||||
const STUB = `#!/bin/bash
|
||||
for a in "$@"; do printf 'ARG:%s\\n' "$a"; done
|
||||
exit 0
|
||||
`;
|
||||
|
||||
/** A temp root with the part-C wrapper, the seven earlier wrappers and the stub as tests/helpers/red-round.sh. */
|
||||
function stubRoot(base) {
|
||||
const dir = join(base, 'root');
|
||||
mkdirSync(join(dir, 'tests', 'helpers'), { recursive: true });
|
||||
for (const rel of [RED, ...OLDER]) writeFileSync(join(dir, rel), bytes(rel));
|
||||
writeFileSync(join(dir, 'tests', 'helpers', 'red-round.sh'), STUB, { mode: 0o755 });
|
||||
return dir;
|
||||
}
|
||||
|
||||
/** The part-C wrapper run from a temp root: status, output and the arguments the stub saw. */
|
||||
function runWrapper(dir) {
|
||||
const r = spawnSync('/bin/bash', [join(dir, RED)], { cwd: tmpdir(), encoding: 'utf8', timeout: 30000, env: childEnv() });
|
||||
const argv = (r.stdout || '').split('\n').filter((l) => l.startsWith('ARG:')).map((l) => l.slice(4));
|
||||
return { status: r.status, out: `${r.stdout}${r.stderr}`, stderr: r.stderr || '', argv };
|
||||
}
|
||||
|
||||
/** The arguments that are neither an option nor an option's value: the test files. */
|
||||
function filesOf(argv) {
|
||||
const files = [];
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
if (VALUE_OPTIONS.includes(argv[i])) { i += 1; continue; }
|
||||
if (argv[i].startsWith('--')) continue;
|
||||
files.push(argv[i]);
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
test('part-C scripts: the red wrapper passes the base, one --other-ids with the seven earlier rounds\' ids in order, its nine ids, its mutations and its seven tagged files', () => {
|
||||
const text = bytes(RED).toString('utf8');
|
||||
const execLines = text.split('\n').filter((l) => /^exec /.test(l));
|
||||
assert.equal(execLines.length, 1, `${RED} must hold one exec line; found ${execLines.length}`);
|
||||
assert.equal(execLines[0].split('--other-ids').length - 1, 1, `${RED}: the exec line must pass --other-ids exactly once: ${execLines[0]}`);
|
||||
|
||||
const base = mkdtempSync(join(tmpdir(), 'partC-red-wrap-'));
|
||||
try {
|
||||
const dir = stubRoot(base);
|
||||
const r = runWrapper(dir);
|
||||
assert.equal(r.status, 0, `${RED} did not run the shared script:\n${r.out}`);
|
||||
const baseAt = r.argv.indexOf('--base');
|
||||
assert.ok(baseAt >= 0, `${RED} passed no --base:\n${r.out}`);
|
||||
assert.equal(r.argv[baseAt + 1], BASE, `${RED}: --base is not ${BASE}\n${r.out}`);
|
||||
const at = r.argv.map((a, i) => (a === '--other-ids' ? i : -1)).filter((i) => i >= 0);
|
||||
assert.equal(at.length, 1, `${RED} handed the shared script --other-ids ${at.length} times:\n${r.out}`);
|
||||
const other = (r.argv[at[0] + 1] || '').split(/\s+/).filter(Boolean);
|
||||
assert.deepEqual(other, OLDER.flatMap((rel) => wordsOf(rel, 'IDS')),
|
||||
`${RED}: --other-ids is not the IDS= words of ${OLDER.join(', then ')}\n${r.out}`);
|
||||
for (const id of PARTC_IDS) {
|
||||
assert.ok(!other.includes(id), `${RED}: the round's tag ${id} is on --other-ids, so it would never be judged\n${r.out}`);
|
||||
}
|
||||
const idsAt = r.argv.indexOf('--ids');
|
||||
assert.ok(idsAt >= 0, `${RED} passed no --ids:\n${r.out}`);
|
||||
assert.deepEqual(r.argv[idsAt + 1].split(/\s+/).filter(Boolean), PARTC_IDS, `${RED}: --ids is not ${PARTC_IDS.join(' ')}\n${r.out}`);
|
||||
const mutAt = r.argv.indexOf('--mutations');
|
||||
assert.ok(mutAt >= 0, `${RED} passed no --mutations:\n${r.out}`);
|
||||
assert.ok(r.argv[mutAt + 1].startsWith('/'), `${RED}: --mutations is not an absolute path: ${r.argv[mutAt + 1]}`);
|
||||
assert.ok(r.argv[mutAt + 1].endsWith(`/${MUTATIONS}`), `${RED}: --mutations is not ${MUTATIONS}: ${r.argv[mutAt + 1]}`);
|
||||
assert.deepEqual([...filesOf(r.argv)].sort(), [...PARTC_FILES].sort(),
|
||||
`${RED}: the test files are not exactly the seven tagged files of this round\n${r.out}`);
|
||||
} finally {
|
||||
rmSync(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('part-C scripts: the red wrapper exits 2 with a setup failure when an earlier wrapper has no IDS= line', () => {
|
||||
for (const missing of OLDER) {
|
||||
const base = mkdtempSync(join(tmpdir(), 'partC-red-noids-'));
|
||||
try {
|
||||
const dir = stubRoot(base);
|
||||
const text = bytes(missing).toString('utf8');
|
||||
writeFileSync(join(dir, missing), text.split('\n').filter((l) => !l.startsWith('IDS=')).join('\n'));
|
||||
const r = runWrapper(dir);
|
||||
assert.equal(r.status, 2, `${RED}: ${missing} without an IDS= line is not exit 2\n${r.out}`);
|
||||
assert.match(r.stderr, /setup failure/, `${RED}: no setup failure line when ${missing} has no IDS= line\n${r.out}`);
|
||||
assert.ok(r.stderr.includes(missing), `${RED}: the setup failure line does not name ${missing}\n${r.out}`);
|
||||
assert.deepEqual(r.argv, [], `${RED} ran the shared script though ${missing} has no IDS= line\n${r.out}`);
|
||||
} finally {
|
||||
rmSync(base, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('part-C scripts: red-mutate.mjs list exits 0 and prints no mutation', () => {
|
||||
assert.ok(existsSync(join(ROOT, MUTATIONS)), `missing file ${MUTATIONS}`);
|
||||
const r = spawnSync(process.execPath, [join(ROOT, 'tests', 'helpers', 'red-mutate.mjs'), 'list', join(ROOT, MUTATIONS)],
|
||||
{ cwd: ROOT, encoding: 'utf8', timeout: 30000, env: childEnv() });
|
||||
assert.equal(r.status, 0, `red-mutate.mjs list ${MUTATIONS} exited ${r.status}:\n${r.stdout}${r.stderr}`);
|
||||
const lines = (r.stdout || '').split('\n').filter((l) => l !== '');
|
||||
assert.deepEqual(lines, [], `red-mutate.mjs list ${MUTATIONS} must print no mutation:\n${r.stdout}${r.stderr}`);
|
||||
});
|
||||
|
||||
/** Every .mjs file under tests/, as a repo-relative path. */
|
||||
// Copied from tests/lib/fix-round-4-5b2-b-scripts.test.mjs (testModules).
|
||||
function testModules(dir = join(ROOT, 'tests')) {
|
||||
const out = [];
|
||||
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = join(dir, e.name);
|
||||
if (e.isDirectory()) out.push(...testModules(p));
|
||||
else if (e.name.endsWith('.mjs')) out.push(relative(ROOT, p).split('\\').join('/'));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
test('part-C scripts: every file holding a part-C tag is one of the seven, and none of the seven is in an earlier wrapper\'s FILES', () => {
|
||||
const tag = new RegExp(`\\[(?:${PARTC_IDS.join('|')})\\]`);
|
||||
const tagged = testModules().filter((rel) => rel !== SELF && tag.test(readFileSync(join(ROOT, rel), 'utf8')));
|
||||
assert.deepEqual([...tagged].sort(), [...PARTC_FILES].sort(),
|
||||
`the files under tests/ that hold a part-C tag are not exactly the seven of ${RED}`);
|
||||
for (const rel of OLDER) {
|
||||
const files = wordsOf(rel, 'FILES');
|
||||
for (const f of PARTC_FILES) {
|
||||
assert.ok(!files.includes(f), `${f} is in the FILES of the earlier wrapper ${rel}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue