test(red-proof): fix round 4.5b2 part B red proof and regression run
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
3cb3958238
commit
d38846f61d
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
7 changed files with 445 additions and 16 deletions
|
|
@ -20,7 +20,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
|
|||
stale approval before the edit; with it, 0, 0, 0, 0 and 3 of 3. Six eval cases
|
||||
`evals/skill-annotating-*`, written, not run.
|
||||
|
||||
**Review gate, 4.5b2 part B (decisions 93-105, 128)**
|
||||
**Review gate, 4.5b2 part B (decisions 93-105, 128-134)**
|
||||
|
||||
Part B makes a WARN open the merge lock and the release check only when the operator has answered every counting MAJOR, and gives `review.md` the fields for it (`review_version` 1.2). Each behaviour has a test seen failing first at the part's red base `96fba75` (`tests/fix-round-4-5b2-b-red.sh` proves it).
|
||||
|
||||
|
|
@ -39,6 +39,9 @@ Part B makes a WARN open the merge lock and the release check only when the oper
|
|||
- **A changed instruction file gets deep review at any depth** (DPR). A changed `CLAUDE.md`, `AGENTS.md` or `.claude/rules/*.md`, `docs/**` included, gets deep review (`instruction-scope.mjs --triage`); the triage table's rows are unchanged.
|
||||
- **Two instruction-file rule keys at MINOR** (CAT). The rule catalogue gains `CLAUDE_MD_DEVIATION` and `CLAUDE_MD_STALE`, both MINOR, category conformance, with no memory route; it holds 17 keys, and the workflow script's tier table holds both.
|
||||
- **The merge question says how the run was tested and how it is undone** (MEV). Before the merge, `/trekexecute` Phase 8 runs `lib/review/merge-evidence.mjs write`, which derives an internal evidence file from the run's artifacts under the git common dir, never tracked. The Bash guard reads it into a "Tested by" and an "Undone by" line of its merge question; its answer stays "ask", and a missing, stale or unreadable file reads "no run evidence" or "not measured". Files older than 30 days are removed.
|
||||
- **A merge after the review cannot bring back the tree from before it** (RMP). `lib/review/reviewed-commit.mjs` checks each commit after the reviewed one against every parent, not only the first, so a merge whose first parent comes from before the review is stale.
|
||||
- **The start of the review range is bound to where the branch left main** (PXB). `lib/review/pre-existing.mjs` finds the fork point on the main branch's first-parent line (`main`, else `master`); a `scope_sha_start` other than that fork point or a later commit reached only through the approval log and `.claude/projects/`, a range that ends on main itself, or a repo with neither branch marks nothing pre-existing.
|
||||
- **An engine finding missing from the review keeps the gate not measured** (FN1). The verdict gate runs the standard-path put-back over the engine payloads beside `review.md` and exits 2 for any MAJOR or BLOCKER whose row is missing from the review, also when `finalized: true` was written by hand; a finalize failure stops `/trekreview` in round 1 as in later rounds.
|
||||
|
||||
### Changed
|
||||
|
||||
|
|
@ -238,6 +241,7 @@ Fix round 4.5a (decisions 57, 67 and 68): the round's 30 findings, where a Voyag
|
|||
- **A MAJOR at an unchanged caller can be marked pre-existing** (part B, Requirements Check 6). The pre-existing mark reads only the lines the range added or changed, so a MAJOR at an unchanged caller that a changed signature broke is marked pre-existing and needs no answer. Lines on both sides of a pure deletion count as changed. A BLOCKER or a security finding there still counts in full.
|
||||
- **`/security-review` re-raises run-range findings in round 2** (part B). The engines stay on the run range from round 2, because `/security-review` takes no range, so it raises its findings from the whole run again. Carry folds each into the previous finding it matches: it is carried, not counted as new, and it stays open while the engine raises it.
|
||||
- **Review answers cannot be recorded with the installed 6.5.0 plugin** (part B). Its `/trekreview` has no answers step, and its review stays at 1.1. This delivery's own review runs on 6.5.0, so before its release check the operator answers its MAJORs with this repo's `lib/review/review-answers.mjs` (`pending`, then `record`), the path `docs/release-runbook.md` gives.
|
||||
- **The fork point is read from a ref the session can move** (part B). `lib/review/pre-existing.mjs` reads the fork point from `refs/heads/main` (or `master`), and that ref can be moved with `git update-ref`, `git branch -f` or a plain commit on main before the branch is made; the Bash guard does not ask about any of them, so the binding is a trace, not a signature.
|
||||
|
||||
### Doors
|
||||
|
||||
|
|
|
|||
32
tests/fix-round-4-5b2-b-mutations.mjs
Normal file
32
tests/fix-round-4-5b2-b-mutations.mjs
Normal file
|
|
@ -0,0 +1,32 @@
|
|||
// tests/fix-round-4-5b2-b-mutations.mjs
|
||||
// The mutations of fix round 4.5b2, part B (plan
|
||||
// .claude/projects/2026-10-31-review-gate-4-5b2, Step 16; decisions 129 and
|
||||
// 132), for the shared red script (tests/helpers/red-round.sh, through
|
||||
// tests/helpers/red-mutate.mjs).
|
||||
// Every other id of the round is new behaviour whose tests fail on an
|
||||
// assertion at the round's start point 96fba75, and kept behaviour is an
|
||||
// untagged keep: test or a Control: assertion inside a red tagged test.
|
||||
// An id named here has tests that pass at 96fba75 and is proven red under its
|
||||
// mutation instead:
|
||||
// - FXS: the main-merge-declined reason list is in
|
||||
// tests/fixtures/jsonl-schemas.md, under the working tree's tests/ that
|
||||
// the script copies into the clone, so it already holds
|
||||
// review-unanswered there; the mutation takes review-unanswered out of
|
||||
// the fixture's list again.
|
||||
// Every span must match exactly once in the clone (96fba75's code with the
|
||||
// working tree's tests/). Data only; never matched by `npm test`.
|
||||
|
||||
export const MUTATIONS = [
|
||||
{
|
||||
key: 'FXS-fixture',
|
||||
id: 'FXS',
|
||||
what: "the fixture's main-merge-declined reasons lose review-unanswered",
|
||||
spans: [
|
||||
{
|
||||
file: 'tests/fixtures/jsonl-schemas.md',
|
||||
find: '`review-stale`, `review-unanswered`, `review-not-measured`',
|
||||
replace: '`review-stale`, `review-not-measured`',
|
||||
},
|
||||
],
|
||||
},
|
||||
];
|
||||
74
tests/fix-round-4-5b2-b-red.sh
Normal file
74
tests/fix-round-4-5b2-b-red.sh
Normal file
|
|
@ -0,0 +1,74 @@
|
|||
#!/usr/bin/env bash
|
||||
# tests/fix-round-4-5b2-b-red.sh
|
||||
# The red proof of fix round 4.5b2, part B (plan
|
||||
# .claude/projects/2026-10-31-review-gate-4-5b2/plan.md, Step 16; decisions
|
||||
# 129 and 132), as a wrapper over the shared red script
|
||||
# tests/helpers/red-round.sh. Every id of the round has tests whose names
|
||||
# start with "[id]"; each must fail on an assertion at the round's start point
|
||||
# 96fba75, except FXS, whose one case reads a fixture under tests/ that the
|
||||
# script copies into the clone and is proven red under its mutation. Kept
|
||||
# behaviour is an untagged keep: test or a Control: assertion inside a red
|
||||
# tagged test.
|
||||
# This wrapper passes the shared script:
|
||||
# --base BASE, the full sha of 96fba75;
|
||||
# --ids IDS (the round's 20 ids);
|
||||
# --other-ids: the ids of fix round 4-5a, of 4-5b part 1, of part 2, of
|
||||
# part 3, of part 4 and of 4.5b2 part A, read with sed from the IDS=
|
||||
# lines of tests/fix-round-4-5a-red.sh, tests/fix-round-4-5b-red.sh,
|
||||
# tests/fix-round-4-5b-part-2-red.sh, tests/fix-round-4-5b-part-3-red.sh,
|
||||
# tests/fix-round-4-5b-part-4-red.sh and tests/fix-round-4-5b2-a-red.sh
|
||||
# and passed as ONE value, in that order (the shared script keeps only
|
||||
# the last --other-ids it is given); a tag on that list is not judged;
|
||||
# --mutations tests/fix-round-4-5b2-b-mutations.mjs (one mutation,
|
||||
# FXS-fixture);
|
||||
# the round's FILES: the twelve new tagged test files. The RLS tests live in
|
||||
# tests/lib/review-answers.test.mjs, so tests/scripts/release-check.test.mjs
|
||||
# is not listed. The scripts test is untagged and not listed.
|
||||
# No --require-sha256: the round has no SHA-256 test.
|
||||
# Exit 0 all red, 1 a rule broken (the judge names it), 2 a setup failure
|
||||
# (also: no ids on one of the six earlier IDS= lines); never 0 on a setup
|
||||
# failure.
|
||||
# bash 3.2 compatible, ASCII only, no arrays.
|
||||
|
||||
set -u
|
||||
set -f
|
||||
|
||||
cd "$(dirname "$0")/.." || { echo "red script: setup failure: cannot cd to the repo root" >&2; exit 2; }
|
||||
ROOT="$(pwd -P)"
|
||||
|
||||
BASE=96fba75996be4fa295d80d7a42c8e3554f7aaae4
|
||||
IDS='CAT V12 RAP ANS RLS PRX RND CAP CRY FXL POL ENG CMD DPR MEV PRB RMP PXB FN1 FXS'
|
||||
FILES='tests/lib/review-answers.test.mjs tests/lib/review-validator-1-2.test.mjs tests/lib/pre-existing.test.mjs tests/lib/review-rounds.test.mjs tests/lib/review-policy.test.mjs tests/lib/engine-findings-compare.test.mjs tests/lib/claude-md-scope.test.mjs tests/hooks/merge-evidence.test.mjs tests/lib/fix-round-4-5b2-b-prose.test.mjs tests/lib/reviewed-commit-parents.test.mjs tests/lib/pre-existing-bound.test.mjs tests/lib/review-finalized.test.mjs'
|
||||
|
||||
OTHER_A=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5a-red.sh")
|
||||
if [ -z "$OTHER_A" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5a-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_B=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-red.sh")
|
||||
if [ -z "$OTHER_B" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_C=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-2-red.sh")
|
||||
if [ -z "$OTHER_C" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-2-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_D=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-3-red.sh")
|
||||
if [ -z "$OTHER_D" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-3-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_E=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b-part-4-red.sh")
|
||||
if [ -z "$OTHER_E" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b-part-4-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
OTHER_F=$(sed -n "s/^IDS='\(.*\)'\$/\1/p" "$ROOT/tests/fix-round-4-5b2-a-red.sh")
|
||||
if [ -z "$OTHER_F" ]; then
|
||||
echo "red script: setup failure: no ids on the IDS= line of $ROOT/tests/fix-round-4-5b2-a-red.sh" >&2
|
||||
exit 2
|
||||
fi
|
||||
|
||||
exec /bin/bash "$ROOT/tests/helpers/red-round.sh" --base "$BASE" --ids "$IDS" --other-ids "$OTHER_A $OTHER_B $OTHER_C $OTHER_D $OTHER_E $OTHER_F" --mutations "$ROOT/tests/fix-round-4-5b2-b-mutations.mjs" $FILES
|
||||
42
tests/fix-round-4-5b2-b-regression.sh
Normal file
42
tests/fix-round-4-5b2-b-regression.sh
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
#!/usr/bin/env bash
|
||||
# tests/fix-round-4-5b2-b-regression.sh
|
||||
# The regression run of fix round 4.5b2, part B (plan
|
||||
# .claude/projects/2026-10-31-review-gate-4-5b2/plan.md, Step 13): the 4-5b
|
||||
# red proof (tests/fix-round-4-5b-red.sh, decision 128: this round updated
|
||||
# one assertion of the protected 4-5b prose test), then the end to end of
|
||||
# 4.5a (tests/e2e-fix-round-4-5a.sh), of 4.5b part 4
|
||||
# (tests/e2e-fix-round-4-5b-part-4.sh) and of 4.5b2 part A
|
||||
# (tests/e2e-fix-round-4-5b2-a.sh). The round's own red proof and end to end
|
||||
# run as their own Verification lines; the criteria runner caps one command
|
||||
# at 900 s.
|
||||
# All four run, in that order; the script prints each exit and exits with the
|
||||
# first non-zero status, or 0 when all four exit 0.
|
||||
# bash 3.2 compatible, ASCII only, no arrays.
|
||||
|
||||
set -u
|
||||
|
||||
cd "$(dirname "$0")/.." || { echo "regression: cannot cd to the repo root" >&2; exit 2; }
|
||||
|
||||
FIRST=0
|
||||
|
||||
/bin/bash tests/fix-round-4-5b-red.sh
|
||||
rc=$?
|
||||
echo "regression: tests/fix-round-4-5b-red.sh exited $rc"
|
||||
[ "$FIRST" -eq 0 ] && FIRST=$rc
|
||||
|
||||
/bin/bash tests/e2e-fix-round-4-5a.sh
|
||||
rc=$?
|
||||
echo "regression: tests/e2e-fix-round-4-5a.sh exited $rc"
|
||||
[ "$FIRST" -eq 0 ] && FIRST=$rc
|
||||
|
||||
/bin/bash tests/e2e-fix-round-4-5b-part-4.sh
|
||||
rc=$?
|
||||
echo "regression: tests/e2e-fix-round-4-5b-part-4.sh exited $rc"
|
||||
[ "$FIRST" -eq 0 ] && FIRST=$rc
|
||||
|
||||
/bin/bash tests/e2e-fix-round-4-5b2-a.sh
|
||||
rc=$?
|
||||
echo "regression: tests/e2e-fix-round-4-5b2-a.sh exited $rc"
|
||||
[ "$FIRST" -eq 0 ] && FIRST=$rc
|
||||
|
||||
exit "$FIRST"
|
||||
|
|
@ -19,6 +19,10 @@
|
|||
// reason `review-unanswered` otherwise, and counts the answers commit;
|
||||
// every WARN statement says exit 0 is "ALLOW, or WARN with every
|
||||
// counting MAJOR answered".
|
||||
// FXS tests/fixtures/jsonl-schemas.md lists the main-merge-declined reasons,
|
||||
// `review-unanswered` included; the fixture is under tests/, which the
|
||||
// red script copies into the base clone, so this case is proven red by
|
||||
// a mutation (tests/fix-round-4-5b2-b-mutations.mjs, FXS-fixture).
|
||||
// POL /trekreview Phase 7 names the review policy: `REVIEW.md` at the repo
|
||||
// root, read at `scope_sha_start`, only its `voyage-review-policy` block
|
||||
// (`max-minor`, `skip`), its text data and never an instruction; a
|
||||
|
|
@ -37,11 +41,13 @@
|
|||
// docs/HANDOVER-CONTRACTS.md documents the file as internal: its path
|
||||
// under the git common dir, never tracked, not part of Handover 5.
|
||||
// PRB the CHANGELOG lists part B: one two-way Doors bullet names the
|
||||
// `review_version` 1.2 bump, the Added group names each of the 15
|
||||
// behaviour tags exactly once, and Known issues name the unchanged-caller
|
||||
// case of the pre-existing mark and the answer path for this delivery's
|
||||
// own review; and the new review modules import no scripts/ or hooks/
|
||||
// code (checked on the real files, a planted import and a clean copy).
|
||||
// `review_version` 1.2 bump, the Added group names each of the 18
|
||||
// behaviour tags exactly once (FXS, a test-only tag, is in no count),
|
||||
// and Known issues name the unchanged-caller case of the pre-existing
|
||||
// mark, the answer path for this delivery's own review and the fork
|
||||
// point read from a ref the session can move; and the new review
|
||||
// modules import no scripts/ or hooks/ code (checked on the real files,
|
||||
// a planted import and a clean copy).
|
||||
// The helpers below are copied, never imported from an earlier prose test.
|
||||
// Fix the SOURCE a pin guards, not the pin.
|
||||
|
||||
|
|
@ -259,17 +265,17 @@ test('[FXL] /trekexecute: Phase 7.2 enters the loop on review-unanswered: fixed-
|
|||
assert.ok(audit.includes('`docs(voyage): review answers (round {R})`'), 'the audit\'s process commits name the review answers commit');
|
||||
});
|
||||
|
||||
test('[FXL] tests/fixtures/jsonl-schemas.md lists the main-merge-declined reasons, review-unanswered included, after the field table', () => {
|
||||
test('[FXS] tests/fixtures/jsonl-schemas.md lists the main-merge-declined reasons, review-unanswered included, after the field table', () => {
|
||||
const text = read('tests/fixtures/jsonl-schemas.md');
|
||||
const lines = text.split('\n');
|
||||
const table = lines.filter((l) => l.startsWith('|'));
|
||||
assert.ok(!table.some((l) => l.includes('review-unanswered')), 'the field table itself does not hold the reason list');
|
||||
const hits = sentences(text).filter((s) => s.includes('main-merge-declined') && s.includes('payload.reason'));
|
||||
assert.equal(hits.length, 1, `one sentence lists the main-merge-declined payload.reason words: ${hits.join(' | ')}`);
|
||||
assert.equal(hits.length, 1, `one sentence lists the main-merge-declined payload.reason words (${hits.length} found)`);
|
||||
for (const word of literalReasons()) {
|
||||
assert.ok(hits[0].includes(`\`${word}\``), `the reason list names ${word}: ${hits[0]}`);
|
||||
assert.ok(hits[0].includes(`\`${word}\``), `the reason list names ${word}: ${hits[0].slice(0, 200)}`);
|
||||
}
|
||||
assert.ok(hits[0].includes('`review-unanswered`'), `the reason list names review-unanswered: ${hits[0]}`);
|
||||
assert.ok(hits[0].includes('`review-unanswered`'), `the reason list names review-unanswered: ${hits[0].slice(0, 200)}`);
|
||||
const tableEnd = lines.findLastIndex((l, i) => l.startsWith('|') && i < lines.indexOf('## Field-allowlist input for Step 11'));
|
||||
const at = lines.findIndex((l) => l.includes('main-merge-declined') && l.includes('payload.reason'));
|
||||
assert.ok(at > tableEnd, 'the reason list stands after the field table');
|
||||
|
|
@ -487,9 +493,11 @@ function changelogBullets(heading) {
|
|||
return out.filter((b) => b !== '').map(flat);
|
||||
}
|
||||
|
||||
const PRB_GROUP = '**Review gate, 4.5b2 part B (decisions 93-105, 128)**';
|
||||
// The round's behaviour tags, every tag of this round but PRB itself.
|
||||
const PRB_TAGS = ['ANS', 'RLS', 'RAP', 'V12', 'PRX', 'RND', 'CAP', 'CRY', 'FXL', 'POL', 'ENG', 'CMD', 'DPR', 'CAT', 'MEV'];
|
||||
const PRB_GROUP = '**Review gate, 4.5b2 part B (decisions 93-105, 128-134)**';
|
||||
// The round's behaviour tags, every tag of this round but PRB itself and FXS,
|
||||
// a test-only tag (the fixture case, proven red by a mutation).
|
||||
const PRB_TAGS = ['ANS', 'RLS', 'RAP', 'V12', 'PRX', 'RND', 'CAP', 'CRY', 'FXL', 'POL', 'ENG', 'CMD', 'DPR', 'CAT', 'MEV',
|
||||
'RMP', 'PXB', 'FN1'];
|
||||
// Strings the earlier prose tests count in this section (plan Gotchas): none
|
||||
// may stand in a bullet of this round.
|
||||
const COUNTED = ['decision 88', 'decision 89', 'decision 90', 'decision 91', 'decision 107', 'decision 122',
|
||||
|
|
@ -535,11 +543,11 @@ test('[PRB] one Doors bullet names the review_version 1.2 bump and is two-way',
|
|||
for (const b of doors) assert.match(b, /^- (one-way|two-way): \S/, `Control: every ### Doors bullet is a door line: ${b}`);
|
||||
});
|
||||
|
||||
test('[PRB] the Added group names each of the 15 behaviour tags exactly once', () => {
|
||||
test('[PRB] the Added group names each of the 18 behaviour tags exactly once', () => {
|
||||
const { intro, bullets } = addedGroup();
|
||||
assert.ok(intro.includes('`96fba75`'), `the group's intro names the red base 96fba75: ${intro}`);
|
||||
assert.ok(intro.includes('`tests/fix-round-4-5b2-b-red.sh`'), `the group's intro names tests/fix-round-4-5b2-b-red.sh: ${intro}`);
|
||||
assert.equal(PRB_TAGS.length, 15, 'Control: 15 behaviour tags');
|
||||
assert.equal(PRB_TAGS.length, 18, 'Control: 18 behaviour tags');
|
||||
const text = [intro, ...bullets].join('\n');
|
||||
for (const tag of PRB_TAGS) {
|
||||
assert.equal(occurrences(text, `(${tag})`), 1, `the group must name (${tag}) exactly once; found ${occurrences(text, `(${tag})`)}`);
|
||||
|
|
@ -567,6 +575,18 @@ test('[PRB] Known issues name the unchanged-caller case and the answer path for
|
|||
}
|
||||
});
|
||||
|
||||
test('[PRB] Known issues hold one bullet saying the fork point is read from a ref the session can move', () => {
|
||||
const known = changelogBullets('### Known issues');
|
||||
const hits = known.filter((b) => b.includes('ref the session can move'));
|
||||
assert.equal(hits.length, 1, `### Known issues must hold exactly one bullet naming a ref the session can move; found ${hits.length}`);
|
||||
for (const s of ['git update-ref', 'git branch -f', 'commit on main']) {
|
||||
assert.ok(hits[0].includes(s), `the fork-point bullet names ${s}: ${hits[0]}`);
|
||||
}
|
||||
assert.ok(!hits[0].includes('unchanged caller') && !hits[0].includes('6.5.0'),
|
||||
`the fork-point bullet is neither the unchanged-caller nor the 6.5.0 bullet: ${hits[0]}`);
|
||||
for (const s of COUNTED) assert.ok(!hits[0].includes(s), `the fork-point bullet must not hold ${JSON.stringify(s)}: ${hits[0]}`);
|
||||
});
|
||||
|
||||
// The new lib files of part B; lib never imports scripts/ or hooks/ code.
|
||||
const NEW_LIB = [
|
||||
'lib/review/review-answers.mjs', 'lib/review/pre-existing.mjs', 'lib/review/review-rounds.mjs',
|
||||
|
|
|
|||
257
tests/lib/fix-round-4-5b2-b-scripts.test.mjs
Normal file
257
tests/lib/fix-round-4-5b2-b-scripts.test.mjs
Normal file
|
|
@ -0,0 +1,257 @@
|
|||
// tests/lib/fix-round-4-5b2-b-scripts.test.mjs
|
||||
// The form of fix round 4.5b2 part B's shell scripts and mutation module (plan
|
||||
// .claude/projects/2026-10-31-review-gate-4-5b2, Step 16): the red wrapper,
|
||||
// the mutations, the regression script and the round's end-to-end run.
|
||||
// Untagged on purpose and listed in no red wrapper's FILES: these checks are
|
||||
// not a part-B fix.
|
||||
// - each script exists, is ASCII only, passes /bin/bash -n and uses no
|
||||
// bash 4 form; the mutation module exists and is ASCII only;
|
||||
// - the red wrapper hands the shared red script the base 96fba75, exactly
|
||||
// one --other-ids, whose words are the IDS= words of the six earlier
|
||||
// wrappers in order (4-5a, 4-5b, part 2, part 3, part 4, 4-5b2-a), the
|
||||
// round's twenty ids, the part-B mutation module and exactly the twelve
|
||||
// tagged test files of this round (run from a temp root next to a stub
|
||||
// red-round.sh that prints its arguments, the method of
|
||||
// tests/lib/red-script.test.mjs);
|
||||
// - with any earlier wrapper lacking its IDS= line, the red wrapper exits 2
|
||||
// with a setup failure and never runs the shared script;
|
||||
// - red-mutate.mjs list prints exactly the one mutation `FXS-fixture FXS`
|
||||
// (the FXS case reads a file under tests/, which the red script copies
|
||||
// into the base clone, so only a mutation makes it red there);
|
||||
// - every test file under tests/ that holds one of the round's tags is one
|
||||
// of the twelve, and none of the twelve is in an earlier wrapper's FILES;
|
||||
// - the regression script runs the 4-5b red proof and the 4.5a, part-4 and
|
||||
// part-A end-to-end scripts in that order, prints each exit and exits
|
||||
// with the first non-zero status (run from a temp root with stubs).
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, mkdtempSync, readdirSync, readFileSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join, relative } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { childEnv } from '../helpers/child-env.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
|
||||
const RED = 'tests/fix-round-4-5b2-b-red.sh';
|
||||
const REGRESSION = 'tests/fix-round-4-5b2-b-regression.sh';
|
||||
const E2E = 'tests/e2e-fix-round-4-5b2-b.sh';
|
||||
const MUTATIONS = 'tests/fix-round-4-5b2-b-mutations.mjs';
|
||||
const SELF = 'tests/lib/fix-round-4-5b2-b-scripts.test.mjs';
|
||||
const BASE = '96fba75996be4fa295d80d7a42c8e3554f7aaae4';
|
||||
const OLDER = [
|
||||
'tests/fix-round-4-5a-red.sh',
|
||||
'tests/fix-round-4-5b-red.sh',
|
||||
'tests/fix-round-4-5b-part-2-red.sh',
|
||||
'tests/fix-round-4-5b-part-3-red.sh',
|
||||
'tests/fix-round-4-5b-part-4-red.sh',
|
||||
'tests/fix-round-4-5b2-a-red.sh',
|
||||
];
|
||||
const PARTB_IDS = ['CAT', 'V12', 'RAP', 'ANS', 'RLS', 'PRX', 'RND', 'CAP', 'CRY', 'FXL', 'POL', 'ENG', 'CMD', 'DPR', 'MEV', 'PRB',
|
||||
'RMP', 'PXB', 'FN1', 'FXS'];
|
||||
const PARTB_FILES = [
|
||||
'tests/lib/review-answers.test.mjs',
|
||||
'tests/lib/review-validator-1-2.test.mjs',
|
||||
'tests/lib/pre-existing.test.mjs',
|
||||
'tests/lib/review-rounds.test.mjs',
|
||||
'tests/lib/review-policy.test.mjs',
|
||||
'tests/lib/engine-findings-compare.test.mjs',
|
||||
'tests/lib/claude-md-scope.test.mjs',
|
||||
'tests/hooks/merge-evidence.test.mjs',
|
||||
'tests/lib/fix-round-4-5b2-b-prose.test.mjs',
|
||||
'tests/lib/reviewed-commit-parents.test.mjs',
|
||||
'tests/lib/pre-existing-bound.test.mjs',
|
||||
'tests/lib/review-finalized.test.mjs',
|
||||
];
|
||||
const REGRESSION_RUNS = [
|
||||
'tests/fix-round-4-5b-red.sh',
|
||||
'tests/e2e-fix-round-4-5a.sh',
|
||||
'tests/e2e-fix-round-4-5b-part-4.sh',
|
||||
'tests/e2e-fix-round-4-5b2-a.sh',
|
||||
];
|
||||
/** The shared script's options that take a value; every other argument after them is a test file. */
|
||||
const VALUE_OPTIONS = ['--base', '--ids', '--other-ids', '--mutations'];
|
||||
|
||||
/** A repo file's bytes; the file must exist. */
|
||||
function bytes(rel) {
|
||||
assert.ok(existsSync(join(ROOT, rel)), `missing file ${rel}`);
|
||||
return readFileSync(join(ROOT, rel));
|
||||
}
|
||||
|
||||
/** The single-quoted words of a wrapper's one <name>='...' line. */
|
||||
function wordsOf(rel, name) {
|
||||
const lines = bytes(rel).toString('utf8').split('\n').filter((l) => l.startsWith(`${name}=`));
|
||||
assert.equal(lines.length, 1, `expected one ${name}= line in ${rel}; found ${lines.length}`);
|
||||
const m = lines[0].match(new RegExp(`^${name}='([^']*)'$`));
|
||||
assert.ok(m, `the ${name}= line of ${rel} is not one single-quoted list: ${lines[0]}`);
|
||||
return m[1].split(/\s+/).filter(Boolean);
|
||||
}
|
||||
|
||||
test('part-B scripts: each exists, is ASCII only, passes /bin/bash -n and uses no bash 4 form', () => {
|
||||
for (const rel of [RED, REGRESSION, E2E]) {
|
||||
const buf = bytes(rel);
|
||||
assert.equal(buf.findIndex((b) => b > 127), -1, `${rel} is not ASCII only`);
|
||||
const r = spawnSync('/bin/bash', ['-n', join(ROOT, rel)], { encoding: 'utf8', env: childEnv() });
|
||||
assert.equal(r.status, 0, `${rel} does not pass /bin/bash -n: ${r.stderr}`);
|
||||
const text = buf.toString('utf8');
|
||||
for (const bash4 of [/\bdeclare -A\b/, /\bmapfile\b/, /\breadarray\b/, /\|&/]) {
|
||||
assert.doesNotMatch(text, bash4, `${rel} uses ${bash4}, which bash 3.2 lacks`);
|
||||
}
|
||||
}
|
||||
const mod = bytes(MUTATIONS);
|
||||
assert.equal(mod.findIndex((b) => b > 127), -1, `${MUTATIONS} is not ASCII only`);
|
||||
});
|
||||
|
||||
/** A stand-in for the shared script: one ARG: line per argument, exit 0. */
|
||||
const STUB = `#!/bin/bash
|
||||
for a in "$@"; do printf 'ARG:%s\\n' "$a"; done
|
||||
exit 0
|
||||
`;
|
||||
|
||||
/** A temp root with the part-B wrapper, the six earlier wrappers and the stub as tests/helpers/red-round.sh. */
|
||||
function stubRoot(base) {
|
||||
const dir = join(base, 'root');
|
||||
mkdirSync(join(dir, 'tests', 'helpers'), { recursive: true });
|
||||
for (const rel of [RED, ...OLDER]) writeFileSync(join(dir, rel), bytes(rel));
|
||||
writeFileSync(join(dir, 'tests', 'helpers', 'red-round.sh'), STUB, { mode: 0o755 });
|
||||
return dir;
|
||||
}
|
||||
|
||||
/** The part-B wrapper run from a temp root: status, output and the arguments the stub saw. */
|
||||
function runWrapper(dir) {
|
||||
const r = spawnSync('/bin/bash', [join(dir, RED)], { cwd: tmpdir(), encoding: 'utf8', timeout: 30000, env: childEnv() });
|
||||
const argv = (r.stdout || '').split('\n').filter((l) => l.startsWith('ARG:')).map((l) => l.slice(4));
|
||||
return { status: r.status, out: `${r.stdout}${r.stderr}`, stderr: r.stderr || '', argv };
|
||||
}
|
||||
|
||||
/** The arguments that are neither an option nor an option's value: the test files. */
|
||||
function filesOf(argv) {
|
||||
const files = [];
|
||||
for (let i = 0; i < argv.length; i += 1) {
|
||||
if (VALUE_OPTIONS.includes(argv[i])) { i += 1; continue; }
|
||||
if (argv[i].startsWith('--')) continue;
|
||||
files.push(argv[i]);
|
||||
}
|
||||
return files;
|
||||
}
|
||||
|
||||
test('part-B scripts: the red wrapper passes the base, one --other-ids with the six earlier rounds\' ids in order, its twenty ids, its mutations and its twelve tagged files', () => {
|
||||
const text = bytes(RED).toString('utf8');
|
||||
const execLines = text.split('\n').filter((l) => /^exec /.test(l));
|
||||
assert.equal(execLines.length, 1, `${RED} must hold one exec line; found ${execLines.length}`);
|
||||
assert.equal(execLines[0].split('--other-ids').length - 1, 1, `${RED}: the exec line must pass --other-ids exactly once: ${execLines[0]}`);
|
||||
|
||||
const base = mkdtempSync(join(tmpdir(), 'partB-red-wrap-'));
|
||||
try {
|
||||
const dir = stubRoot(base);
|
||||
const r = runWrapper(dir);
|
||||
assert.equal(r.status, 0, `${RED} did not run the shared script:\n${r.out}`);
|
||||
const baseAt = r.argv.indexOf('--base');
|
||||
assert.ok(baseAt >= 0, `${RED} passed no --base:\n${r.out}`);
|
||||
assert.equal(r.argv[baseAt + 1], BASE, `${RED}: --base is not ${BASE}\n${r.out}`);
|
||||
const at = r.argv.map((a, i) => (a === '--other-ids' ? i : -1)).filter((i) => i >= 0);
|
||||
assert.equal(at.length, 1, `${RED} handed the shared script --other-ids ${at.length} times:\n${r.out}`);
|
||||
const other = (r.argv[at[0] + 1] || '').split(/\s+/).filter(Boolean);
|
||||
assert.deepEqual(other, OLDER.flatMap((rel) => wordsOf(rel, 'IDS')),
|
||||
`${RED}: --other-ids is not the IDS= words of ${OLDER.join(', then ')}\n${r.out}`);
|
||||
for (const id of PARTB_IDS) {
|
||||
assert.ok(!other.includes(id), `${RED}: the round's tag ${id} is on --other-ids, so it would never be judged\n${r.out}`);
|
||||
}
|
||||
const idsAt = r.argv.indexOf('--ids');
|
||||
assert.ok(idsAt >= 0, `${RED} passed no --ids:\n${r.out}`);
|
||||
assert.deepEqual(r.argv[idsAt + 1].split(/\s+/).filter(Boolean), PARTB_IDS, `${RED}: --ids is not ${PARTB_IDS.join(' ')}\n${r.out}`);
|
||||
const mutAt = r.argv.indexOf('--mutations');
|
||||
assert.ok(mutAt >= 0, `${RED} passed no --mutations:\n${r.out}`);
|
||||
assert.ok(r.argv[mutAt + 1].startsWith('/'), `${RED}: --mutations is not an absolute path: ${r.argv[mutAt + 1]}`);
|
||||
assert.ok(r.argv[mutAt + 1].endsWith(`/${MUTATIONS}`), `${RED}: --mutations is not ${MUTATIONS}: ${r.argv[mutAt + 1]}`);
|
||||
assert.deepEqual([...filesOf(r.argv)].sort(), [...PARTB_FILES].sort(),
|
||||
`${RED}: the test files are not exactly the twelve tagged files of this round\n${r.out}`);
|
||||
} finally {
|
||||
rmSync(base, { recursive: true, force: true });
|
||||
}
|
||||
});
|
||||
|
||||
test('part-B scripts: the red wrapper exits 2 with a setup failure when an earlier wrapper has no IDS= line', () => {
|
||||
for (const missing of OLDER) {
|
||||
const base = mkdtempSync(join(tmpdir(), 'partB-red-noids-'));
|
||||
try {
|
||||
const dir = stubRoot(base);
|
||||
const text = bytes(missing).toString('utf8');
|
||||
writeFileSync(join(dir, missing), text.split('\n').filter((l) => !l.startsWith('IDS=')).join('\n'));
|
||||
const r = runWrapper(dir);
|
||||
assert.equal(r.status, 2, `${RED}: ${missing} without an IDS= line is not exit 2\n${r.out}`);
|
||||
assert.match(r.stderr, /setup failure/, `${RED}: no setup failure line when ${missing} has no IDS= line\n${r.out}`);
|
||||
assert.ok(r.stderr.includes(missing), `${RED}: the setup failure line does not name ${missing}\n${r.out}`);
|
||||
assert.deepEqual(r.argv, [], `${RED} ran the shared script though ${missing} has no IDS= line\n${r.out}`);
|
||||
} finally {
|
||||
rmSync(base, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
test('part-B scripts: red-mutate.mjs list prints exactly the mutation `FXS-fixture FXS`', () => {
|
||||
assert.ok(existsSync(join(ROOT, MUTATIONS)), `missing file ${MUTATIONS}`);
|
||||
const r = spawnSync(process.execPath, [join(ROOT, 'tests', 'helpers', 'red-mutate.mjs'), 'list', MUTATIONS],
|
||||
{ cwd: ROOT, encoding: 'utf8', timeout: 30000, env: childEnv() });
|
||||
assert.equal(r.status, 0, `red-mutate.mjs list ${MUTATIONS} exited ${r.status}:\n${r.stdout}${r.stderr}`);
|
||||
const lines = (r.stdout || '').split('\n').filter((l) => l !== '');
|
||||
assert.deepEqual(lines, ['FXS-fixture FXS'],
|
||||
`red-mutate.mjs list ${MUTATIONS} does not print exactly the mutation FXS-fixture FXS:\n${r.stdout}${r.stderr}`);
|
||||
});
|
||||
|
||||
/** Every .mjs file under tests/, as a repo-relative path. */
|
||||
function testModules(dir = join(ROOT, 'tests')) {
|
||||
const out = [];
|
||||
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
||||
const p = join(dir, e.name);
|
||||
if (e.isDirectory()) out.push(...testModules(p));
|
||||
else if (e.name.endsWith('.mjs')) out.push(relative(ROOT, p).split('\\').join('/'));
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
test('part-B scripts: every file holding a part-B tag is one of the twelve, and none of the twelve is in an earlier wrapper\'s FILES', () => {
|
||||
const tag = new RegExp(`\\[(?:${PARTB_IDS.join('|')})\\]`);
|
||||
const tagged = testModules().filter((rel) => rel !== SELF && tag.test(readFileSync(join(ROOT, rel), 'utf8')));
|
||||
assert.deepEqual([...tagged].sort(), [...PARTB_FILES].sort(),
|
||||
`the files under tests/ that hold a part-B tag are not exactly the twelve of ${RED}`);
|
||||
for (const rel of OLDER) {
|
||||
const files = wordsOf(rel, 'FILES');
|
||||
for (const f of PARTB_FILES) {
|
||||
assert.ok(!files.includes(f), `${f} is in the FILES of the earlier wrapper ${rel}`);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
/** A temp root with the regression script and a stub per script it runs, each exiting with its given status. */
|
||||
function regressionRoot(base, codes) {
|
||||
const dir = join(base, 'root');
|
||||
mkdirSync(join(dir, 'tests'), { recursive: true });
|
||||
writeFileSync(join(dir, REGRESSION), bytes(REGRESSION));
|
||||
REGRESSION_RUNS.forEach((rel, i) => {
|
||||
writeFileSync(join(dir, rel), `#!/bin/bash\necho "RAN:${rel}"\nexit ${codes[i]}\n`, { mode: 0o755 });
|
||||
});
|
||||
return dir;
|
||||
}
|
||||
|
||||
test('part-B scripts: the regression script runs the 4-5b red proof and three end-to-end scripts in order and exits with the first non-zero status', () => {
|
||||
for (const [codes, want] of [[[0, 0, 0, 0], 0], [[0, 3, 0, 5], 3], [[4, 0, 0, 0], 4], [[0, 0, 0, 6], 6]]) {
|
||||
const base = mkdtempSync(join(tmpdir(), 'partB-regression-'));
|
||||
try {
|
||||
const dir = regressionRoot(base, codes);
|
||||
const r = spawnSync('/bin/bash', [join(dir, REGRESSION)], { cwd: tmpdir(), encoding: 'utf8', timeout: 30000, env: childEnv() });
|
||||
const out = `${r.stdout}${r.stderr}`;
|
||||
assert.equal(r.status, want, `${REGRESSION} with exits ${codes.join(',')} exited ${r.status}, not ${want}\n${out}`);
|
||||
const ran = (r.stdout || '').split('\n').filter((l) => l.startsWith('RAN:')).map((l) => l.slice(4));
|
||||
assert.deepEqual(ran, REGRESSION_RUNS, `${REGRESSION} did not run all four scripts in order\n${out}`);
|
||||
REGRESSION_RUNS.forEach((rel, i) => {
|
||||
assert.ok((r.stdout || '').includes(`regression: ${rel} exited ${codes[i]}`),
|
||||
`${REGRESSION} printed no "regression: ${rel} exited ${codes[i]}" line\n${out}`);
|
||||
});
|
||||
} finally {
|
||||
rmSync(base, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
|
@ -22,7 +22,7 @@ import { tmpdir } from 'node:os';
|
|||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { computeFindingId, computeFindingIds } from '../../lib/parsers/finding-id.mjs';
|
||||
import { decideOpenFindings } from '../../lib/review/verdict-gate.mjs';
|
||||
const { decideOpenFindings } = await import('../../lib/review/verdict-gate.mjs');
|
||||
import { childEnv } from '../helpers/child-env.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue