feat(review): review rounds carry findings until resolved, one cap of three
This commit is contained in:
parent
1464605e35
commit
635003e8aa
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
13 changed files with 2021 additions and 100 deletions
|
|
@ -306,6 +306,9 @@ prefix). Flow-style `findings: [a, b]` breaks the frontmatter parser. With
|
|||
zero findings write `findings: []` — the empty list is the one inline form
|
||||
the parser reads; a bare `findings:` is null and fails validation.
|
||||
|
||||
You never write `ran`, `finalized`, `round`, `round_base`, `resolved` or
|
||||
`## Answers`; `lib/review/review-finalize.mjs` and `/trekreview` Phase 8.8 do.
|
||||
|
||||
### Body sections (in order)
|
||||
|
||||
1. `# Review: {task}`
|
||||
|
|
|
|||
|
|
@ -2348,10 +2348,12 @@ the run is `stopped`, and the merge stays closed.
|
|||
`{report dir}/review-round-{R}-steps.md`, and stop. A later `--resume`
|
||||
finds the plan as approved and comes back to this review.
|
||||
|
||||
**The cap:** `TREKEXECUTE_MAX_REVIEW_ROUNDS` reviews per run, default **3**
|
||||
(the first review and two fix loops; env-overridable).
|
||||
A BLOCK at the cap goes to the operator: stop, and the report names every
|
||||
BLOCKER still open. Each loop also takes 1 from `iterations_remaining`, which
|
||||
**The cap:** `MAX_REVIEW_ROUNDS` in `lib/review/review-rounds.mjs`, **3**
|
||||
reviews per run (the first and two fix loops), read with
|
||||
`node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-rounds.mjs --cap`; `/trekreview`
|
||||
reads the same constant and refuses a fourth round.
|
||||
A review at the cap with BLOCKERs or MAJORs answered "fixed" still open goes to the operator: stop, and the report names every one.
|
||||
Each loop also takes 1 from `iterations_remaining`, which
|
||||
bounds it a second time (the completion gate's clause 3 counts it).
|
||||
|
||||
Borrowed: the order (run the thing, simplify, review) is Cherny's; a verdict in
|
||||
|
|
@ -3235,7 +3237,7 @@ set from the effort.
|
|||
itself (Phase 7.2). Exit 1 (BLOCK) or 2 (not measured) from the verdict
|
||||
gate keeps the merge into main closed (Phase 8). A BLOCKER is fixed by new
|
||||
numbered plan steps the operator approves, in a loop capped at
|
||||
`TREKEXECUTE_MAX_REVIEW_ROUNDS`, never by improvising. When the lock
|
||||
`MAX_REVIEW_ROUNDS` (`lib/review/review-rounds.mjs`), never by improvising. When the lock
|
||||
opens, the guard asks before the merge, and the executor never asks again.
|
||||
Exit 2 from the step check (K9), the architecture rules (K5) or the
|
||||
plan-file sweep (K6) is "not measured", never green.
|
||||
|
|
|
|||
|
|
@ -155,6 +155,28 @@ Determine the "before" SHA that bounds the review:
|
|||
|
||||
Compute the "after" SHA: `after_sha=$(git rev-parse HEAD)`.
|
||||
|
||||
**Then the round.** Rounds converge, at most `MAX_REVIEW_ROUNDS` (**3**, in
|
||||
`lib/review/review-rounds.mjs`) per run:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-rounds.mjs prepare --project "{project_dir}" --since {before_sha} --json
|
||||
```
|
||||
|
||||
Add `--workflow` when `workflow_substrate == true` and `--dry-run` when
|
||||
`mode == dry-run`. It prints `{round, round_base, previous}`; keep all three for
|
||||
Phases 5 and 7. A later round is one where `{project_dir}/review.md` has this
|
||||
`scope_sha_start` (the same run): `prepare` counts the round from the run's
|
||||
archives, keeps the previous review as `{project_dir}/{previous}`
|
||||
(`review-round-<run>-<n>.md`, `<run>` = the first 12 hex of `before_sha`), and
|
||||
`round_base` is its `scope_sha_end`. Every round it deletes the previous
|
||||
round's `.review-out-*.md` and `.review-engine-payload-*.json` files, so each
|
||||
such file finalize reads was written this round. Exit 1 stops: the round cap is
|
||||
reached (the operator re-plans), or a round 2 or later was asked on
|
||||
`--workflow` (re-run without it). Exit 2 stops "not measured" (the round base
|
||||
is not an ancestor of HEAD, the round count does not match the archives, or an
|
||||
archive differs). Print the reason in one line; no review is written in either
|
||||
case.
|
||||
|
||||
Capture working-tree changes (uncommitted at review time):
|
||||
```bash
|
||||
git diff --name-only "$before_sha".."$after_sha"
|
||||
|
|
@ -406,6 +428,21 @@ Each reviewer prompt includes:
|
|||
- **Brief path** — `{brief_path}` (read on demand; do not inline).
|
||||
- **Rule catalogue** — reference to `lib/review/rule-catalogue.mjs`.
|
||||
|
||||
**From round 2** (`round` ≥ 2 from Phase 3), each reviewer prompt also gets
|
||||
the round input:
|
||||
- the round diff, `git diff {round_base}..{after_sha}`, beside the run's diff;
|
||||
- the previous findings, from the trailing JSON block of
|
||||
`{project_dir}/{previous}`, each with its id, to mark in the reviewer's own
|
||||
JSON block as `previous: [{id, resolved}]` (`resolved: true` only when the
|
||||
round diff fixes it);
|
||||
- the line "you may read the files around it; a previous finding is never
|
||||
re-raised as new; flag every occurrence of a finding class".
|
||||
|
||||
Finalize (Phase 7) carries a previous finding until every reviewer that ran in
|
||||
its round marks it resolved, and an engine finding until its engine ran this
|
||||
round and did not raise it again. The engines (Phase 4.7) stay on the run range
|
||||
(`/security-review` takes no range).
|
||||
|
||||
`brief-conformance-reviewer` additionally receives `sc_evidence_block` from
|
||||
Phase 4.5, pasted verbatim — the command, exit code and first line of output for
|
||||
every Success Criterion. It is the ONLY evidence that agent has about whether a
|
||||
|
|
@ -628,13 +665,30 @@ parser at `lib/util/frontmatter.mjs` does not support flow-style arrays.
|
|||
**Then finalize the review** — one deterministic pass after `fill-finding-ids`:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-finalize.mjs --review "{project_dir}/review.md" --repo . --project "{project_dir}" --round 1
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-finalize.mjs --review "{project_dir}/review.md" --repo . --project "{project_dir}" --round {round}
|
||||
```
|
||||
|
||||
On exit 1 or 2 the file stays as `fill-finding-ids` wrote it; print the reason in one line and continue.
|
||||
From round 2 it also gets the round base and the previous review, as `prepare`
|
||||
printed them, and the carried answers follow it:
|
||||
|
||||
```bash
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-finalize.mjs --review "{project_dir}/review.md" --repo . --project "{project_dir}" --round {round} --round-base {round_base} --previous "{project_dir}/{previous}"
|
||||
node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-answers.mjs carry --review "{project_dir}/review.md" --previous "{project_dir}/{previous}" --repo .
|
||||
```
|
||||
|
||||
In round 1, on exit 1 or 2 the file stays as `fill-finding-ids` wrote it; print the reason in one line and continue.
|
||||
From round 2, finalize exit 1 or 2 stops the review: print the reason and "not measured", and stop. The previous
|
||||
findings reach review.md only through finalize's carry, so the verdict gate reads a round 2+ review without
|
||||
`finalized: true` as exit 2 (`review-not-measured: round 2+ review not finalized`), never green.
|
||||
Finalize writes `round`, `round_base`, `finalized: true`, `ran` (what ran this round, read from the
|
||||
`.review-out-*.md` and `.review-engine-payload-*.json` files) and, from round 2, `resolved` with a
|
||||
`## Resolved since round <n>` section; a carried BLOCKER gets its `### Step R<n>`. `review-answers.mjs carry`
|
||||
writes an `## Answers` row with `Carried from` for each carried MAJOR the operator accepted in an
|
||||
earlier round; on exit 1 or 2 print the reason in one line and continue: no answer is carried.
|
||||
A finding is pre-existing when it sits on a line the range `scope_sha_start..scope_sha_end` did not add or
|
||||
change; finalize marks it from git, the verdict gate re-derives the mark, and a BLOCKER or security finding
|
||||
counts in full whatever the mark.
|
||||
counts in full whatever the mark. A carried finding (`carried_from`) keeps its line from the previous round's end,
|
||||
so it is never pre-existing, whatever the mark.
|
||||
|
||||
**The verdict is in the file (K7).** The frontmatter carries
|
||||
`verdict: ALLOW | WARN | BLOCK`, equal to the trailing JSON block's
|
||||
|
|
|
|||
|
|
@ -436,7 +436,7 @@ missing `exit_code` (C15, `PROGRESS_FAILED_CRITERIA_KEY`).
|
|||
|
||||
**Path conventions:**
|
||||
- Project-dir mode (recommended): `{project_dir}/review.md` (one per review iteration; subsequent runs overwrite atomically).
|
||||
- Multiple review iterations are allowed in the same project; each overwrites the canonical path. Audit trail lives in git history.
|
||||
- Multiple review iterations are allowed in the same project; each overwrites the canonical path. Audit trail lives in git history. Before round 2 and later, the previous `review.md` is kept as `review-round-<run>-<n>.md` beside it (`<run>` = first 12 hex of the run's `scope_sha_start`); those archives are read by the next round and by the gate for a carried answer.
|
||||
|
||||
**Frontmatter schema:**
|
||||
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ If `{project_dir}/architecture/overview.md` exists (typically produced by an opt
|
|||
| `--gates [true\|false\|closed\|open]` | (v3.4.0) Autonomy flag: bare, `true` or `closed` = on; `false` or `open` = off; any other next token is not its value. See `## Autonomy mode` in `docs/operations.md`. `/trekexecute` accepts the flag and pauses on nothing; it adds no question of its own before a merge: the guard asks there (operator decision 2026-10-08). |
|
||||
| `--profile <name>` | (v4.1.0) Profile for this run's resolver; the step helper and the controller stay on opus. Inherited from plan.md frontmatter `profile:` if present. |
|
||||
|
||||
Every mode that executes refuses a plan whose approval in the approval log does not cover its current sha256 (K2). A run ends with the review it starts by itself (`/simplify` → `/trekreview` → verdict gate); a BLOCK or an undecided verdict keeps the merge into main closed. `TREKEXECUTE_MAX_REVIEW_ROUNDS` (default 3) caps the reviews per run.
|
||||
Every mode that executes refuses a plan whose approval in the approval log does not cover its current sha256 (K2). A run ends with the review it starts by itself (`/simplify` → `/trekreview` → verdict gate); a BLOCK or an undecided verdict keeps the merge into main closed. `MAX_REVIEW_ROUNDS` in `lib/review/review-rounds.mjs` (3) caps the reviews per run; `/trekreview` refuses a fourth round.
|
||||
|
||||
## /trekreview modes
|
||||
|
||||
|
|
|
|||
|
|
@ -16,6 +16,30 @@
|
|||
// `scope_sha_end`, so an answer goes stale with the review it answers;
|
||||
// - the row's Record is that record's hash.
|
||||
// A `fixed` row never answers: the finding stays open as fixed-pending.
|
||||
// A carried row (Carried from `review-round-<run>-<n>.md#<old id>`, written by
|
||||
// `carry` from round 2) answers a carried MAJOR only when all of these hold:
|
||||
// - the finding is carried (`carried_from`) and, like the old finding in
|
||||
// the archive, no security finding; the reason is not hollow;
|
||||
// - `<run>` is the first 12 hex of this review's `scope_sha_start`, and `<n>`
|
||||
// is lower than this review's `round` (no `round`: no carried link), so an
|
||||
// answer from another run, or from a later round, is never carried;
|
||||
// - the archive is a regular file beside review.md (read through
|
||||
// `ctx.archived`), of the same run, and holds the old finding, which
|
||||
// matches this one by the carry rule (lib/review/review-rounds.mjs
|
||||
// `sameFinding`);
|
||||
// - the archive's own `## Answers` row for the old id is that answer as
|
||||
// given there (`accepted`, Carried from `-`, the same reason and Record);
|
||||
// - the link is this finding's own carry chain: with `<n>` = round - 1 the
|
||||
// old id is the finding's `carried_from`; with an older `<n>` the run's
|
||||
// archive of round - 1 (its `round` field that number) holds the finding
|
||||
// `carried_from` names, whose own row has the same link, Record and
|
||||
// reason and is itself bound by these rules (one answer never answers
|
||||
// another occurrence of the same class);
|
||||
// - the K2 record with that hash is the latest of kind `review` for
|
||||
// `<review path>#<old id>`, and its sha256 is `answerSha` over the
|
||||
// archive's trailing JSON block, its `scope_sha_end`, the old id and the
|
||||
// reason.
|
||||
// Otherwise the MAJOR is unanswered.
|
||||
// BLOCKER findings are never answered (the gate stops on them before this).
|
||||
//
|
||||
// The answer log is the top-level one: `logPath(answerLogRepo(dir))`, whatever
|
||||
|
|
@ -38,18 +62,27 @@
|
|||
// with its reason), 1 = refused (bad trailer, empty or hollow reason, an id
|
||||
// that is not a counting MAJOR, a BLOCKER, `accepted` on a security
|
||||
// finding), 2 = could not run
|
||||
// node lib/review/review-answers.mjs carry --review <p> --previous <archive>
|
||||
// --repo <d>
|
||||
// after finalize, from round 2: for each carried MAJOR whose previous
|
||||
// review holds a valid `accepted` row, writes its row with the link to the
|
||||
// archive where it was given and that answer's record hash (no K2 record
|
||||
// is appended, nothing is committed); exit 0 = done (`written`, `skipped`),
|
||||
// 1 = refused (the previous review is not this run's earlier archive beside
|
||||
// review.md), 2 = could not run
|
||||
//
|
||||
// The accept shape (an answer bound to the reviewed text, never to a BLOCKER)
|
||||
// follows lib/validators/plan-gate.mjs `acceptOpen`.
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { accessSync, constants as fsConstants, readFileSync, realpathSync, renameSync, writeFileSync } from 'node:fs';
|
||||
import { accessSync, constants as fsConstants, lstatSync, readFileSync, realpathSync, renameSync, writeFileSync } from 'node:fs';
|
||||
import { basename, dirname, join, resolve } from 'node:path';
|
||||
import { fencedBlocks, isJsonBlock } from './findings-schema.mjs';
|
||||
import { getRule } from './rule-catalogue.mjs';
|
||||
import { reviewedSha } from './reviewed-commit.mjs';
|
||||
import { preExistingCheck } from './pre-existing.mjs';
|
||||
import { preExistingCheck, scopeShas } from './pre-existing.mjs';
|
||||
import { archiveName, frontmatterField, runId, sameFinding } from './review-rounds.mjs';
|
||||
import {
|
||||
appendApproval, isHollowAnswer, logPath, normalizeSubject, readLog, verifyLog,
|
||||
} from '../util/approval-log.mjs';
|
||||
|
|
@ -66,6 +99,8 @@ const HEX40_RE = /^[0-9a-f]{40}$/;
|
|||
const ANSWER_FINDING_RE = /^([0-9a-f]{40})(?:#\d+)?$/;
|
||||
const SHA256_RE = /^[0-9a-f]{64}$/;
|
||||
const CARRIED_FROM_RE = /^review-round-[0-9a-f]{12}-\d+\.md#[0-9a-f]{40}$/;
|
||||
const LINK_RE = /^(review-round-([0-9a-f]{12})-(\d+)\.md)#([0-9a-f]{40})$/;
|
||||
const ARCHIVE_NAME_RE = /^review-round-([0-9a-f]{12})-(\d+)\.md$/;
|
||||
|
||||
const sha256 = (s) => createHash('sha256').update(s).digest('hex');
|
||||
const isMajor = (f) => String(f?.severity ?? '').toUpperCase() === 'MAJOR';
|
||||
|
|
@ -213,19 +248,96 @@ function latestReviewRecord(records, subject) {
|
|||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The archived-review reader for `dir` (the review's folder): an archive name
|
||||
* `review-round-<run>-<n>.md` gives its text when it is a regular file there,
|
||||
* anything else gives null.
|
||||
* @param {string} dir
|
||||
* @returns {(name: string) => string|null}
|
||||
*/
|
||||
export function archivedReader(dir) {
|
||||
return (name) => {
|
||||
if (!dir || !ARCHIVE_NAME_RE.test(String(name))) return null;
|
||||
const p = join(dir, name);
|
||||
try {
|
||||
if (!lstatSync(p).isFile()) return null;
|
||||
return readFileSync(p, 'utf8');
|
||||
} catch { return null; }
|
||||
};
|
||||
}
|
||||
|
||||
/** Is `row` the answer `answer` to finding `id`, bound by the latest K2 record to `text` as it is? */
|
||||
function directBound(text, id, row, answer, { records, reviewSubject }) {
|
||||
const blockSha = findingsBlockSha(text);
|
||||
const scopeShaEnd = reviewedSha(text) || null;
|
||||
if (!row || !blockSha || !scopeShaEnd || !reviewSubject) return false;
|
||||
const reason = normalizeReason(row.reason);
|
||||
const rec = latestReviewRecord(Array.isArray(records) ? records : [], `${reviewSubject}#${id}`);
|
||||
return Boolean(rec) && rec.hash === row.record
|
||||
&& rec.sha256 === answerSha({ blockSha, scopeShaEnd, findingId: id, answer, reason });
|
||||
}
|
||||
|
||||
/** An accepted reason that is a reason: not empty, not hollow. */
|
||||
const realReason = (row) => {
|
||||
const reason = normalizeReason(row?.reason);
|
||||
return reason !== '' && !isHollowAnswer(reason);
|
||||
};
|
||||
|
||||
/**
|
||||
* Is the carried row `row` of finding `f` in review `text` bound to an
|
||||
* accepted answer in this run's earlier archive (the rule in the header)?
|
||||
*/
|
||||
function carriedBound(text, f, row, { records, reviewSubject, archived }) {
|
||||
if (!row || row.answer !== 'accepted' || typeof archived !== 'function' || !reviewSubject) return false;
|
||||
if (typeof f?.carried_from !== 'string' || isSecurityFinding(f) || !realReason(row)) return false;
|
||||
const m = LINK_RE.exec(String(row.carriedFrom));
|
||||
if (!m) return false;
|
||||
const [, name, run, n, oldId] = m;
|
||||
const { start } = scopeShas(text);
|
||||
if (!start || run !== runId(start)) return false;
|
||||
const round = frontmatterField(text, 'round');
|
||||
if (!/^\d+$/.test(round ?? '') || Number(n) >= Number(round)) return false;
|
||||
const old = archived(name);
|
||||
if (typeof old !== 'string') return false;
|
||||
const oldStart = scopeShas(old).start;
|
||||
if (!oldStart || runId(oldStart) !== run) return false;
|
||||
const oldF = (payloadOf(old)?.findings ?? []).find((x) => x && typeof x === 'object' && x.id === oldId);
|
||||
if (!oldF || isSecurityFinding(oldF) || !sameFinding(oldF, f)) return false;
|
||||
const oldRow = parseAnswers(bodyOf(old)).rows.find((r) => r.id === oldId);
|
||||
if (!oldRow || oldRow.answer !== 'accepted' || oldRow.carriedFrom !== '-' || oldRow.record !== row.record
|
||||
|| normalizeReason(oldRow.reason) !== normalizeReason(row.reason)) return false;
|
||||
if (!directBound(old, oldId, oldRow, 'accepted', { records, reviewSubject })) return false;
|
||||
// The link is bound to this finding's own carry chain, so an answer to one
|
||||
// occurrence never answers another occurrence of the same class.
|
||||
const before = Number(round) - 1;
|
||||
if (Number(n) === before) return oldId === f.carried_from;
|
||||
// An older answer reached this round through the previous one: the run's
|
||||
// round-(round-1) archive holds the finding this one was carried from, with
|
||||
// the same link, record and reason, itself bound the same way.
|
||||
const prev = archived(archiveName(run, before));
|
||||
if (typeof prev !== 'string' || frontmatterField(prev, 'round') !== String(before)) return false;
|
||||
const prevF = (payloadOf(prev)?.findings ?? []).find((x) => x && typeof x === 'object' && x.id === f.carried_from);
|
||||
const prevRow = parseAnswers(bodyOf(prev)).rows.find((r) => r.id === f.carried_from);
|
||||
if (!prevF || !prevRow || prevRow.carriedFrom !== row.carriedFrom || prevRow.record !== row.record
|
||||
|| normalizeReason(prevRow.reason) !== normalizeReason(row.reason)) return false;
|
||||
return carriedBound(prev, prevF, prevRow, { records, reviewSubject, archived });
|
||||
}
|
||||
|
||||
/**
|
||||
* Decide whether every counting MAJOR of a review is answered. Pure: the K2
|
||||
* records, the review's subject (its path relative to the git top level) and
|
||||
* the `preExisting` predicate (the gate's is lib/review/pre-existing.mjs
|
||||
* records, the review's subject (its path relative to the git top level), the
|
||||
* `preExisting` predicate (the gate's is lib/review/pre-existing.mjs
|
||||
* `preExistingCheck`, re-derived from git; absent = nothing is pre-existing)
|
||||
* come in `ctx`; with no subject or no records nothing is answered.
|
||||
* and the `archived` reader for carried rows (`archivedReader`; absent = no
|
||||
* carried row answers) come in `ctx`; with no subject or no records nothing
|
||||
* is answered.
|
||||
* @param {string} text - review.md
|
||||
* @param {{ records?: object[], reviewSubject?: string, preExisting?: (f) => boolean, archived?: unknown }} [ctx]
|
||||
* @param {{ records?: object[], reviewSubject?: string, preExisting?: (f) => boolean, archived?: (name: string) => string|null }} [ctx]
|
||||
* A review with no readable trailing JSON block is `review-not-measured:`
|
||||
* (not ok), never ok.
|
||||
* @returns {{ok: true} | {ok: false, unanswered: string[], fixedPending: string[], reason: string}}
|
||||
*/
|
||||
export function decideAnswers(text, { records = [], reviewSubject, preExisting = () => false } = {}) {
|
||||
export function decideAnswers(text, { records = [], reviewSubject, preExisting = () => false, archived } = {}) {
|
||||
const payload = payloadOf(text);
|
||||
// A review whose findings cannot be read is never answered (fail closed).
|
||||
if (!payload) {
|
||||
|
|
@ -233,25 +345,21 @@ export function decideAnswers(text, { records = [], reviewSubject, preExisting =
|
|||
}
|
||||
const majors = countingMajors(payload, { preExisting });
|
||||
if (majors.length === 0) return { ok: true };
|
||||
const blockSha = findingsBlockSha(text);
|
||||
const scopeShaEnd = reviewedSha(text) || null;
|
||||
const { rows } = parseAnswers(bodyOf(text));
|
||||
const rowOf = new Map(rows.map((r) => [r.id, r]));
|
||||
const unanswered = [];
|
||||
const fixedPending = [];
|
||||
const ctx = { records, reviewSubject, archived };
|
||||
for (const f of majors) {
|
||||
const row = rowOf.get(f.id);
|
||||
const bound = (answer) => {
|
||||
if (!row || !blockSha || !scopeShaEnd || !reviewSubject) return false;
|
||||
const reason = normalizeReason(row.reason);
|
||||
const rec = latestReviewRecord(Array.isArray(records) ? records : [], `${reviewSubject}#${f.id}`);
|
||||
return Boolean(rec) && rec.hash === row.record
|
||||
&& rec.sha256 === answerSha({ blockSha, scopeShaEnd, findingId: f.id, answer, reason });
|
||||
};
|
||||
if (row && row.answer === 'accepted' && !isSecurityFinding(f)) {
|
||||
const reason = normalizeReason(row.reason);
|
||||
if (reason !== '' && !isHollowAnswer(reason) && bound('accepted')) continue;
|
||||
if (row && row.carriedFrom !== '-') {
|
||||
// A carried row answers only through its link.
|
||||
if (carriedBound(text, f, row, ctx)) continue;
|
||||
unanswered.push(f.id);
|
||||
continue;
|
||||
}
|
||||
const bound = (answer) => directBound(text, f.id, row, answer, ctx);
|
||||
if (row && row.answer === 'accepted' && !isSecurityFinding(f) && realReason(row) && bound('accepted')) continue;
|
||||
if (row && row.answer === 'fixed' && bound('fixed')) { fixedPending.push(f.id); continue; }
|
||||
unanswered.push(f.id);
|
||||
}
|
||||
|
|
@ -352,8 +460,13 @@ export function pendingFindings({ reviewPath, repo }) {
|
|||
if (!payload) return { exit: 2, reason: `${reviewPath} has no readable trailing JSON block` };
|
||||
const { records, top, problem } = answerRecords(repo ?? dirname(resolve(reviewPath)));
|
||||
// The gate's pre-existing reading (lib/review/verdict-gate.mjs `decideOpenFindings`), so the list asks what the gate counts.
|
||||
const preExisting = preExistingCheck(text, repo ?? top ?? undefined);
|
||||
const d = decideAnswers(text, { records, reviewSubject: top ? reviewSubject(top, reviewPath) : undefined, preExisting });
|
||||
// A finding with `carried_from` is never pre-existing there either.
|
||||
const pre = preExistingCheck(text, repo ?? top ?? undefined);
|
||||
const preExisting = (f) => !f.carried_from && pre(f);
|
||||
const d = decideAnswers(text, {
|
||||
records, reviewSubject: top ? reviewSubject(top, reviewPath) : undefined, preExisting,
|
||||
archived: archivedReader(dirname(resolve(reviewPath))),
|
||||
});
|
||||
if (d.ok) return { exit: 0, pending: [] };
|
||||
const open = new Set([...d.unanswered, ...d.fixedPending]);
|
||||
const pending = countingMajors(payload, { preExisting }).filter((f) => open.has(f.id)).map((f) => ({
|
||||
|
|
@ -436,6 +549,74 @@ export function recordAnswer({ reviewPath, finding, answer, repo, round, store,
|
|||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* From round 2, after finalize: carry the previous review's valid `accepted`
|
||||
* answers to the carried MAJORs of this review (see the header). Writes rows
|
||||
* only; appends no K2 record and commits nothing.
|
||||
*/
|
||||
export function carryAnswers({ reviewPath, previousPath, repo }) {
|
||||
const refuse = (reason) => ({ exit: 1, written: [], reason });
|
||||
const cannot = (reason) => ({ exit: 2, written: [], reason });
|
||||
if (!reviewPath) return cannot('--review is required');
|
||||
if (!previousPath) return cannot('--previous is required');
|
||||
if (!repo) return cannot('--repo is required');
|
||||
let text;
|
||||
try { text = readFileSync(reviewPath, 'utf8'); } catch (e) { return cannot(`cannot read ${reviewPath}: ${e.message}`); }
|
||||
const dir = dirname(resolve(reviewPath));
|
||||
const name = basename(previousPath);
|
||||
const m = ARCHIVE_NAME_RE.exec(name);
|
||||
if (!m || dirname(resolve(previousPath)) !== dir) {
|
||||
return refuse(`--previous ${previousPath} is not an archive review-round-<run>-<n>.md beside ${reviewPath}`);
|
||||
}
|
||||
const { start } = scopeShas(text);
|
||||
const round = frontmatterField(text, 'round');
|
||||
if (!start || m[1] !== runId(start)) return refuse(`--previous ${name} is not an archive of this review's run`);
|
||||
if (!/^\d+$/.test(round ?? '') || Number(m[2]) >= Number(round)) {
|
||||
return refuse(`--previous ${name} is not an earlier round than this review's (round: ${round ?? 'none'})`);
|
||||
}
|
||||
const archived = archivedReader(dir);
|
||||
const prevText = archived(name);
|
||||
if (typeof prevText !== 'string') return cannot(`cannot read --previous ${previousPath} as a regular file`);
|
||||
const payload = payloadOf(text);
|
||||
const prevPayload = payloadOf(prevText);
|
||||
if (!payload || !Array.isArray(payload.findings)) return cannot(`${reviewPath} has no readable trailing JSON block`);
|
||||
if (!prevPayload || !Array.isArray(prevPayload.findings)) return cannot(`${previousPath} has no readable trailing JSON block`);
|
||||
const { records, top, problem } = answerRecords(repo);
|
||||
const ctx = { records, reviewSubject: top ? reviewSubject(top, reviewPath) : undefined, archived };
|
||||
const prevRows = new Map(parseAnswers(bodyOf(prevText)).rows.map((r) => [r.id, r]));
|
||||
const written = [];
|
||||
const skipped = [];
|
||||
let next = text;
|
||||
for (const f of payload.findings) {
|
||||
if (!f || typeof f !== 'object' || !isMajor(f) || typeof f.carried_from !== 'string') continue;
|
||||
const row = prevRows.get(f.carried_from);
|
||||
if (!row || row.answer !== 'accepted') continue;
|
||||
const oldF = prevPayload.findings.find((x) => x && typeof x === 'object' && x.id === row.id);
|
||||
let link = null;
|
||||
if (row.carriedFrom === '-') {
|
||||
// Given in the previous review itself.
|
||||
const ok = oldF && !isSecurityFinding(oldF) && !isSecurityFinding(f) && realReason(row)
|
||||
&& sameFinding(oldF, f) && directBound(prevText, row.id, row, 'accepted', ctx);
|
||||
if (ok) link = `${name}#${row.id}`;
|
||||
} else if (oldF && carriedBound(prevText, oldF, row, ctx)) {
|
||||
// Carried into the previous review: the link to where it was given
|
||||
// holds when that finding also matches this one.
|
||||
const [, archive, , , givenId] = LINK_RE.exec(row.carriedFrom);
|
||||
const given = (payloadOf(archived(archive))?.findings ?? []).find((x) => x && x.id === givenId);
|
||||
if (given && sameFinding(given, f)) link = row.carriedFrom;
|
||||
}
|
||||
if (link === null) { skipped.push(f.id); continue; }
|
||||
const placed = withAnswerRow(next, f.id, `| ${f.id} | accepted | ${normalizeReason(row.reason)} | ${row.record} | ${link} |`);
|
||||
if (placed === null) return cannot(`${reviewPath}: the ## Answers table could not be placed; nothing written`);
|
||||
next = placed;
|
||||
written.push(f.id);
|
||||
}
|
||||
if (written.length > 0) {
|
||||
try { writeAtomic(reviewPath, next); } catch (e) { return cannot(`cannot write ${reviewPath}: ${e.message}`); }
|
||||
}
|
||||
return { exit: 0, written, skipped, ...(problem ? { log: problem } : {}) };
|
||||
}
|
||||
|
||||
// ---- CLI ----
|
||||
|
||||
function opt(args, name) {
|
||||
|
|
@ -458,8 +639,10 @@ if (isMain(import.meta.url)) {
|
|||
repo: opt(args, '--repo'), round: opt(args, '--round'), store: opt(args, '--store'),
|
||||
trailer: opt(args, '--trailer'), reason,
|
||||
});
|
||||
} else if (cmd === 'carry') {
|
||||
result = carryAnswers({ reviewPath: opt(args, '--review'), previousPath: opt(args, '--previous'), repo: opt(args, '--repo') });
|
||||
} else {
|
||||
process.stderr.write('Usage: review-answers.mjs pending|record — see the header of lib/review/review-answers.mjs\n');
|
||||
process.stderr.write('Usage: review-answers.mjs pending|record|carry — see the header of lib/review/review-answers.mjs\n');
|
||||
process.exit(2);
|
||||
}
|
||||
} catch (e) {
|
||||
|
|
|
|||
|
|
@ -5,41 +5,79 @@
|
|||
// instruction, carry, pre-existing, policy); later steps insert theirs at
|
||||
// their places in PASSES. Today it runs:
|
||||
//
|
||||
// - carry (round 2 and later, `--previous <archive>`): `carry`
|
||||
// (lib/review/review-rounds.mjs) keeps every previous finding that this
|
||||
// round gives no positive evidence of being resolved, with `carried_from`;
|
||||
// a current finding that matches one folds into it. What ran this round is
|
||||
// read from the project folder, never inferred from what is missing:
|
||||
// an engine ran when a regular `.review-engine-payload-*.json` there holds
|
||||
// a top-level `reviewer` that maps back through `ENGINES`
|
||||
// (lib/review/engine-findings.mjs), and a reviewer ran when its regular
|
||||
// `.review-out-<reviewer>.md` passes the findings schema; that output's
|
||||
// `previous: [{id, resolved}]` are its marks. The previous review's `ran`
|
||||
// counts only when it is `finalized: true`. Writes `resolved` and a
|
||||
// `## Resolved since round <n>` section just before `## Declined to Judge`.
|
||||
// - pre-existing: `markFindings` (lib/review/pre-existing.mjs) sets
|
||||
// `pre_existing` on every JSON finding from git over
|
||||
// `scope_sha_start..scope_sha_end` (read raw, as `reviewedSha` reads; a
|
||||
// YAML number or `null` means no mark), writes `pre_existing_counts`, and
|
||||
// writes a `## Pre-existing` list (the count, then one line per title)
|
||||
// just before `## Declined to Judge`. It never moves or lowers a finding.
|
||||
// just before `## Resolved since round <n>` or `## Declined to Judge`. It
|
||||
// never moves or lowers a finding. A finding with `carried_from` keeps its
|
||||
// line from the previous round's end, so it is never pre-existing
|
||||
// (`pre_existing: false`, not judged against this range).
|
||||
//
|
||||
// Then, always last, the id recomputation: every id is computed again with
|
||||
// `computeFindingIds` (lib/parsers/finding-id.mjs) over the final array, and
|
||||
// any id that changed is rewritten with one simultaneous old -> new map,
|
||||
// applied in one pass so no rewrite chains, in four places only: each JSON
|
||||
// finding's `id`, the frontmatter `findings:` list, the id line of each body
|
||||
// entry (its `### <id>` heading or `**ID:**` line), and each
|
||||
// `**Source finding:**` line. The file is written atomically (temp + rename).
|
||||
// applied in one pass so no rewrite chains: each JSON finding's `id`, the id
|
||||
// line of each body entry (its `### <id>` heading or `**ID:**` line), and each
|
||||
// `Source finding:` line; the frontmatter `findings:` list is written from the
|
||||
// final ids. A folded finding keeps the current finding's id until then, so
|
||||
// every reference to it, its R-step included, names the carried finding.
|
||||
// `carried_from`, `resolved` and `## Answers` values keep the previous ids.
|
||||
// After it: a body entry for each finding finalize added (under its
|
||||
// severity's `## Findings` section; an entry under another severity's section
|
||||
// moves), counts recomputed when the carry ran, a verdict that is only ever
|
||||
// raised (frontmatter and JSON block), one `### Step R<n>` for each BLOCKER
|
||||
// that no R-step names (the template's form, numbered after the last), and
|
||||
// `ran` in the JSON block with `round`, `round_base` and `finalized: true` in
|
||||
// the frontmatter. An R-step whose `Source finding:` names no finding of the
|
||||
// review is refused (exit 1). The file is written atomically (temp + rename).
|
||||
//
|
||||
// CLI:
|
||||
// node lib/review/review-finalize.mjs --review <p> --repo <d> --project <dir>
|
||||
// --round <R> [--round-base <sha>] [--previous <p>]
|
||||
// (from round 2 both --round-base and --previous are required; --previous
|
||||
// names this run's archive of round R-1, `review-round-<run>-<R-1>.md`)
|
||||
// prints one JSON line; exit 0 = written, 1 = refused (a placeholder left,
|
||||
// an id that cannot be computed, two findings under one id), 2 = could not
|
||||
// run (bad arguments, unreadable file, no trailing JSON block). On 1 or 2
|
||||
// the review is left as fill-finding-ids wrote it.
|
||||
// an id that cannot be computed, two findings under one id, an orphan
|
||||
// R-step), 2 = could not run (bad arguments, unreadable file, no trailing
|
||||
// JSON block, an unreadable previous review). On 1 or 2 the review is left
|
||||
// as fill-finding-ids wrote it.
|
||||
|
||||
import { readFileSync, renameSync, statSync, writeFileSync } from 'node:fs';
|
||||
import { lstatSync, readFileSync, readdirSync, renameSync, statSync, writeFileSync } from 'node:fs';
|
||||
import { basename, dirname, join } from 'node:path';
|
||||
import { computeFindingIds } from '../parsers/finding-id.mjs';
|
||||
import { fencedBlocks, isJsonBlock } from './findings-schema.mjs';
|
||||
import { fencedBlocks, isJsonBlock, validateReviewerOutput } from './findings-schema.mjs';
|
||||
import { FULL_SHA_RE } from './reviewed-commit.mjs';
|
||||
import { markFindings, scopeShas } from './pre-existing.mjs';
|
||||
import { ENGINES } from './engine-findings.mjs';
|
||||
import { carry, frontmatterField, MAX_REVIEW_ROUNDS, runId } from './review-rounds.mjs';
|
||||
import { isMain } from '../util/is-main.mjs';
|
||||
|
||||
export const MAX_ROUND = 3;
|
||||
const SEVERITIES = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'];
|
||||
const VERDICT_RANK = { ALLOW: 0, WARN: 1, BLOCK: 2 };
|
||||
const HEX40 = /^[0-9a-f]{40}$/;
|
||||
const HEX40_G = /[0-9a-f]{40}/g;
|
||||
const SOURCE_RE = /Source finding:\**\s*`?([0-9a-f]{40})`?/;
|
||||
const ARCHIVE_RE = /^review-round-([0-9a-f]{12})-(\d+)\.md$/;
|
||||
const OUT_RE = /^\.review-out-(.+)\.md$/;
|
||||
const PAYLOAD_RE = /^\.review-engine-payload-.+\.json$/;
|
||||
const ENGINE_OF_REVIEWER = Object.freeze(Object.fromEntries(Object.entries(ENGINES).map(([engine, s]) => [s.reviewer, engine])));
|
||||
|
||||
const isPlainObject = (x) => x !== null && typeof x === 'object' && !Array.isArray(x);
|
||||
const sevOf = (f) => String(f?.severity ?? '').toUpperCase();
|
||||
|
||||
/** The review split at the trailing closed `json` block: {head, fence, body, tail}, or null. */
|
||||
function splitTrailing(text) {
|
||||
|
|
@ -56,9 +94,111 @@ function splitTrailing(text) {
|
|||
};
|
||||
}
|
||||
|
||||
/** The parsed trailing JSON block when it is an object with a findings array, else null. */
|
||||
function payloadOf(text) {
|
||||
const parts = splitTrailing(text);
|
||||
if (!parts) return null;
|
||||
try {
|
||||
const p = JSON.parse(parts.body);
|
||||
return isPlainObject(p) && Array.isArray(p.findings) ? p : null;
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
/** One line of prose: whitespace runs made one space, backticks removed. */
|
||||
const oneLine = (s) => String(s ?? '').replace(/`/g, '').replace(/\s+/g, ' ').trim();
|
||||
|
||||
// ---- sections of the head ----------------------------------------------------------
|
||||
|
||||
/** The index of the first line matching any of `res`, else -1. */
|
||||
const firstIndex = (lines, res) => lines.findIndex((l) => res.some((re) => re.test(l)));
|
||||
|
||||
/** The end (exclusive) of the `## ` section that starts at `at`. */
|
||||
function sectionEnd(lines, at) {
|
||||
let i = at + 1;
|
||||
while (i < lines.length && !/^##\s/.test(lines[i])) i++;
|
||||
return i;
|
||||
}
|
||||
|
||||
/**
|
||||
* `head` with every section whose heading matches `headingRe` taken out and
|
||||
* `section` put just before the first line matching any of `before` (else at
|
||||
* the end).
|
||||
*/
|
||||
function replaceSection(head, headingRe, section, before) {
|
||||
const lines = head.split('\n');
|
||||
const kept = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (headingRe.test(lines[i])) {
|
||||
i = sectionEnd(lines, i) - 1;
|
||||
continue;
|
||||
}
|
||||
kept.push(lines[i]);
|
||||
}
|
||||
const at = firstIndex(kept, before);
|
||||
if (at === -1) return `${kept.join('\n').replace(/\n*$/, '\n\n')}${section}`;
|
||||
kept.splice(at, 0, ...section.replace(/\n$/, '').split('\n'));
|
||||
return kept.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Append `block` lines at the end of the section whose heading matches
|
||||
* `headingRe`; when there is none, create it (`heading`) just before the first
|
||||
* line matching any of `before` (else at the end). Returns the new lines.
|
||||
*/
|
||||
function appendToSection(lines, headingRe, heading, block, before) {
|
||||
let at = lines.findIndex((l) => headingRe.test(l));
|
||||
if (at === -1) {
|
||||
const where = firstIndex(lines, before);
|
||||
const add = [heading, ''];
|
||||
if (where === -1) {
|
||||
while (lines.length && lines.at(-1) === '') lines.pop();
|
||||
lines.push('', ...add);
|
||||
at = lines.length - 2;
|
||||
} else {
|
||||
lines.splice(where, 0, ...add);
|
||||
at = where;
|
||||
}
|
||||
}
|
||||
let end = sectionEnd(lines, at);
|
||||
while (end - 1 > at && lines[end - 1] === '') end--;
|
||||
const out = ['', ...block, ''];
|
||||
lines.splice(end, 0, ...out);
|
||||
// One blank line before what follows (the blank lines trimmed above).
|
||||
const next = end + out.length;
|
||||
while (next < lines.length && lines[next] === '' && lines[next - 1] === '') lines.splice(next, 1);
|
||||
return lines;
|
||||
}
|
||||
|
||||
// ---- pass: carry -------------------------------------------------------------------
|
||||
|
||||
const LATER_SECTIONS = [/^##\s+Suppressed Findings\b/, /^##\s+Pre-existing\s*$/, /^##\s+Resolved since round\b/,
|
||||
/^##\s+Declined to Judge\b/, /^##\s+Remediation Steps\b/, /^##\s+Remediation Summary\b/, /^##\s+Answers\s*$/];
|
||||
|
||||
/** The `## Resolved since round <n>` section. */
|
||||
function resolvedSection(n, list) {
|
||||
if (list.length === 0) return `## Resolved since round ${n}\n\nNone.\n\n`;
|
||||
const items = list.map((f) => `- ${oneLine(f.title) || '(no title)'} (\`${oneLine(f.file)}:${f.line}\`, \`${f.id}\`)`);
|
||||
return `## Resolved since round ${n}\n\nPrevious findings this round no longer raises: ${list.length}.\n\n${items.join('\n')}\n\n`;
|
||||
}
|
||||
|
||||
/** Pass `carry`: only with a previous review. */
|
||||
function passCarry(payload, ctx) {
|
||||
if (!ctx.previous) return {};
|
||||
const before = payload.findings.length;
|
||||
const r = carry(ctx.previous.findings, payload.findings, {
|
||||
marks: ctx.evidence.marks, ran: ctx.evidence.ran, previousRan: ctx.previous.ran,
|
||||
});
|
||||
payload.findings = r.findings;
|
||||
for (const f of r.findings.slice(before)) ctx.added.add(f);
|
||||
payload.resolved = r.resolved;
|
||||
ctx.carried = true;
|
||||
const list = ctx.previous.findings.filter((p) => r.resolved.includes(p.id));
|
||||
const section = resolvedSection(ctx.round - 1, list);
|
||||
return { headEdit: (head) => replaceSection(head, /^##\s+Resolved since round\b/, section, [/^##\s+Declined to Judge\b/]) };
|
||||
}
|
||||
|
||||
// ---- pass: pre-existing ----------------------------------------------------------
|
||||
|
||||
/** The `## Pre-existing` section for the marked findings. */
|
||||
function preExistingSection(findings, counts) {
|
||||
const marked = findings.filter((f) => f && f.pre_existing === true);
|
||||
|
|
@ -69,94 +209,223 @@ function preExistingSection(findings, counts) {
|
|||
+ `| Severity | Count |\n|----------|-------|\n${rows.join('\n')}\n\n${items.join('\n')}\n\n`;
|
||||
}
|
||||
|
||||
/** `head` with every `## Pre-existing` section taken out and `section` put just before `## Declined to Judge` (else at the end). */
|
||||
function withPreExistingSection(head, section) {
|
||||
const lines = head.split('\n');
|
||||
const kept = [];
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
if (/^##\s+Pre-existing\s*$/.test(lines[i])) {
|
||||
i++;
|
||||
while (i < lines.length && !/^##\s/.test(lines[i])) i++;
|
||||
i--;
|
||||
continue;
|
||||
}
|
||||
kept.push(lines[i]);
|
||||
}
|
||||
const at = kept.findIndex((l) => /^##\s+Declined to Judge\b/.test(l));
|
||||
const sectionLines = section.replace(/\n$/, '').split('\n');
|
||||
if (at === -1) {
|
||||
const out = kept.join('\n').replace(/\n*$/, '\n\n');
|
||||
return `${out}${section}`;
|
||||
}
|
||||
kept.splice(at, 0, ...sectionLines);
|
||||
return kept.join('\n');
|
||||
}
|
||||
|
||||
/** Pass `pre-existing`. */
|
||||
function passPreExisting(payload, { repo, text }) {
|
||||
const { start, end } = scopeShas(text);
|
||||
markFindings(payload, { repo, start, end });
|
||||
// A carried finding keeps its line from the previous round's end, so this
|
||||
// range's lines say nothing about it: it is never pre-existing.
|
||||
const carried = (f) => Boolean(f && typeof f === 'object' && f.carried_from);
|
||||
markFindings({ findings: payload.findings.filter((f) => !carried(f)) }, { repo, start, end });
|
||||
for (const f of payload.findings) if (carried(f)) f.pre_existing = false;
|
||||
const counts = Object.fromEntries(SEVERITIES.map((s) => [s, 0]));
|
||||
for (const f of payload.findings) {
|
||||
const sev = String(f?.severity ?? '').toUpperCase();
|
||||
const sev = sevOf(f);
|
||||
if (f && f.pre_existing === true && sev in counts) counts[sev] += 1;
|
||||
}
|
||||
payload.pre_existing_counts = counts;
|
||||
return { headEdit: (head) => withPreExistingSection(head, preExistingSection(payload.findings, counts)), counts };
|
||||
const section = preExistingSection(payload.findings, counts);
|
||||
return {
|
||||
headEdit: (head) => replaceSection(head, /^##\s+Pre-existing\s*$/, section,
|
||||
[/^##\s+Resolved since round\b/, /^##\s+Declined to Judge\b/]),
|
||||
counts,
|
||||
};
|
||||
}
|
||||
|
||||
/** The passes, in the Architecture order; the id recomputation runs after them, always last. */
|
||||
export const PASSES = Object.freeze([
|
||||
{ name: 'carry', run: passCarry },
|
||||
{ name: 'pre-existing', run: passPreExisting },
|
||||
]);
|
||||
|
||||
// ---- after the ids ---------------------------------------------------------------
|
||||
|
||||
/** Rewrite the 40-hex tokens of one line through `map` (one pass: no chaining). */
|
||||
const remap = (line, map) => line.replace(HEX40_G, (m) => map.get(m) ?? m);
|
||||
|
||||
/** `head` with ids rewritten in the frontmatter list, body id lines and Source finding lines. */
|
||||
/** The frontmatter's closing line index (the opening is line 0), or -1. */
|
||||
function frontmatterEnd(lines) {
|
||||
return lines[0] === '---' ? lines.indexOf('---', 1) : -1;
|
||||
}
|
||||
|
||||
/** `head` with ids rewritten in body id lines and Source finding lines. */
|
||||
function rewriteIds(head, map) {
|
||||
if (map.size === 0) return head;
|
||||
const lines = head.split('\n');
|
||||
let fmEnd = -1;
|
||||
if (lines[0] === '---') fmEnd = lines.indexOf('---', 1);
|
||||
let inList = false;
|
||||
const fmEnd = frontmatterEnd(lines);
|
||||
return lines.map((l, i) => {
|
||||
if (i > 0 && i < fmEnd) {
|
||||
if (/^findings\s*:/.test(l)) { inList = true; return l; }
|
||||
if (inList && /^\s+-\s/.test(l)) return remap(l, map);
|
||||
if (!/^\s/.test(l)) inList = false;
|
||||
return l;
|
||||
}
|
||||
if (i <= fmEnd) return l;
|
||||
if (/^###\s+[0-9a-f]{40}\b/.test(l) || /\*\*ID:\*\*/.test(l) || /\*\*Source finding:\*\*/.test(l)) return remap(l, map);
|
||||
if (/^###\s+[0-9a-f]{40}\b/.test(l) || /\*\*ID:\*\*/.test(l) || /Source finding:/.test(l)) return remap(l, map);
|
||||
return l;
|
||||
}).join('\n');
|
||||
}
|
||||
|
||||
/** Frontmatter edits: the `findings:` list written from `ids`, `fields` set (replaced, or added before `findings:`). */
|
||||
function editFrontmatter(head, ids, fields) {
|
||||
const lines = head.split('\n');
|
||||
const fmEnd = frontmatterEnd(lines);
|
||||
if (fmEnd === -1) return head;
|
||||
const fm = lines.slice(1, fmEnd);
|
||||
const out = [];
|
||||
for (let i = 0; i < fm.length; i++) {
|
||||
const l = fm[i];
|
||||
const key = /^([A-Za-z_][\w-]*)\s*:/.exec(l)?.[1];
|
||||
if (key && Object.hasOwn(fields, key)) {
|
||||
while (i + 1 < fm.length && /^\s/.test(fm[i + 1])) i++;
|
||||
continue;
|
||||
}
|
||||
if (key === 'findings') {
|
||||
while (i + 1 < fm.length && /^\s/.test(fm[i + 1])) i++;
|
||||
for (const [k, v] of Object.entries(fields)) out.push(`${k}: ${v}`);
|
||||
out.push(ids.length === 0 ? 'findings: []' : 'findings:', ...ids.map((id) => ` - ${id}`));
|
||||
continue;
|
||||
}
|
||||
out.push(l);
|
||||
}
|
||||
if (!fm.some((l) => /^findings\s*:/.test(l))) {
|
||||
for (const [k, v] of Object.entries(fields)) out.push(`${k}: ${v}`);
|
||||
}
|
||||
return [lines[0], ...out, ...lines.slice(fmEnd)].join('\n');
|
||||
}
|
||||
|
||||
/** A body entry for a finding finalize added, in the coordinator's form. */
|
||||
function entryLines(f) {
|
||||
const lines = [`### ${f.id}`, '', `- file: ${oneLine(f.file)}`, `- line: ${f.line}`, `- rule_key: ${oneLine(f.rule_key)}`];
|
||||
for (const k of ['brief_ref', 'title', 'detail', 'recommended_action', 'engine', 'carried_from']) {
|
||||
if (f[k] !== undefined && f[k] !== null && oneLine(f[k]) !== '') lines.push(`- ${k}: ${oneLine(f[k])}`);
|
||||
}
|
||||
return lines;
|
||||
}
|
||||
|
||||
const findingsHeading = (sev) => new RegExp(`^##\\s+Findings\\s*\\(${sev}\\)\\s*$`);
|
||||
|
||||
/** Where a new `## Findings (<sev>)` section goes: before a later severity's section or any later section. */
|
||||
function findingsBefore(sev) {
|
||||
return [...SEVERITIES.slice(SEVERITIES.indexOf(sev) + 1).map(findingsHeading), ...LATER_SECTIONS];
|
||||
}
|
||||
|
||||
/**
|
||||
* The body entries: one added for each finding in `added`, and a `### <id>`
|
||||
* entry that sits under another severity's `## Findings` section moved to its own.
|
||||
*/
|
||||
function placeEntries(head, findings, added) {
|
||||
let lines = head.split('\n');
|
||||
const fmEnd = frontmatterEnd(lines);
|
||||
for (const f of findings) {
|
||||
const sev = sevOf(f);
|
||||
if (!SEVERITIES.includes(sev) || typeof f.id !== 'string') continue;
|
||||
if (added.has(f)) {
|
||||
lines = appendToSection(lines, findingsHeading(sev), `## Findings (${sev})`, entryLines(f), findingsBefore(sev));
|
||||
continue;
|
||||
}
|
||||
const at = lines.findIndex((l, i) => i > fmEnd && new RegExp(`^###\\s+${f.id}\\s*$`).test(l));
|
||||
if (at === -1) continue;
|
||||
let h = at - 1;
|
||||
while (h > fmEnd && !/^##\s/.test(lines[h])) h--;
|
||||
const m = /^##\s+Findings\s*\((BLOCKER|MAJOR|MINOR|SUGGESTION)\)\s*$/.exec(lines[h] ?? '');
|
||||
if (!m || m[1] === sev) continue;
|
||||
let end = at + 1;
|
||||
while (end < lines.length && !/^#{2,3}\s/.test(lines[end])) end++;
|
||||
const block = lines.splice(at, end - at);
|
||||
while (block.length && block.at(-1) === '') block.pop();
|
||||
lines = appendToSection(lines, findingsHeading(sev), `## Findings (${sev})`, block, findingsBefore(sev));
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/** The `### Step R<n>` blocks of `## Remediation Steps`: [{n, source}], and the section's bounds. */
|
||||
function remediationSteps(lines) {
|
||||
const at = lines.findIndex((l) => /^##\s+Remediation Steps\b/.test(l));
|
||||
if (at === -1) return { at: -1, end: -1, steps: [] };
|
||||
const end = sectionEnd(lines, at);
|
||||
const steps = [];
|
||||
for (let i = at + 1; i < end; i++) {
|
||||
const m = /^###\s+Step\s+R(\d+)\b/.exec(lines[i]);
|
||||
if (m) steps.push({ n: Number(m[1]), source: null, line: i });
|
||||
else if (steps.length && steps.at(-1).source === null) {
|
||||
const s = SOURCE_RE.exec(lines[i]);
|
||||
if (s) steps.at(-1).source = s[1];
|
||||
}
|
||||
}
|
||||
return { at, end, steps };
|
||||
}
|
||||
|
||||
/** One R-step per BLOCKER that has none, in the template's form; returns {head, added}. */
|
||||
function withBlockerSteps(head, findings) {
|
||||
let lines = head.split('\n');
|
||||
const { steps } = remediationSteps(lines);
|
||||
const named = new Set(steps.map((s) => s.source).filter(Boolean));
|
||||
let n = steps.reduce((m, s) => Math.max(m, s.n), 0);
|
||||
const added = [];
|
||||
for (const f of findings) {
|
||||
if (sevOf(f) !== 'BLOCKER' || named.has(f.id)) continue;
|
||||
n += 1;
|
||||
named.add(f.id);
|
||||
added.push(`R${n}`);
|
||||
const action = oneLine(f.recommended_action) || `Fix: ${oneLine(f.title) || f.id}`;
|
||||
lines = appendToSection(lines, /^##\s+Remediation Steps\b/, '## Remediation Steps', [
|
||||
`### Step R${n}: ${action}`,
|
||||
'',
|
||||
`- **Files:** \`${oneLine(f.file)}\``,
|
||||
`- **Source finding:** \`${f.id}\``,
|
||||
'- **Test first:** a test that reproduces the finding and fails today',
|
||||
'- **Verify:** re-run `/trekreview`; the finding is absent',
|
||||
'- **On failure:** escalate',
|
||||
], [/^##\s+Remediation Summary\b/, /^##\s+Answers\s*$/]);
|
||||
}
|
||||
if (added.length > 0) {
|
||||
// A "None." left in the section is no longer true.
|
||||
const { at, end } = remediationSteps(lines);
|
||||
for (let i = end - 1; i > at; i--) {
|
||||
if (lines[i].trim() === 'None.' || (lines[i] === '' && lines[i + 1] === '')) lines.splice(i, 1);
|
||||
}
|
||||
}
|
||||
return { head: lines.join('\n'), added };
|
||||
}
|
||||
|
||||
/** The verdict the findings need, never lower than `current`. */
|
||||
function raisedVerdict(current, findings) {
|
||||
if (!Object.hasOwn(VERDICT_RANK, current)) return current;
|
||||
const need = findings.some((f) => sevOf(f) === 'BLOCKER') ? 'BLOCK'
|
||||
: findings.some((f) => sevOf(f) === 'MAJOR') ? 'WARN' : 'ALLOW';
|
||||
return VERDICT_RANK[need] > VERDICT_RANK[current] ? need : current;
|
||||
}
|
||||
|
||||
/** `head` with every frontmatter `verdict:` line set to `verdict`. */
|
||||
function withVerdict(head, verdict) {
|
||||
const lines = head.split('\n');
|
||||
const fmEnd = frontmatterEnd(lines);
|
||||
for (let i = 1; i < fmEnd; i++) if (/^verdict\s*:/.test(lines[i])) lines[i] = `verdict: ${verdict}`;
|
||||
return lines.join('\n');
|
||||
}
|
||||
|
||||
/**
|
||||
* Finalize review.md text. Pure apart from the git reads of the passes.
|
||||
* @param {string} text
|
||||
* @param {{ repo: string }} ctx
|
||||
* @returns {{exit: 0, text: string, passes: string[], pre_existing_counts: object, ids_rewritten: number} | {exit: 1|2, reason: string}}
|
||||
* @param {{ repo: string, round?: number, roundBase?: string|null,
|
||||
* previous?: {findings: object[], ran: object|null}|null,
|
||||
* evidence?: {ran: {engines: string[], reviewers: string[]}, marks: object[]} }} ctx
|
||||
* @returns {{exit: 0, text: string, passes: string[], ...} | {exit: 1|2, reason: string}}
|
||||
*/
|
||||
export function finalizeReview(text, { repo } = {}) {
|
||||
export function finalizeReview(text, { repo, round, roundBase = null, previous = null, evidence } = {}) {
|
||||
if (/@@ID_\d+@@/.test(text)) return { exit: 1, reason: 'an @@ID_<n>@@ placeholder is left: run fill-finding-ids.mjs first' };
|
||||
const parts = splitTrailing(text);
|
||||
if (!parts) return { exit: 2, reason: 'no trailing closed json block' };
|
||||
let payload;
|
||||
try { payload = JSON.parse(parts.body); } catch (e) { return { exit: 2, reason: `the trailing JSON block does not parse: ${e.message}` }; }
|
||||
if (!payload || typeof payload !== 'object' || Array.isArray(payload) || !Array.isArray(payload.findings)) {
|
||||
if (!isPlainObject(payload) || !Array.isArray(payload.findings)) {
|
||||
return { exit: 2, reason: 'the trailing JSON block has no findings array' };
|
||||
}
|
||||
if (payload.findings.some((f) => !f || typeof f !== 'object')) return { exit: 1, reason: 'a finding is not an object' };
|
||||
if (payload.findings.some((f) => !isPlainObject(f))) return { exit: 1, reason: 'a finding is not an object' };
|
||||
const before = payload.findings.map((f) => f.id);
|
||||
const dup = before.find((id, i) => before.indexOf(id) !== i);
|
||||
if (dup !== undefined) return { exit: 1, reason: `two findings carry the id ${JSON.stringify(dup)}` };
|
||||
|
||||
const ev = evidence ?? { ran: { engines: [], reviewers: [] }, marks: [] };
|
||||
const ctx = { repo, text, round, previous, evidence: ev, added: new Set(), carried: false };
|
||||
let head = parts.head;
|
||||
const out = { passes: [] };
|
||||
for (const pass of PASSES) {
|
||||
const r = pass.run(payload, { repo, text });
|
||||
const r = pass.run(payload, ctx);
|
||||
if (r.headEdit) head = r.headEdit(head);
|
||||
if (r.counts) out.pre_existing_counts = r.counts;
|
||||
out.passes.push(pass.name);
|
||||
|
|
@ -173,8 +442,108 @@ export function finalizeReview(text, { repo } = {}) {
|
|||
});
|
||||
head = rewriteIds(head, map);
|
||||
out.passes.push('ids');
|
||||
|
||||
head = placeEntries(head, payload.findings, ctx.added);
|
||||
if (ctx.carried) {
|
||||
payload.counts = Object.fromEntries(SEVERITIES.map((s) => [s, payload.findings.filter((f) => sevOf(f) === s).length]));
|
||||
}
|
||||
const verdict = raisedVerdict(payload.verdict, payload.findings);
|
||||
if (verdict !== payload.verdict) {
|
||||
payload.verdict = verdict;
|
||||
head = withVerdict(head, verdict);
|
||||
}
|
||||
const steps = withBlockerSteps(head, payload.findings);
|
||||
head = steps.head;
|
||||
const known = new Set(payload.findings.map((f) => f.id));
|
||||
const orphans = remediationSteps(head.split('\n')).steps.filter((s) => s.source !== null && !known.has(s.source));
|
||||
if (orphans.length > 0) {
|
||||
return { exit: 1, reason: `R-step ${orphans.map((s) => `R${s.n} (Source finding ${s.source})`).join(', ')} names no finding of this review` };
|
||||
}
|
||||
payload.ran = { engines: [...ev.ran.engines], reviewers: [...ev.ran.reviewers] };
|
||||
const fields = {
|
||||
...(Number.isInteger(round) ? { round: String(round), round_base: roundBase ?? 'null' } : {}),
|
||||
finalized: 'true',
|
||||
};
|
||||
head = editFrontmatter(head, payload.findings.map((f) => f.id), fields);
|
||||
const body = `${JSON.stringify(payload, null, 2)}\n`;
|
||||
return { exit: 0, text: `${head}${parts.fence}${body}${parts.tail}`, ...out, ids_rewritten: map.size };
|
||||
return {
|
||||
exit: 0, text: `${head}${parts.fence}${body}${parts.tail}`, ...out, ids_rewritten: map.size,
|
||||
carried: ctx.added.size, resolved: payload.resolved ?? [], ran: payload.ran, r_steps_added: steps.added,
|
||||
};
|
||||
}
|
||||
|
||||
// ---- evidence and the previous review ------------------------------------------------
|
||||
|
||||
/**
|
||||
* What ran this round, from the project folder (positive evidence only), and
|
||||
* the reviewers' marks of previous findings.
|
||||
* @param {string} dir
|
||||
* @returns {{ ran: {engines: string[], reviewers: string[]}, marks: Array<{id, resolved, reviewer}> }}
|
||||
*/
|
||||
export function roundEvidence(dir) {
|
||||
const engines = new Set();
|
||||
const reviewers = new Set();
|
||||
const marks = [];
|
||||
let names = [];
|
||||
try { names = readdirSync(dir).sort(); } catch { names = []; }
|
||||
for (const name of names) {
|
||||
const p = join(dir, name);
|
||||
try { if (!lstatSync(p).isFile()) continue; } catch { continue; }
|
||||
if (PAYLOAD_RE.test(name)) {
|
||||
let payload;
|
||||
try { payload = JSON.parse(readFileSync(p, 'utf8')); } catch { continue; }
|
||||
const engine = isPlainObject(payload) && typeof payload.reviewer === 'string' && Object.hasOwn(ENGINE_OF_REVIEWER, payload.reviewer)
|
||||
? ENGINE_OF_REVIEWER[payload.reviewer] : null;
|
||||
if (engine && Array.isArray(payload.findings)) engines.add(engine);
|
||||
continue;
|
||||
}
|
||||
const m = OUT_RE.exec(name);
|
||||
if (!m) continue;
|
||||
let raw;
|
||||
try { raw = readFileSync(p, 'utf8'); } catch { continue; }
|
||||
const v = validateReviewerOutput(raw);
|
||||
if (!v.valid) continue;
|
||||
reviewers.add(m[1]);
|
||||
for (const mark of Array.isArray(v.parsed?.previous) ? v.parsed.previous : []) {
|
||||
if (isPlainObject(mark) && typeof mark.id === 'string' && HEX40.test(mark.id) && typeof mark.resolved === 'boolean') {
|
||||
marks.push({ id: mark.id, resolved: mark.resolved, reviewer: m[1] });
|
||||
}
|
||||
}
|
||||
}
|
||||
return { ran: { engines: [...engines].sort(), reviewers: [...reviewers].sort() }, marks };
|
||||
}
|
||||
|
||||
/** A `ran` object of the 1.2 shape, else null. */
|
||||
function ranOf(r) {
|
||||
const ok = isPlainObject(r)
|
||||
&& Array.isArray(r.engines) && r.engines.every((e) => Object.hasOwn(ENGINES, e))
|
||||
&& Array.isArray(r.reviewers) && r.reviewers.every((s) => typeof s === 'string');
|
||||
return ok ? { engines: r.engines, reviewers: r.reviewers } : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The previous round's review: this run's archive of round `round - 1`.
|
||||
* `ran` is null unless it is `finalized: true` with a well-formed `ran`.
|
||||
*/
|
||||
export function readPrevious(path, text, round) {
|
||||
const m = ARCHIVE_RE.exec(basename(String(path)));
|
||||
if (!m) return { exit: 2, reason: `--previous ${path} is not an archive review-round-<run>-<n>.md` };
|
||||
const { start } = scopeShas(text);
|
||||
if (!start) return { exit: 2, reason: 'the review has no full scope_sha_start: no run to carry from' };
|
||||
if (m[1] !== runId(start)) return { exit: 2, reason: `--previous ${basename(path)} is an archive of another run (this run is ${runId(start)})` };
|
||||
if (Number(m[2]) !== round - 1) return { exit: 2, reason: `--previous ${basename(path)} is not round ${round - 1}` };
|
||||
let prevText;
|
||||
try {
|
||||
if (!lstatSync(path).isFile()) return { exit: 2, reason: `--previous ${path} is not a regular file` };
|
||||
prevText = readFileSync(path, 'utf8');
|
||||
} catch (e) { return { exit: 2, reason: `cannot read --previous ${path}: ${e.message}` }; }
|
||||
const payload = payloadOf(prevText);
|
||||
if (!payload) return { exit: 2, reason: `--previous ${path} has no readable trailing JSON block with findings` };
|
||||
const finalized = frontmatterField(prevText, 'finalized') === 'true';
|
||||
return {
|
||||
exit: 0,
|
||||
previous: { findings: payload.findings.filter(isPlainObject), ran: finalized ? ranOf(payload.ran) : null },
|
||||
};
|
||||
}
|
||||
|
||||
function writeAtomic(p, text) {
|
||||
|
|
@ -184,20 +553,31 @@ function writeAtomic(p, text) {
|
|||
}
|
||||
|
||||
/** The CLI's work: check the arguments, read, finalize, write. */
|
||||
export function finalizeFile({ review, repo, project, round, roundBase }) {
|
||||
export function finalizeFile({ review, repo, project, round, roundBase, previous }) {
|
||||
if (!review) return { exit: 2, reason: '--review is required' };
|
||||
if (!repo) return { exit: 2, reason: '--repo is required' };
|
||||
if (!project) return { exit: 2, reason: '--project is required' };
|
||||
if (!/^[1-9]\d*$/.test(String(round ?? '')) || Number(round) > MAX_ROUND) {
|
||||
return { exit: 2, reason: `--round must be an integer 1-${MAX_ROUND}` };
|
||||
if (!/^[1-9]\d*$/.test(String(round ?? '')) || Number(round) > MAX_REVIEW_ROUNDS) {
|
||||
return { exit: 2, reason: `--round must be an integer 1-${MAX_REVIEW_ROUNDS}` };
|
||||
}
|
||||
const R = Number(round);
|
||||
if (roundBase !== undefined && !FULL_SHA_RE.test(roundBase)) return { exit: 2, reason: '--round-base must be a full commit id' };
|
||||
if (R >= 2 && (previous === undefined || roundBase === undefined)) {
|
||||
return { exit: 2, reason: 'round 2 and later need --previous and --round-base (the values prepare printed)' };
|
||||
}
|
||||
if (R === 1 && previous !== undefined) return { exit: 2, reason: '--previous is for round 2 and later' };
|
||||
try {
|
||||
if (!statSync(project).isDirectory()) return { exit: 2, reason: `--project ${project} is not a directory` };
|
||||
} catch (e) { return { exit: 2, reason: `--project ${project}: ${e.message}` }; }
|
||||
let text;
|
||||
try { text = readFileSync(review, 'utf8'); } catch (e) { return { exit: 2, reason: `cannot read ${review}: ${e.message}` }; }
|
||||
const r = finalizeReview(text, { repo });
|
||||
let prev = null;
|
||||
if (previous !== undefined) {
|
||||
const p = readPrevious(previous, text, R);
|
||||
if (p.exit !== 0) return p;
|
||||
prev = p.previous;
|
||||
}
|
||||
const r = finalizeReview(text, { repo, round: R, roundBase: roundBase ?? null, previous: prev, evidence: roundEvidence(project) });
|
||||
if (r.exit !== 0) return r;
|
||||
try { writeAtomic(review, r.text); } catch (e) { return { exit: 2, reason: `cannot write ${review}: ${e.message}` }; }
|
||||
const { text: _t, ...rest } = r;
|
||||
|
|
@ -215,7 +595,7 @@ if (isMain(import.meta.url)) {
|
|||
try {
|
||||
result = finalizeFile({
|
||||
review: opt(args, '--review'), repo: opt(args, '--repo'), project: opt(args, '--project'),
|
||||
round: opt(args, '--round'), roundBase: opt(args, '--round-base'),
|
||||
round: opt(args, '--round'), roundBase: opt(args, '--round-base'), previous: opt(args, '--previous'),
|
||||
});
|
||||
} catch (e) {
|
||||
result = { exit: 2, reason: `review-finalize crashed: ${e.message}` };
|
||||
|
|
|
|||
314
lib/review/review-rounds.mjs
Normal file
314
lib/review/review-rounds.mjs
Normal file
|
|
@ -0,0 +1,314 @@
|
|||
// lib/review/review-rounds.mjs
|
||||
// /trekreview rounds (review_version 1.2, fix round 4.5b2 part B): round 2
|
||||
// and later see the diff since the previous round and the previous findings,
|
||||
// carry what is not resolved, and stop at one cap.
|
||||
//
|
||||
// `MAX_REVIEW_ROUNDS` is the one cap: /trekreview refuses a fourth round, and
|
||||
// /trekexecute's review loop reads the same constant (`--cap`).
|
||||
//
|
||||
// prepare({projectDir, since, repo, workflow, dryRun}): the round of a run.
|
||||
// `<run>` is the first 12 hex of `since` (the run's `scope_sha_start`).
|
||||
// - Same run (review.md exists and its `scope_sha_start` equals `since`): a
|
||||
// later round. The archives `review-round-<run>-<n>.md` whose bytes differ
|
||||
// from review.md are A (an archive equal to review.md was written before a
|
||||
// stop at this point). The previous review's round is P = 1 + |A|, derived,
|
||||
// never read from the `round` field alone: a `round` field that disagrees
|
||||
// is exit 2. The new round is P + 1.
|
||||
// - Round MAX_REVIEW_ROUNDS + 1 is refused (exit 1); so is round 2 or later
|
||||
// with `workflow` (exit 1): the --workflow path neither hands the reviewers
|
||||
// the round input nor writes their outputs.
|
||||
// - The round base is the previous `scope_sha_end`: a full sha that is an
|
||||
// ancestor of HEAD, else exit 2 (not measured).
|
||||
// - review.md is copied to `review-round-<run>-<P>.md`; an existing file
|
||||
// there is kept when its bytes are equal, else exit 2.
|
||||
// - Otherwise (no review.md, or another run): round 1.
|
||||
// - Clean slate, every round: the regular files `.review-engine-payload-*.json`
|
||||
// and `.review-out-*.md` of the project folder are deleted, so any such
|
||||
// file finalize finds was written this round.
|
||||
// - `dryRun` computes the same result and writes, copies and deletes nothing.
|
||||
// Nothing is written before every check has passed.
|
||||
//
|
||||
// carry(previous, current, {marks, ran, previousRan}): a current finding
|
||||
// matches a previous one in the same file with the same rule_key, or with
|
||||
// `contentOverlap` >= DEFAULT_THRESHOLD (lib/review/plan-review-dedup.mjs); ids
|
||||
// are never compared, and each finding matches at most one other. A previous
|
||||
// finding resolves only on positive evidence of this round (the isResolved
|
||||
// rule of lib/review/critic-diff.mjs, with "every reviewer that ran then"
|
||||
// standing in for the raisers):
|
||||
// - a matched previous finding is carried;
|
||||
// - an engine finding is resolved only when its engine is in `ran.engines`,
|
||||
// it was not raised again, and no reviewer marks it `resolved: false`;
|
||||
// - a reviewer-only finding is resolved only when `previousRan.reviewers` is
|
||||
// non-empty, every reviewer in it is in `ran.reviewers`, each marked it
|
||||
// `resolved: true`, and none marked it `resolved: false`;
|
||||
// - `previousRan` null (the previous review is not finalized): nothing resolves.
|
||||
// An unresolved previous finding stays in `findings` with `carried_from` (the
|
||||
// previous id), at the higher of the two severities; a matching current
|
||||
// finding folds into it and keeps its place (and its id until finalize
|
||||
// recomputes ids), so it is not counted new. A new finding keeps its severity
|
||||
// wherever it cites: rule A never reaches reviews (decision 95). Two
|
||||
// occurrences of one class stay two findings (`rule_key#n` ids).
|
||||
//
|
||||
// CLI:
|
||||
// node lib/review/review-rounds.mjs prepare --project <d> --since <sha>
|
||||
// [--repo <d>] [--workflow] [--dry-run] --json
|
||||
// prints one JSON line {round, round_base, previous, run} or {reason};
|
||||
// exit 0 = the round, 1 = refused (the cap, or round 2+ on --workflow),
|
||||
// 2 = not measured (2 is never green)
|
||||
// node lib/review/review-rounds.mjs --cap prints MAX_REVIEW_ROUNDS
|
||||
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { copyFileSync, lstatSync, readFileSync, readdirSync, unlinkSync } from 'node:fs';
|
||||
import { join } from 'node:path';
|
||||
import { contentOverlap, DEFAULT_THRESHOLD } from './plan-review-dedup.mjs';
|
||||
import { FULL_SHA_RE, reviewedSha } from './reviewed-commit.mjs';
|
||||
import { scopeShas } from './pre-existing.mjs';
|
||||
import { getRule } from './rule-catalogue.mjs';
|
||||
import { isMain } from '../util/is-main.mjs';
|
||||
|
||||
export const MAX_REVIEW_ROUNDS = 3;
|
||||
|
||||
const SEVERITY_RANK = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'];
|
||||
const CLEAN_SLATE_RE = /^(?:\.review-engine-payload-.+\.json|\.review-out-.+\.md)$/;
|
||||
|
||||
/** The run id of a round: the first 12 hex of the run's `scope_sha_start`. */
|
||||
export const runId = (sha) => String(sha ?? '').toLowerCase().slice(0, 12);
|
||||
|
||||
/** The archive name of round `n` of run `run`. */
|
||||
export const archiveName = (run, n) => `review-round-${run}-${n}.md`;
|
||||
|
||||
/** A frontmatter field as written (raw), or null when absent. */
|
||||
export function frontmatterField(text, key) {
|
||||
const fm = /^---\r?\n([\s\S]*?)\r?\n---\r?\n/.exec(String(text ?? ''));
|
||||
if (!fm) return null;
|
||||
const m = new RegExp(`^${key}:[ \\t]*(.*?)[ \\t]*$`, 'm').exec(fm[1]);
|
||||
return m ? m[1] : null;
|
||||
}
|
||||
|
||||
/** A regular file's bytes, or null (missing, a symlink, a directory, unreadable). */
|
||||
function readRegular(p) {
|
||||
try {
|
||||
if (!lstatSync(p).isFile()) return null;
|
||||
return readFileSync(p);
|
||||
} catch { return null; }
|
||||
}
|
||||
|
||||
/** The regular files of `dir` whose names match `re`. */
|
||||
function regularFiles(dir, re) {
|
||||
let names;
|
||||
try { names = readdirSync(dir); } catch { return []; }
|
||||
return names.filter((n) => re.test(n)).filter((n) => {
|
||||
try { return lstatSync(join(dir, n)).isFile(); } catch { return false; }
|
||||
}).sort();
|
||||
}
|
||||
|
||||
const notMeasured = (reason, detail = '') => ({ exit: 2, reason: `${reason} — not measured${detail ? ` (${detail})` : ''}` });
|
||||
|
||||
/**
|
||||
* The round of a run, with its archive and clean slate (see the header).
|
||||
* @returns {{exit: 0, round: number, round_base: string|null, previous: string|null, run: string, archived?: boolean, deleted: string[]} | {exit: 1|2, reason: string}}
|
||||
*/
|
||||
export function prepare({ projectDir, since, repo, workflow = false, dryRun = false } = {}) {
|
||||
if (!projectDir) return { exit: 2, reason: '--project is required' };
|
||||
if (!FULL_SHA_RE.test(String(since ?? ''))) return notMeasured('--since must be a full commit id');
|
||||
try {
|
||||
if (!lstatSync(projectDir).isDirectory()) return notMeasured(`--project ${projectDir} is not a directory`);
|
||||
} catch (e) { return notMeasured(`--project ${projectDir}: ${e.message}`); }
|
||||
const run = runId(since);
|
||||
const slate = regularFiles(projectDir, CLEAN_SLATE_RE);
|
||||
const clean = () => {
|
||||
if (dryRun) return [];
|
||||
for (const n of slate) unlinkSync(join(projectDir, n));
|
||||
return slate;
|
||||
};
|
||||
const reviewPath = join(projectDir, 'review.md');
|
||||
let reviewThere = true;
|
||||
try { lstatSync(reviewPath); } catch { reviewThere = false; }
|
||||
const review = readRegular(reviewPath);
|
||||
if (reviewThere && review === null) return notMeasured('review.md is not a readable regular file');
|
||||
const text = review === null ? null : review.toString('utf8');
|
||||
const sameRun = text !== null && scopeShas(text).start !== '' && scopeShas(text).start.toLowerCase() === String(since).toLowerCase();
|
||||
if (!sameRun) {
|
||||
// Round 1, on either path.
|
||||
return { exit: 0, round: 1, round_base: null, previous: null, run, deleted: clean() };
|
||||
}
|
||||
|
||||
const archiveRe = new RegExp(`^review-round-${run}-(\\d+)\\.md$`);
|
||||
const archives = regularFiles(projectDir, archiveRe);
|
||||
const others = archives.filter((n) => !readRegular(join(projectDir, n)).equals(review));
|
||||
const P = 1 + others.length;
|
||||
const roundField = frontmatterField(text, 'round');
|
||||
if (roundField !== null && roundField !== String(P)) {
|
||||
return notMeasured('round count does not match the archives', `round: ${roundField}, ${others.length} earlier archive(s) of run ${run}`);
|
||||
}
|
||||
const round = P + 1;
|
||||
if (round > MAX_REVIEW_ROUNDS) {
|
||||
return { exit: 1, reason: `review round cap reached (${MAX_REVIEW_ROUNDS}) — the operator re-plans` };
|
||||
}
|
||||
if (workflow) {
|
||||
return { exit: 1, reason: 'from round 2 /trekreview runs on the standard path — re-run without --workflow' };
|
||||
}
|
||||
const base = reviewedSha(text);
|
||||
if (!base || !isAncestorOfHead(repo ?? projectDir, base)) {
|
||||
return notMeasured('round base is not an ancestor of HEAD');
|
||||
}
|
||||
const previous = archiveName(run, P);
|
||||
const target = join(projectDir, previous);
|
||||
let present = true;
|
||||
try { lstatSync(target); } catch { present = false; }
|
||||
// A file there that is not a regular file with review.md's bytes "differs".
|
||||
if (present && !readRegular(target)?.equals(review)) return notMeasured(`archive ${previous} differs`);
|
||||
let archived = false;
|
||||
if (!dryRun && !present) {
|
||||
copyFileSync(reviewPath, target);
|
||||
archived = true;
|
||||
}
|
||||
return { exit: 0, round, round_base: base, previous, run, archived, deleted: clean() };
|
||||
}
|
||||
|
||||
/** `git merge-base --is-ancestor <sha> HEAD` in `repo` succeeded. */
|
||||
function isAncestorOfHead(repo, sha) {
|
||||
const r = spawnSync('git', ['-C', repo, 'merge-base', '--is-ancestor', sha, 'HEAD'], { encoding: 'utf8' });
|
||||
return !r.error && r.status === 0;
|
||||
}
|
||||
|
||||
// ---- carry ----------------------------------------------------------------------
|
||||
|
||||
const sevOf = (f) => String(f?.severity ?? '').toUpperCase();
|
||||
/** The more severe of two severities; an unknown one ranks as BLOCKER (never lower). */
|
||||
function higher(a, b) {
|
||||
const ra = SEVERITY_RANK.indexOf(a);
|
||||
const rb = SEVERITY_RANK.indexOf(b);
|
||||
if (ra === -1) return a || 'BLOCKER';
|
||||
if (rb === -1) return b || 'BLOCKER';
|
||||
return ra <= rb ? a : b;
|
||||
}
|
||||
const textOf = (f) => `${f?.title ?? ''} ${f?.detail ?? ''}`;
|
||||
|
||||
/**
|
||||
* The carry rule: same file, then the same rule_key or a content overlap at
|
||||
* or above the dedup threshold. Ids are never compared.
|
||||
*/
|
||||
export function sameFinding(a, b) {
|
||||
if (!a || !b || typeof a.file !== 'string' || a.file === '' || a.file !== b.file) return false;
|
||||
if (typeof a.rule_key === 'string' && a.rule_key !== '' && a.rule_key === b.rule_key) return true;
|
||||
return contentOverlap(textOf(a), textOf(b)) >= DEFAULT_THRESHOLD;
|
||||
}
|
||||
|
||||
/** Lexicographic comparison of two equal-length number lists. */
|
||||
function before(a, b) {
|
||||
for (let j = 0; j < a.length; j++) if (a[j] !== b[j]) return a[j] < b[j];
|
||||
return false;
|
||||
}
|
||||
|
||||
/** The current finding `p` folds into: same rule_key first, then the nearest line, then the larger overlap, then order. */
|
||||
function bestMatch(p, current, taken) {
|
||||
let best = -1;
|
||||
let key = null;
|
||||
current.forEach((c, i) => {
|
||||
if (taken.has(i) || !sameFinding(p, c)) return;
|
||||
const k = [
|
||||
c.rule_key === p.rule_key ? 0 : 1,
|
||||
Number.isInteger(c.line) && Number.isInteger(p.line) ? Math.abs(c.line - p.line) : Number.MAX_SAFE_INTEGER,
|
||||
-contentOverlap(textOf(p), textOf(c)),
|
||||
];
|
||||
if (key === null || before(k, key)) {
|
||||
best = i;
|
||||
key = k;
|
||||
}
|
||||
});
|
||||
return best;
|
||||
}
|
||||
|
||||
/** A security finding (the review-answers.mjs `isSecurityFinding` rule; not imported, that module imports this one). */
|
||||
const isSecurityLike = (f) => getRule(f?.rule_key)?.category === 'security' || f?.engine === 'security-review';
|
||||
|
||||
/** The folded finding: the current one's place and text, the higher severity, a security origin never lost. */
|
||||
function fold(p, c) {
|
||||
const out = { ...c, severity: higher(sevOf(p), sevOf(c)), carried_from: p.id };
|
||||
if (isSecurityLike(p) && !isSecurityLike(c)) {
|
||||
out.rule_key = p.rule_key;
|
||||
if (p.engine !== undefined) out.engine = p.engine;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** A previous finding carried as it was: no id (finalize computes it), no old marks. */
|
||||
function carried(p) {
|
||||
const { id: _id, pre_existing: _pre, carried_from: _cf, ...rest } = p;
|
||||
return { id: null, ...rest, severity: sevOf(p) || 'BLOCKER', carried_from: p.id };
|
||||
}
|
||||
|
||||
/** Was previous finding `p` resolved this round (positive evidence only)? */
|
||||
function resolvedNow(p, { marks, ran, previousRan }) {
|
||||
if (!previousRan || typeof previousRan !== 'object') return false;
|
||||
const ms = (Array.isArray(marks) ? marks : []).filter((m) => m && m.id === p.id);
|
||||
if (ms.some((m) => m.resolved === false)) return false;
|
||||
const ranEngines = Array.isArray(ran?.engines) ? ran.engines : [];
|
||||
const ranReviewers = Array.isArray(ran?.reviewers) ? ran.reviewers : [];
|
||||
if (typeof p.engine === 'string' && p.engine !== '') return ranEngines.includes(p.engine);
|
||||
const then = Array.isArray(previousRan.reviewers) ? previousRan.reviewers : [];
|
||||
if (then.length === 0) return false;
|
||||
return then.every((r) => ranReviewers.includes(r) && ms.some((m) => m.reviewer === r && m.resolved === true));
|
||||
}
|
||||
|
||||
/**
|
||||
* Carry the previous round's findings into this round's (see the header).
|
||||
* @param {object[]} previous - the previous review's JSON findings
|
||||
* @param {object[]} current - this round's JSON findings
|
||||
* @param {{ marks?: Array<{id, resolved, reviewer}>, ran?: {engines, reviewers}, previousRan?: {engines, reviewers}|null }} ctx
|
||||
* @returns {{ findings: object[], resolved: string[], folded: number }}
|
||||
*/
|
||||
export function carry(previous, current, { marks = [], ran = { engines: [], reviewers: [] }, previousRan = null } = {}) {
|
||||
const prev = (Array.isArray(previous) ? previous : []).filter((p) => p && typeof p === 'object');
|
||||
const cur = (Array.isArray(current) ? current : []).filter((c) => c && typeof c === 'object');
|
||||
const findings = cur.map((c) => ({ ...c }));
|
||||
const taken = new Set();
|
||||
const appended = [];
|
||||
const resolved = [];
|
||||
for (const p of prev) {
|
||||
const i = bestMatch(p, cur, taken);
|
||||
if (i >= 0) {
|
||||
taken.add(i);
|
||||
findings[i] = fold(p, cur[i]);
|
||||
continue;
|
||||
}
|
||||
if (resolvedNow(p, { marks, ran, previousRan })) {
|
||||
if (typeof p.id === 'string') resolved.push(p.id);
|
||||
continue;
|
||||
}
|
||||
appended.push(carried(p));
|
||||
}
|
||||
return { findings: [...findings, ...appended], resolved, folded: taken.size };
|
||||
}
|
||||
|
||||
// ---- CLI ------------------------------------------------------------------------
|
||||
|
||||
function opt(args, name) {
|
||||
const i = args.indexOf(name);
|
||||
return i >= 0 ? args[i + 1] : undefined;
|
||||
}
|
||||
|
||||
if (isMain(import.meta.url)) {
|
||||
const args = process.argv.slice(2);
|
||||
if (args[0] === '--cap') {
|
||||
process.stdout.write(`${MAX_REVIEW_ROUNDS}\n`);
|
||||
process.exit(0);
|
||||
}
|
||||
let result;
|
||||
if (args[0] !== 'prepare') {
|
||||
process.stderr.write('Usage: review-rounds.mjs prepare --project <d> --since <sha> [--repo <d>] [--workflow] [--dry-run] --json | --cap\n');
|
||||
process.exit(2);
|
||||
}
|
||||
try {
|
||||
result = prepare({
|
||||
projectDir: opt(args, '--project'), since: opt(args, '--since'), repo: opt(args, '--repo'),
|
||||
workflow: args.includes('--workflow'), dryRun: args.includes('--dry-run'),
|
||||
});
|
||||
} catch (e) {
|
||||
result = { exit: 2, reason: `review-rounds crashed: ${e.message} — not measured` };
|
||||
}
|
||||
const { exit, ...rest } = result;
|
||||
process.stdout.write(`${JSON.stringify(rest)}\n`);
|
||||
process.exit(exit);
|
||||
}
|
||||
|
|
@ -29,7 +29,13 @@
|
|||
// open one is answered `fixed`; `decideOpenFindings`, read against
|
||||
// the top-level approval log of `--repo`, else of the review file's
|
||||
// git work tree, else no records). The `--json` output carries
|
||||
// `unanswered` and `fixed_pending` id arrays.
|
||||
// `unanswered` and `fixed_pending` id arrays. A round 2+ review that
|
||||
// review-finalize.mjs did not finalize is exit 2 too (reason
|
||||
// `review-not-measured: round 2+ review not finalized`): it lacks
|
||||
// `finalized: true` while an archive `review-round-<run>-<n>.md` of its
|
||||
// own run (<run> = the first 12 hex of `scope_sha_start`) sits beside
|
||||
// it; the previous findings reach a later round only through
|
||||
// finalize's carry.
|
||||
// Exit 2 is NEVER green.
|
||||
//
|
||||
// The pure `decideVerdict` stays text-only: it never reads the answers. The
|
||||
|
|
@ -49,7 +55,7 @@
|
|||
// `templates/checklist-template.md:8-9`, MIT). The blocking itself is Voyage's
|
||||
// own design — neither source blocks on its own.
|
||||
|
||||
import { readFileSync, existsSync } from 'node:fs';
|
||||
import { readFileSync, existsSync, readdirSync } from 'node:fs';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { dirname } from 'node:path';
|
||||
import { validateReviewContent, VERDICT_VALUES } from '../validators/review-validator.mjs';
|
||||
|
|
@ -58,8 +64,9 @@ import { illegalBriefLowering } from './coordinator-contract.mjs';
|
|||
import { reviewedSha, checkReviewedCommit } from './reviewed-commit.mjs';
|
||||
import { isMain } from '../util/is-main.mjs';
|
||||
import { resolveRepo } from '../util/approval-log.mjs';
|
||||
import { answerRecords, decideAnswers, reviewSubject } from './review-answers.mjs';
|
||||
import { preExistingCheck } from './pre-existing.mjs';
|
||||
import { answerRecords, archivedReader, decideAnswers, reviewSubject } from './review-answers.mjs';
|
||||
import { preExistingCheck, scopeShas } from './pre-existing.mjs';
|
||||
import { frontmatterField, runId } from './review-rounds.mjs';
|
||||
|
||||
export const EXIT = Object.freeze({ PASS: 0, BLOCK: 1, UNDECIDED: 2 });
|
||||
|
||||
|
|
@ -185,6 +192,31 @@ function blockerSectionEntries(body) {
|
|||
return (section.match(/^###\s+\S/gm) ?? []).length;
|
||||
}
|
||||
|
||||
const NOT_FINALIZED = 'review-not-measured: round 2+ review not finalized';
|
||||
|
||||
/**
|
||||
* A round 2+ review that finalize did not finalize, or null. From round 2 the
|
||||
* previous findings reach review.md only through finalize's carry, so such a
|
||||
* review is never read: it lacks `finalized: true` while an archive
|
||||
* `review-round-<run>-<n>.md` of its own run (lib/review/review-rounds.mjs
|
||||
* `runId` of `scope_sha_start`) sits beside it. A folder that cannot be
|
||||
* listed is not measured either.
|
||||
* @param {string} text - review.md
|
||||
* @param {string|undefined} reviewPath
|
||||
* @returns {string|null}
|
||||
*/
|
||||
export function unfinalizedRound(text, reviewPath) {
|
||||
if (frontmatterField(text, 'finalized') === 'true') return null;
|
||||
const { start } = scopeShas(text);
|
||||
if (!reviewPath || !start) return null;
|
||||
const archive = new RegExp(`^review-round-${runId(start)}-\\d+\\.md$`);
|
||||
let names;
|
||||
try { names = readdirSync(dirname(reviewPath)); } catch (e) {
|
||||
return `review-not-measured: cannot list the review folder (${e.message})`;
|
||||
}
|
||||
return names.some((n) => archive.test(n)) ? NOT_FINALIZED : null;
|
||||
}
|
||||
|
||||
/**
|
||||
* The open-findings check: every counting MAJOR of an ALLOW/WARN review must be
|
||||
* answered (lib/review/review-answers.mjs `decideAnswers`). K2 records come
|
||||
|
|
@ -194,15 +226,25 @@ function blockerSectionEntries(body) {
|
|||
* `pre_existing: true` and git agrees over `scope_sha_start..scope_sha_end`
|
||||
* in `repo` (lib/review/pre-existing.mjs `preExistingCheck`); with no repo, or
|
||||
* no full `scope_sha_start`, no finding is pre-existing. A BLOCKER or a
|
||||
* security finding counts whatever the mark.
|
||||
* security finding counts whatever the mark. A carried answer row is read
|
||||
* against its archive beside the review file (`archivedReader`); with no
|
||||
* review path no carried row answers. A finding with `carried_from` is never
|
||||
* pre-existing: it keeps its line from the previous round's end. A round 2+
|
||||
* review that was not finalized (`unfinalizedRound`) is exit 2 first.
|
||||
* @param {string} text - review.md
|
||||
* @param {{ repo?: string, reviewPath: string }} opts
|
||||
* @returns {{ exit: 0|2, reason: string, unanswered: string[], fixed_pending: string[] }}
|
||||
*/
|
||||
export function decideOpenFindings(text, { repo, reviewPath } = {}) {
|
||||
const notFinal = unfinalizedRound(text, reviewPath);
|
||||
if (notFinal) return { exit: EXIT.UNDECIDED, reason: notFinal, unanswered: [], fixed_pending: [] };
|
||||
const { records, top, problem } = answerRecords(repo ?? (reviewPath ? dirname(reviewPath) : undefined));
|
||||
const subject = top && reviewPath ? reviewSubject(top, reviewPath) : undefined;
|
||||
const d = decideAnswers(text, { records, reviewSubject: subject, preExisting: preExistingCheck(text, repo) });
|
||||
// A carried `## Answers` row (Carried from review-round-<run>-<n>.md#<id>)
|
||||
// is checked against that archive, read beside the review file.
|
||||
const archived = reviewPath ? archivedReader(dirname(reviewPath)) : undefined;
|
||||
const pre = preExistingCheck(text, repo);
|
||||
const d = decideAnswers(text, { records, reviewSubject: subject, preExisting: (f) => !f.carried_from && pre(f), archived });
|
||||
if (d.ok) return { exit: EXIT.PASS, reason: 'every counting MAJOR is answered', unanswered: [], fixed_pending: [] };
|
||||
return {
|
||||
exit: EXIT.UNDECIDED,
|
||||
|
|
|
|||
4
tests/fixtures/jsonl-schemas.md
vendored
4
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -22,10 +22,10 @@
|
|||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, intent_approved, project_dir, order_id, order_claimed_at, order_reason | commands/trekbrief.md (orchestrator-emit Phase 7; order_* added by lib/stats/stats-append.mjs) | trekbrief.md:1141-1165 | profile, phase_models, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | lib/stats/stats-append.mjs (trekresearch.md Stats tracking) | trekresearch.md:829-862 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | lib/stats/stats-append.mjs (trekplan.md Phase 12) | trekplan.md:1307-1335 | profile, phase_models, parallel_agents, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2990-3012 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | lib/stats/stats-append.mjs (trekexecute.md Phase 9) | trekexecute.md:2992-3014 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:78-99 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
| trekexecute-stats (PostToolUse Bash) | ts, session_id, command_excerpt, duration_ms, success | hooks/scripts/post-bash-stats.mjs (Bash PostToolUse) | post-bash-stats.mjs:42-54 | none (hook is plugin-level, not profile-aware) | command_excerpt (CWE-212) |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:654-704 | profile, phase_models, profile_source | none |
|
||||
| trekreview-stats | ts, slug, verdict, counts (BLOCKER/MAJOR/MINOR/SUGGESTION), reviewed_files_count, mode, duration_ms | lib/stats/stats-append.mjs (trekreview.md Phase 8) | trekreview.md:708-758 | profile, phase_models, profile_source | none |
|
||||
| trekcontinue-stats | ts, project, next_session_label, status | lib/stats/stats-append.mjs (trekcontinue.md Phase 5) | trekcontinue.md:304-308 | profile, profile_source | none |
|
||||
| token-usage | ts, session_id, scope, model, tokens_input, tokens_output, tokens_cache_creation, tokens_cache_read, cost_usd, is_estimate, price_table_version | lib/stats/token-usage.mjs `captureTokenUsage` (via hooks/scripts/otel-export.mjs Stop hook, opt-in `VOYAGE_TOKEN_METER`) | token-usage.mjs:buildRecord | n/a (SKAL-2 schema) | session_id (stripped at export) |
|
||||
|
||||
|
|
|
|||
|
|
@ -56,11 +56,12 @@ test('P11: the review runs by itself after Phase 7, in Cherny\'s order, before P
|
|||
|
||||
test('P11: a BLOCKER becomes new numbered plan steps, in a bounded loop that ends with the operator', () => {
|
||||
const p = section(CMD, '## Phase 7.2 — Simplify, review, verdict (P11)');
|
||||
assert.match(p, /TREKEXECUTE_MAX_REVIEW_ROUNDS/, 'the cap is named');
|
||||
assert.match(p, /default \*\*3\*\*/);
|
||||
assert.match(p, /`MAX_REVIEW_ROUNDS`/, 'the cap is named');
|
||||
assert.match(p, /lib\/review\/review-rounds\.mjs/, 'the constant\'s module is named');
|
||||
assert.match(p, /\*\*3\*\*/);
|
||||
assert.match(p, /### Step R<n>/, 'the review\'s remediation steps are the source');
|
||||
assert.match(p, /never reworded/);
|
||||
assert.match(p, /at the cap[^\n]*operator/i);
|
||||
assert.match(p, /at the cap with BLOCKERs or MAJORs answered "fixed" still open goes to the operator/i);
|
||||
assert.match(p, /not improvised/);
|
||||
});
|
||||
|
||||
|
|
|
|||
|
|
@ -8,6 +8,11 @@
|
|||
// the Versioning paragraph says what 1.1 -> 1.2 changed, MIGRATION.md
|
||||
// holds the 1.2 section with its N-1 window, and every other place that
|
||||
// states the current review_version says 1.2.
|
||||
// RND /trekreview names the round input: `review-rounds.mjs prepare` in
|
||||
// Phase 3, the round diff from `{round_base}` and the previous findings
|
||||
// to mark in Phase 5, and the round fields passed to finalize in Phase 7.
|
||||
// CAP one cap of three rounds: /trekreview and /trekexecute name
|
||||
// `review-rounds.mjs` and `MAX_REVIEW_ROUNDS`, and no env name.
|
||||
// The helpers below are copied, never imported from an earlier prose test.
|
||||
// Fix the SOURCE a pin guards, not the pin.
|
||||
|
||||
|
|
@ -114,3 +119,30 @@ test('[V12] every current review_version statement names 1.2', () => {
|
|||
assert.ok(arch.includes('review_version 1.2'), 'docs/architecture.md names review_version 1.2');
|
||||
assert.ok(!arch.includes('review_version 1.1'), 'docs/architecture.md no longer names review_version 1.1');
|
||||
});
|
||||
|
||||
// --- RND / CAP: review rounds (Step 6) ----------------------------------------------
|
||||
|
||||
test('[RND] /trekreview names the round input: prepare, the round diff, the previous findings to mark, never re-raised as new, every occurrence', () => {
|
||||
const cmd = read('commands/trekreview.md');
|
||||
const p3 = section(cmd, '## Phase 3 — Discover scope SHA range');
|
||||
assert.ok(p3.includes('node ${CLAUDE_PLUGIN_ROOT}/lib/review/review-rounds.mjs prepare --project "{project_dir}" --since {before_sha} --json'),
|
||||
'Phase 3 runs review-rounds.mjs prepare');
|
||||
const p5 = flat(section(cmd, '## Phase 5 — Launch parallel reviewers'));
|
||||
assert.ok(p5.includes('git diff {round_base}..{after_sha}'), 'Phase 5 gives the reviewers the diff from {round_base}');
|
||||
assert.ok(p5.includes('previous: [{id, resolved}]'), 'Phase 5 asks the reviewers to mark the previous findings as previous: [{id, resolved}]');
|
||||
assert.match(p5, /a previous finding is never re-raised as new/, 'Phase 5 says a previous finding is never re-raised as new');
|
||||
assert.match(p5, /flag every occurrence of a finding class/, 'Phase 5 asks for every occurrence of a finding class');
|
||||
const p7 = flat(section(cmd, '## Phase 7 — Write review.md'));
|
||||
for (const part of ['--round {round}', '--round-base {round_base}', '--previous "{project_dir}/{previous}"', 'review-answers.mjs carry']) {
|
||||
assert.ok(p7.includes(part), `Phase 7 names ${part}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('[CAP] /trekreview and /trekexecute both name review-rounds.mjs and MAX_REVIEW_ROUNDS, and neither names TREKEXECUTE_MAX_REVIEW_ROUNDS', () => {
|
||||
for (const rel of ['commands/trekreview.md', 'commands/trekexecute.md']) {
|
||||
const text = read(rel);
|
||||
assert.ok(text.includes('review-rounds.mjs'), `${rel} names review-rounds.mjs`);
|
||||
assert.ok(text.includes('MAX_REVIEW_ROUNDS'), `${rel} names MAX_REVIEW_ROUNDS`);
|
||||
assert.ok(!text.includes('TREKEXECUTE_MAX_REVIEW_ROUNDS'), `${rel} no longer names TREKEXECUTE_MAX_REVIEW_ROUNDS`);
|
||||
}
|
||||
});
|
||||
|
|
|
|||
910
tests/lib/review-rounds.test.mjs
Normal file
910
tests/lib/review-rounds.test.mjs
Normal file
|
|
@ -0,0 +1,910 @@
|
|||
// tests/lib/review-rounds.test.mjs
|
||||
// [RND] [CAP] [CRY] fix round 4.5b2 part B, Step 6: /trekreview rounds converge.
|
||||
// RND `lib/review/review-rounds.mjs prepare` derives the round from the
|
||||
// run's archives (`review-round-<run>-<n>.md`, <run> = the first 12 hex
|
||||
// of scope_sha_start), checks the round base, archives the previous
|
||||
// review and clears the previous round's reviewer and engine files;
|
||||
// `carry` keeps a previous finding until this round gives positive
|
||||
// evidence that it is resolved; review-finalize.mjs runs the carry pass
|
||||
// with `--previous`, writes `ran`, `resolved`, `round`, `round_base`,
|
||||
// `finalized` and an R-step for a carried BLOCKER.
|
||||
// CAP one cap of three rounds (`MAX_REVIEW_ROUNDS`).
|
||||
// CRY an accepted answer is carried to the next round only by a link to the
|
||||
// same run's earlier archive, checked by the gate.
|
||||
// Tests run the CLIs (prepare, finalize, review-answers, validator, gate) in
|
||||
// throwaway git repos with no hooks; `carry` is also called directly. Each
|
||||
// tagged test's main assertion is red at the step's start; a case that
|
||||
// already holds there is a Control inside it.
|
||||
|
||||
import { after, test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { appendFileSync, copyFileSync, existsSync, mkdirSync, mkdtempSync, readFileSync, readdirSync, rmSync, writeFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url';
|
||||
import { computeFindingId, computeFindingIds } from '../../lib/parsers/finding-id.mjs';
|
||||
import { childEnv } from '../helpers/child-env.mjs';
|
||||
|
||||
const ROOT = join(dirname(fileURLToPath(import.meta.url)), '..', '..');
|
||||
const ROUNDS = join(ROOT, 'lib', 'review', 'review-rounds.mjs');
|
||||
const FINALIZE = join(ROOT, 'lib', 'review', 'review-finalize.mjs');
|
||||
const ANSWERS = join(ROOT, 'lib', 'review', 'review-answers.mjs');
|
||||
const ENGINE = join(ROOT, 'lib', 'review', 'engine-findings.mjs');
|
||||
const GATE = join(ROOT, 'lib', 'review', 'verdict-gate.mjs');
|
||||
const VALIDATOR = join(ROOT, 'lib', 'validators', 'review-validator.mjs');
|
||||
const PROJECT_REL = '.claude/projects/2026-11-02-r';
|
||||
const TRAILER = 'Co-Authored-By: Claude Test <noreply@example.invalid>';
|
||||
const CCR = 'code-correctness-reviewer';
|
||||
const BCR = 'brief-conformance-reviewer';
|
||||
|
||||
// Loaded so that each new assertion fails on its own while the module is absent.
|
||||
let rounds = {};
|
||||
try {
|
||||
rounds = await import(pathToFileURL(ROUNDS).href);
|
||||
} catch (e) {
|
||||
if (existsSync(ROUNDS)) throw e;
|
||||
}
|
||||
|
||||
const made = [];
|
||||
after(() => { for (const b of made) rmSync(b, { recursive: true, force: true }); });
|
||||
|
||||
function g(cwd, ...args) {
|
||||
const r = spawnSync('git', args, { cwd, encoding: 'utf8', env: childEnv() });
|
||||
if (r.status !== 0) throw new Error(`git ${args.join(' ')}: ${r.stderr}`);
|
||||
return r.stdout.trim();
|
||||
}
|
||||
|
||||
function putIn(dir, rel, text) {
|
||||
mkdirSync(dirname(join(dir, rel)), { recursive: true });
|
||||
writeFileSync(join(dir, rel), text);
|
||||
}
|
||||
|
||||
const parse = (r) => { try { return JSON.parse(r.stdout); } catch { return {}; } };
|
||||
const run = (args, opts = {}) => spawnSync(process.execPath, args, { encoding: 'utf8', env: childEnv(), ...opts });
|
||||
// Ten lines `export const <tag><n> = <n>;`; `changes` maps a line number to new text.
|
||||
const ten = (tag, changes = {}) => Array.from({ length: 10 }, (_, i) => (i + 1 in changes ? changes[i + 1] : `export const ${tag}${i + 1} = ${i + 1};`)).join('\n') + '\n';
|
||||
const jsonOf = (text) => JSON.parse(text.slice(text.lastIndexOf('```json\n') + 8, text.lastIndexOf('```')));
|
||||
const fmOf = (text) => text.slice(0, text.indexOf('\n---\n', 4) + 5);
|
||||
|
||||
/**
|
||||
* A finding as a reviewer writes it. `f` holds severity, file, line and the
|
||||
* optional rule_key, engine, title, detail, id and carried_from.
|
||||
*/
|
||||
function finding(f, i) {
|
||||
return {
|
||||
...(f.id ? { id: f.id } : {}),
|
||||
severity: f.severity, rule_key: f.rule_key ?? 'MISSING_TEST', ...(f.engine ? { engine: f.engine } : {}),
|
||||
file: f.file, line: f.line, brief_ref: 'SC1', title: f.title ?? `title ${f.file} ${f.line} ${i + 1}`,
|
||||
detail: f.detail ?? `detail ${f.file} ${f.line} ${i + 1}`, recommended_action: f.action ?? 'Add a test.',
|
||||
...(f.carried_from ? { carried_from: f.carried_from } : {}),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A review_version 1.2 review of `start..end`, ids filled as fill-finding-ids
|
||||
* fills them (computed over the array). One `### Step R<n>` per BLOCKER
|
||||
* unless `steps` names them ([{title, source}]).
|
||||
*/
|
||||
function rndReview({ start, end, findings, verdict, steps, extraSteps = '' }) {
|
||||
const list = findings.map(finding);
|
||||
const ids = computeFindingIds(list);
|
||||
list.forEach((f, i) => { if (!f.id) f.id = ids[i]; });
|
||||
const ordered = list.map(({ id, ...rest }) => ({ id, ...rest }));
|
||||
const counts = { BLOCKER: 0, MAJOR: 0, MINOR: 0, SUGGESTION: 0 };
|
||||
for (const f of ordered) counts[f.severity] += 1;
|
||||
const v = verdict ?? (counts.BLOCKER > 0 ? 'BLOCK' : counts.MAJOR > 0 ? 'WARN' : 'ALLOW');
|
||||
const json = JSON.stringify({ verdict: v, counts, allow_blocked_by: [], findings: ordered }, null, 2);
|
||||
const sections = ['BLOCKER', 'MAJOR', 'MINOR', 'SUGGESTION'].map((sev) => {
|
||||
const of = ordered.filter((f) => f.severity === sev);
|
||||
if (of.length === 0) return '';
|
||||
return `## Findings (${sev})\n\n${of.map((f) => `### ${f.id}\n\n- file: ${f.file}\n- line: ${f.line}\n- rule_key: ${f.rule_key}\n- title: ${f.title}\n`).join('\n')}\n`;
|
||||
}).join('');
|
||||
const stepList = steps ?? ordered.filter((f) => f.severity === 'BLOCKER').map((f) => ({ title: `fix ${f.title}`, source: f.id, file: f.file }));
|
||||
const stepText = stepList.map((s, i) => `### Step R${i + 1}: ${s.title}\n\n- **Files:** \`${s.file ?? 'a.mjs'}\`\n- **Source finding:** \`${s.source}\`\n- **Test first:** a test that fails today\n- **Verify:** re-run /trekreview\n- **On failure:** escalate\n`).join('\n');
|
||||
return {
|
||||
ids: ordered.map((f) => f.id),
|
||||
text: `---
|
||||
type: trekreview
|
||||
review_version: "1.2"
|
||||
task: "t"
|
||||
slug: r
|
||||
project_dir: ${PROJECT_REL}/
|
||||
brief_path: ${PROJECT_REL}/brief.md
|
||||
scope_sha_start: ${start}
|
||||
scope_sha_end: ${end}
|
||||
reviewed_files_count: 2
|
||||
verdict: ${v}
|
||||
findings:
|
||||
${ordered.length ? ordered.map((f) => ` - ${f.id}`).join('\n') : ' []'}
|
||||
---
|
||||
|
||||
# Review: t
|
||||
|
||||
## Executive Summary
|
||||
|
||||
x
|
||||
|
||||
## Coverage
|
||||
|
||||
| File | Treatment | Reason |
|
||||
|------|-----------|--------|
|
||||
| a.mjs | deep-review | default |
|
||||
|
||||
${sections}## Declined to Judge
|
||||
|
||||
None.
|
||||
|
||||
Trimmed: 0
|
||||
|
||||
## Remediation Steps
|
||||
|
||||
${stepText || 'None.\n'}${extraSteps}
|
||||
## Remediation Summary
|
||||
|
||||
x
|
||||
|
||||
\`\`\`json
|
||||
${json}
|
||||
\`\`\`
|
||||
`.replace('findings:\n []', 'findings: []'),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* A throwaway repo: start (a.mjs, b.mjs, ten lines each), c1 (a.mjs and b.mjs
|
||||
* line 3 changed: round 1's work), c2 (a.mjs line 5 changed: round 2's work).
|
||||
* HEAD is c2. The project folder is excluded from git, as a local artifact.
|
||||
*/
|
||||
function rndRepo() {
|
||||
const base = mkdtempSync(join(tmpdir(), 'review-rounds-rnd-'));
|
||||
made.push(base);
|
||||
const repo = join(base, 'repo');
|
||||
mkdirSync(repo, { recursive: true });
|
||||
mkdirSync(join(base, 'hooks-none'));
|
||||
g(repo, 'init', '-q', '-b', 'main');
|
||||
appendFileSync(join(repo, '.git', 'config'), '[user]\n\tname = Test\n\temail = test@example.invalid\n'
|
||||
+ `[commit]\n\tgpgsign = false\n[core]\n\thooksPath = "${join(base, 'hooks-none')}"\n`);
|
||||
appendFileSync(join(repo, '.git', 'info', 'exclude'), '.claude/projects/\n');
|
||||
const commit = (files, msg) => {
|
||||
for (const [rel, text] of Object.entries(files)) putIn(repo, rel, text);
|
||||
g(repo, 'add', '-A');
|
||||
g(repo, 'commit', '-q', '-m', msg);
|
||||
return g(repo, 'rev-parse', 'HEAD');
|
||||
};
|
||||
const start = commit({ 'a.mjs': ten('a'), 'b.mjs': ten('b') }, 'chore: start');
|
||||
const c1 = commit({ 'a.mjs': ten('a', { 3: 'export const a3 = 33;' }), 'b.mjs': ten('b', { 3: 'export const b3 = 33;' }) }, 'feat: round 1');
|
||||
const c2 = commit({ 'a.mjs': ten('a', { 3: 'export const a3 = 33;', 5: 'export const a5 = 55;' }) }, 'feat: round 2');
|
||||
const project = join(repo, ...PROJECT_REL.split('/'));
|
||||
mkdirSync(project, { recursive: true });
|
||||
const reviewPath = join(project, 'review.md');
|
||||
const fx = {
|
||||
base, repo, start, c1, c2, project, reviewPath, run: start.slice(0, 12), commit,
|
||||
write: (text, p = reviewPath) => writeFileSync(p, text),
|
||||
read: (p = reviewPath) => readFileSync(p, 'utf8'),
|
||||
archive: (n, runId = fx.run) => join(project, `review-round-${runId}-${n}.md`),
|
||||
/** Every file name in the project folder with its bytes. */
|
||||
snapshot: () => Object.fromEntries(readdirSync(project).sort().map((n) => [n, readFileSync(join(project, n), 'utf8')])),
|
||||
prepare: (extra = [], since = start) => run([ROUNDS, 'prepare', '--project', project, '--since', since, '--json', ...extra], { cwd: repo }),
|
||||
finalize: (round, extra = []) => run([FINALIZE, '--review', reviewPath, '--repo', repo, '--project', project,
|
||||
'--round', String(round), ...extra], { cwd: repo }),
|
||||
finalize2: () => fx.finalize(2, ['--round-base', c1, '--previous', fx.archive(1)]),
|
||||
gate: () => run([GATE, '--json', reviewPath, '--repo', repo]),
|
||||
validate: () => run([VALIDATOR, '--strict', reviewPath]),
|
||||
/** A reviewer's whole answer, saved as Phase 5 saves it; `previous` marks previous findings by id. */
|
||||
out: (reviewer, previous = [], findings = []) => writeFileSync(join(project, `.review-out-${reviewer}.md`),
|
||||
`Reviewer prose.\n\n\`\`\`json\n${JSON.stringify({ reviewer, findings, previous }, null, 2)}\n\`\`\`\n`),
|
||||
/** An engine payload, converted by the real engine-findings.mjs CLI. */
|
||||
engine: (engine, findings = []) => {
|
||||
const input = join(base, `engine-${engine}.json`);
|
||||
writeFileSync(input, JSON.stringify({ engine, findings }));
|
||||
const r = run([ENGINE, input]);
|
||||
assert.equal(r.status, 0, `engine-findings: ${r.stdout}${r.stderr}`);
|
||||
writeFileSync(join(project, `.review-engine-payload-${engine}.json`), r.stdout);
|
||||
},
|
||||
};
|
||||
return fx;
|
||||
}
|
||||
|
||||
/** Round 1 written, finalized and archived by prepare: the folder at the start of round 2. */
|
||||
function roundOne(fx, findings, { outs = [CCR, BCR], engines = ['code-review', 'security-review'], finalize = true } = {}) {
|
||||
const r1 = rndReview({ start: fx.start, end: fx.c1, findings });
|
||||
fx.write(r1.text);
|
||||
for (const name of outs) fx.out(name);
|
||||
for (const e of engines) fx.engine(e);
|
||||
if (finalize) {
|
||||
const f = fx.finalize(1);
|
||||
assert.equal(f.status, 0, `round-1 finalize: ${f.stdout}${f.stderr}`);
|
||||
}
|
||||
const p = fx.prepare();
|
||||
assert.equal(p.status, 0, `prepare for round 2: ${p.stdout}${p.stderr}`);
|
||||
return { ...r1, prev: jsonOf(fx.read(fx.archive(1))) };
|
||||
}
|
||||
|
||||
/** Round 2's coordinator output, with fresh reviewer and engine files. */
|
||||
function roundTwo(fx, findings, { outs = {}, engines = ['code-review', 'security-review'], ...opts } = {}) {
|
||||
const r2 = rndReview({ start: fx.start, end: fx.c2, findings, ...opts });
|
||||
fx.write(r2.text);
|
||||
for (const [name, marks] of Object.entries(outs)) fx.out(name, marks);
|
||||
for (const e of engines) fx.engine(e);
|
||||
return r2;
|
||||
}
|
||||
|
||||
const MAJ_A3 = { severity: 'MAJOR', file: 'a.mjs', line: 3, title: 'Parser drops the last token', detail: 'parse() stops one token early' };
|
||||
const BLK_B3 = { severity: 'BLOCKER', file: 'b.mjs', line: 3, rule_key: 'BROKEN_SUCCESS_CRITERION', title: 'Build output is never written', detail: 'write() returns before the flush' };
|
||||
|
||||
// ---- [RND] range and the fail-closed base ----------------------------------------
|
||||
|
||||
test('[RND] from round 2, prepare returns the previous scope_sha_end as round_base', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
const r1 = rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] });
|
||||
fx.write(r1.text);
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const out = parse(r);
|
||||
assert.equal(out.round, 2, r.stdout);
|
||||
assert.equal(out.round_base, fx.c1, r.stdout);
|
||||
assert.equal(out.previous, `review-round-${fx.run}-1.md`, r.stdout);
|
||||
assert.equal(fx.read(fx.archive(1)), r1.text, 'the previous review is archived byte for byte');
|
||||
});
|
||||
|
||||
test('[RND] prepare exits 2 when the previous scope_sha_end is not an ancestor of HEAD', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
// A side branch from c1, as in [R6] verdict-gate.test.mjs.
|
||||
g(fx.repo, 'checkout', '-q', '-b', 'side', fx.c1);
|
||||
const side = fx.commit({ 'b.mjs': ten('b', { 3: 'export const b3 = 33;', 9: 'export const b9 = 99;' }) }, 'feat: side');
|
||||
g(fx.repo, 'checkout', '-q', 'main');
|
||||
fx.write(rndReview({ start: fx.start, end: side, findings: [MAJ_A3] }).text);
|
||||
const before = fx.snapshot();
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
assert.match(String(parse(r).reason), /round base is not an ancestor of HEAD — not measured/, r.stdout);
|
||||
assert.deepEqual(fx.snapshot(), before, 'a refused prepare archives nothing');
|
||||
// A short sha is no round base either.
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1.slice(0, 12), findings: [MAJ_A3] }).text);
|
||||
const s = fx.prepare();
|
||||
assert.equal(s.status, 2, s.stdout + s.stderr);
|
||||
// Control: the round-1 end on main is an ancestor of HEAD.
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
const c = fx.prepare(['--dry-run']);
|
||||
assert.equal(c.status, 0, `Control: ${c.stdout}${c.stderr}`);
|
||||
});
|
||||
|
||||
// ---- [RND] carry -------------------------------------------------------------------
|
||||
|
||||
const P = (f, i = 0) => ({ id: computeFindingId(f.file, f.line, f.rule_key ?? 'MISSING_TEST'), ...finding(f, i) });
|
||||
|
||||
test('[RND] a previous finding is carried until marked resolved by every reviewer', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const p = P(MAJ_A3);
|
||||
const both = { engines: [], reviewers: [CCR, BCR] };
|
||||
const one = rounds.carry([p], [], { marks: [{ id: p.id, resolved: true, reviewer: CCR }], ran: both, previousRan: both });
|
||||
assert.equal(one.findings.length, 1, JSON.stringify(one));
|
||||
assert.equal(one.findings[0].carried_from, p.id);
|
||||
assert.equal(one.findings[0].severity, 'MAJOR');
|
||||
assert.deepEqual(one.resolved, []);
|
||||
const all = rounds.carry([p], [], {
|
||||
marks: [{ id: p.id, resolved: true, reviewer: CCR }, { id: p.id, resolved: true, reviewer: BCR }], ran: both, previousRan: both,
|
||||
});
|
||||
assert.deepEqual(all.findings, []);
|
||||
assert.deepEqual(all.resolved, [p.id]);
|
||||
});
|
||||
|
||||
test('[RND] a repeated finding is folded and not counted as new', () => {
|
||||
const fx = rndRepo();
|
||||
roundOne(fx, [MAJ_A3]);
|
||||
// The same finding raised again with other words: same file, same rule_key.
|
||||
roundTwo(fx, [{ ...MAJ_A3, title: 'Last token lost by the parser' }], { outs: { [CCR]: [] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const payload = jsonOf(fx.read());
|
||||
assert.equal(payload.findings.length, 1, JSON.stringify(payload.findings));
|
||||
assert.equal(payload.findings[0].carried_from, computeFindingId('a.mjs', 3, 'MISSING_TEST'), 'the re-raised finding is the carried one');
|
||||
assert.deepEqual(payload.counts, { BLOCKER: 0, MAJOR: 1, MINOR: 0, SUGGESTION: 0 }, 'one MAJOR, not two');
|
||||
});
|
||||
|
||||
test('[RND] a new MAJOR outside the round diff keeps MAJOR', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const fresh = P({ severity: 'MAJOR', file: 'b.mjs', line: 3, title: 'New thing' });
|
||||
const r = rounds.carry([], [fresh], { marks: [], ran: { engines: [], reviewers: [CCR] }, previousRan: { engines: [], reviewers: [CCR] } });
|
||||
assert.deepEqual(r.findings.map((x) => [x.severity, x.carried_from]), [['MAJOR', undefined]]);
|
||||
// In a finalized round 2: b.mjs:3 changed in round 1 only (outside c1..c2), and it still counts as MAJOR.
|
||||
const fx = rndRepo();
|
||||
roundOne(fx, [MAJ_A3]);
|
||||
roundTwo(fx, [MAJ_A3, { severity: 'MAJOR', file: 'b.mjs', line: 3, title: 'New thing' }], { outs: { [CCR]: [] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const fresh2 = jsonOf(fx.read()).findings.find((x) => x.file === 'b.mjs');
|
||||
assert.equal(fresh2.severity, 'MAJOR');
|
||||
assert.equal(fresh2.carried_from, undefined);
|
||||
const gate = fx.gate();
|
||||
assert.equal(gate.status, 2, gate.stdout);
|
||||
assert.ok(parse(gate).unanswered.includes(fresh2.id), `the new MAJOR counts: ${gate.stdout}`);
|
||||
});
|
||||
|
||||
test('[RND] two occurrences of one class stay two findings', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const ran = { engines: [], reviewers: [CCR] };
|
||||
const p1 = P({ severity: 'MAJOR', file: 'a.mjs', line: 3 });
|
||||
const p2 = P({ severity: 'MAJOR', file: 'a.mjs', line: 8 });
|
||||
const both = rounds.carry([p1, p2], [P({ severity: 'MAJOR', file: 'a.mjs', line: 8 }), P({ severity: 'MAJOR', file: 'a.mjs', line: 3 })],
|
||||
{ marks: [], ran, previousRan: ran });
|
||||
assert.deepEqual(both.findings.map((x) => `${x.line}<-${x.carried_from}`).sort(), [`3<-${p1.id}`, `8<-${p2.id}`]);
|
||||
// One previous occurrence, two current ones on the same line: one carried, one new, two ids (rule_key#2).
|
||||
const fx = rndRepo();
|
||||
roundOne(fx, [{ severity: 'MAJOR', file: 'a.mjs', line: 3 }]);
|
||||
roundTwo(fx, [{ severity: 'MAJOR', file: 'a.mjs', line: 3 }, { severity: 'MAJOR', file: 'a.mjs', line: 3, title: 'second' }], { outs: { [CCR]: [] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const list = jsonOf(fx.read()).findings;
|
||||
assert.equal(list.length, 2, JSON.stringify(list));
|
||||
assert.equal(list.filter((x) => x.carried_from).length, 1, JSON.stringify(list));
|
||||
assert.deepEqual(list.map((x) => x.id), computeFindingIds(list));
|
||||
assert.notEqual(list[0].id, list[1].id);
|
||||
});
|
||||
|
||||
test('[RND] a review.md from another run starts at round 1', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.c1, end: fx.c2, findings: [MAJ_A3] }).text);
|
||||
const before = fx.snapshot();
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
assert.deepEqual([parse(r).round, parse(r).round_base, parse(r).previous], [1, null, null], r.stdout);
|
||||
assert.deepEqual(fx.snapshot(), before, 'nothing is archived for another run');
|
||||
});
|
||||
|
||||
// ---- [RND] engine findings ----------------------------------------------------------
|
||||
|
||||
const ENG_MAJ = { severity: 'MAJOR', file: 'b.mjs', line: 3, rule_key: 'CODE_REVIEW_DEFECT', engine: 'code-review', title: 'Off by one in loop' };
|
||||
|
||||
test('[RND] an engine MAJOR is resolved when its engine ran this round and did not raise it again', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const p = P(ENG_MAJ);
|
||||
const prevRan = { engines: ['code-review'], reviewers: [CCR] };
|
||||
const r = rounds.carry([p], [], { marks: [], ran: { engines: ['code-review'], reviewers: [CCR] }, previousRan: prevRan });
|
||||
assert.deepEqual([r.findings.length, r.resolved], [0, [p.id]]);
|
||||
});
|
||||
|
||||
test('[RND] an engine MAJOR is carried when its engine did not run, or raised it again, whatever a reviewer marks', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const p = P(ENG_MAJ);
|
||||
const prevRan = { engines: ['code-review'], reviewers: [CCR] };
|
||||
const marks = [{ id: p.id, resolved: true, reviewer: CCR }];
|
||||
const notRun = rounds.carry([p], [], { marks, ran: { engines: [], reviewers: [CCR] }, previousRan: prevRan });
|
||||
assert.deepEqual(notRun.findings.map((x) => x.carried_from), [p.id]);
|
||||
assert.deepEqual(notRun.resolved, []);
|
||||
const again = rounds.carry([p], [P({ ...ENG_MAJ, line: 4 })], { marks, ran: { engines: ['code-review'], reviewers: [CCR] }, previousRan: prevRan });
|
||||
assert.deepEqual(again.findings.map((x) => x.carried_from), [p.id]);
|
||||
assert.deepEqual(again.resolved, []);
|
||||
});
|
||||
|
||||
test('[RND] a --quick round 2 (no security payload) carries every previous security-review finding', () => {
|
||||
const fx = rndRepo();
|
||||
const sec = { severity: 'BLOCKER', file: 'b.mjs', line: 3, rule_key: 'SECURITY_REVIEW_VULNERABILITY', engine: 'security-review', title: 'Path escapes the root' };
|
||||
const { prev } = roundOne(fx, [sec, ENG_MAJ], { outs: [CCR] });
|
||||
const [secId, majId] = prev.findings.map((x) => x.id);
|
||||
// --quick: only /code-review ran and wrote its payload.
|
||||
roundTwo(fx, [], { outs: { [CCR]: [{ id: secId, resolved: true }, { id: majId, resolved: true }] }, engines: ['code-review'] });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const payload = jsonOf(fx.read());
|
||||
assert.deepEqual(payload.ran, { engines: ['code-review'], reviewers: [CCR] });
|
||||
assert.deepEqual(payload.findings.map((x) => x.carried_from), [secId], 'the security finding is carried');
|
||||
assert.deepEqual(payload.resolved, [majId], 'Control: the code-review MAJOR is resolved by a clean code-review run');
|
||||
});
|
||||
|
||||
test('[RND] an engine that ran clean while a reviewer marks the finding resolved: false carries it', () => {
|
||||
assert.equal(typeof rounds.carry, 'function', 'review-rounds.mjs exports carry');
|
||||
const p = P(ENG_MAJ);
|
||||
const ran = { engines: ['code-review'], reviewers: [CCR] };
|
||||
const r = rounds.carry([p], [], { marks: [{ id: p.id, resolved: false, reviewer: CCR }], ran, previousRan: ran });
|
||||
assert.deepEqual([r.findings.map((x) => x.carried_from), r.resolved], [[p.id], []]);
|
||||
});
|
||||
|
||||
// ---- [RND] reviewer-only findings ---------------------------------------------------
|
||||
|
||||
test('[RND] a reviewer-only finding: one reviewer\'s true while the other round-1 reviewer did not run carries it; both true resolves it', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [MAJ_A3]);
|
||||
const [id] = prev.findings.map((x) => x.id);
|
||||
roundTwo(fx, [], { outs: { [CCR]: [{ id, resolved: true }] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const one = jsonOf(fx.read());
|
||||
assert.deepEqual(one.findings.map((x) => x.carried_from), [id], 'carried: brief-conformance-reviewer ran in round 1, not in round 2');
|
||||
assert.deepEqual(one.resolved, []);
|
||||
roundTwo(fx, [], { outs: { [CCR]: [{ id, resolved: true }], [BCR]: [{ id, resolved: true }] } });
|
||||
const f2 = fx.finalize2();
|
||||
assert.equal(f2.status, 0, f2.stdout + f2.stderr);
|
||||
const both = jsonOf(fx.read());
|
||||
assert.deepEqual(both.findings, []);
|
||||
assert.deepEqual(both.resolved, [id]);
|
||||
const text = fx.read();
|
||||
assert.match(text, /^## Resolved since round 1$/m, text);
|
||||
assert.ok(text.slice(text.indexOf('## Resolved since round 1')).includes('Parser drops the last token'), text);
|
||||
});
|
||||
|
||||
// ---- [RND] round count ------------------------------------------------------------------
|
||||
|
||||
test('[RND] a round-1 review with no round field, then prepare, returns round 2 and writes review-round-<run>-1.md; a second prepare after a stop returns round 2 again', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
const r1 = rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] });
|
||||
assert.doesNotMatch(fmOf(r1.text), /^round:/m, 'the round-1 review has no round field (finalize failed)');
|
||||
fx.write(r1.text);
|
||||
const a = fx.prepare();
|
||||
assert.equal(a.status, 0, a.stdout + a.stderr);
|
||||
assert.equal(parse(a).round, 2, a.stdout);
|
||||
assert.equal(fx.read(fx.archive(1)), r1.text);
|
||||
// A stop at this point: review.md is still round 1, and the archive equals it.
|
||||
const b = fx.prepare();
|
||||
assert.equal(b.status, 0, b.stdout + b.stderr);
|
||||
assert.deepEqual([parse(b).round, parse(b).previous], [2, `review-round-${fx.run}-1.md`], b.stdout);
|
||||
assert.equal(fx.read(fx.archive(1)), r1.text, 'the archive is kept');
|
||||
assert.ok(!existsSync(fx.archive(2)), 'no second archive');
|
||||
});
|
||||
|
||||
test('[RND] an archive with other bytes is exit 2; a round field that disagrees with the archives is exit 2', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
fx.write('another review\n', fx.archive(2));
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
assert.match(String(parse(r).reason), new RegExp(`archive review-round-${fx.run}-2\\.md differs — not measured`), r.stdout);
|
||||
assert.equal(fx.read(fx.archive(2)), 'another review\n', 'the differing archive is left alone');
|
||||
rmSync(fx.archive(2));
|
||||
// round: 2 with no archive of round 1.
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text.replace('\nfindings:', '\nround: 2\nfindings:'));
|
||||
const m = fx.prepare();
|
||||
assert.equal(m.status, 2, m.stdout + m.stderr);
|
||||
assert.match(String(parse(m).reason), /round count does not match the archives — not measured/, m.stdout);
|
||||
});
|
||||
|
||||
test('[RND] prepare --dry-run returns the round and leaves the folder byte for byte unchanged', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
fx.out(CCR);
|
||||
fx.engine('code-review');
|
||||
const before = fx.snapshot();
|
||||
const r = fx.prepare(['--dry-run']);
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
assert.deepEqual([parse(r).round, parse(r).round_base], [2, fx.c1], r.stdout);
|
||||
assert.deepEqual(fx.snapshot(), before);
|
||||
});
|
||||
|
||||
test('[RND] a second run in the same folder starts at round 1, and its round 2 archives under its own run', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
assert.equal(fx.prepare().status, 0);
|
||||
const first = fx.read(fx.archive(1));
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c2, findings: [MAJ_A3] }).text);
|
||||
// Run 2 starts at c1.
|
||||
const one = fx.prepare([], fx.c1);
|
||||
assert.equal(one.status, 0, one.stdout + one.stderr);
|
||||
assert.equal(parse(one).round, 1, one.stdout);
|
||||
fx.write(rndReview({ start: fx.c1, end: fx.c2, findings: [MAJ_A3] }).text);
|
||||
const two = fx.prepare([], fx.c1);
|
||||
assert.equal(two.status, 0, two.stdout + two.stderr);
|
||||
const run2 = fx.c1.slice(0, 12);
|
||||
assert.deepEqual([parse(two).round, parse(two).previous], [2, `review-round-${run2}-1.md`], two.stdout);
|
||||
assert.ok(existsSync(fx.archive(1, run2)));
|
||||
assert.equal(fx.read(fx.archive(1)), first, 'the first run\'s archive is untouched');
|
||||
assert.deepEqual(readdirSync(fx.project).filter((n) => n.startsWith('review-round-')).sort(),
|
||||
[`review-round-${fx.run}-1.md`, `review-round-${run2}-1.md`].sort());
|
||||
});
|
||||
|
||||
test('[RND] prepare --workflow for a round 2 exits 1', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
// Control: round 1 on --workflow is allowed.
|
||||
const c = fx.prepare(['--workflow']);
|
||||
assert.equal(c.status, 0, `Control: ${c.stdout}${c.stderr}`);
|
||||
assert.equal(parse(c).round, 1);
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
const before = fx.snapshot();
|
||||
const r = fx.prepare(['--workflow']);
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.match(String(parse(r).reason), /from round 2 \/trekreview runs on the standard path — re-run without --workflow/, r.stdout);
|
||||
assert.deepEqual(fx.snapshot(), before);
|
||||
});
|
||||
|
||||
// ---- [RND] resolution by evidence -----------------------------------------------------
|
||||
|
||||
test('[RND] a previous review without finalized: true resolves nothing', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [MAJ_A3, ENG_MAJ], { finalize: false });
|
||||
assert.doesNotMatch(fmOf(fx.read(fx.archive(1))), /^finalized:/m);
|
||||
const marks = prev.findings.map((x) => ({ id: x.id, resolved: true }));
|
||||
roundTwo(fx, [], { outs: { [CCR]: marks, [BCR]: marks } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const payload = jsonOf(fx.read());
|
||||
assert.deepEqual(payload.findings.map((x) => x.carried_from).sort(), prev.findings.map((x) => x.id).sort());
|
||||
assert.deepEqual(payload.resolved, []);
|
||||
});
|
||||
|
||||
test('[RND] a previous review with ran.reviewers: [] resolves no reviewer finding', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [MAJ_A3], { outs: [] });
|
||||
assert.deepEqual(prev.ran?.reviewers, [], 'round 1 was finalized with no reviewer output');
|
||||
const [id] = prev.findings.map((x) => x.id);
|
||||
roundTwo(fx, [], { outs: { [CCR]: [{ id, resolved: true }], [BCR]: [{ id, resolved: true }] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const payload = jsonOf(fx.read());
|
||||
assert.deepEqual(payload.findings.map((x) => x.carried_from), [id]);
|
||||
assert.deepEqual(payload.resolved, []);
|
||||
});
|
||||
|
||||
test('[RND] a previous code-review MAJOR is resolved by a code-review payload converted from an empty engine run', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [ENG_MAJ], { outs: [CCR] });
|
||||
const [id] = prev.findings.map((x) => x.id);
|
||||
// Control: with no code-review payload this round, the finding is carried.
|
||||
roundTwo(fx, [], { outs: { [CCR]: [] }, engines: [] });
|
||||
const c = fx.finalize2();
|
||||
assert.equal(c.status, 0, c.stdout + c.stderr);
|
||||
assert.deepEqual(jsonOf(fx.read()).findings.map((x) => x.carried_from), [id]);
|
||||
// `node lib/review/engine-findings.mjs` on {"engine":"code-review","findings":[]}.
|
||||
roundTwo(fx, [], { outs: { [CCR]: [] }, engines: ['code-review'] });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const payload = jsonOf(fx.read());
|
||||
assert.deepEqual(payload.ran.engines, ['code-review']);
|
||||
assert.deepEqual(payload.findings, []);
|
||||
assert.deepEqual(payload.resolved, [id]);
|
||||
});
|
||||
|
||||
test('[RND] prepare deletes old .review-engine-payload-*.json and .review-out-*.md files', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
fx.out(CCR);
|
||||
fx.out(BCR);
|
||||
fx.engine('code-review');
|
||||
fx.engine('security-review');
|
||||
writeFileSync(join(fx.project, '.review-engine-code-review.json'), '{}');
|
||||
writeFileSync(join(fx.project, '.review-scope.txt'), 'a.mjs\n');
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
const left = readdirSync(fx.project).sort();
|
||||
assert.deepEqual(left.filter((n) => n.startsWith('.review-out-') || n.startsWith('.review-engine-payload-')), [], left.join(', '));
|
||||
// Control: the engine transcription and the scope list stay.
|
||||
assert.ok(left.includes('.review-engine-code-review.json') && left.includes('.review-scope.txt'), left.join(', '));
|
||||
});
|
||||
|
||||
// ---- [RND] ids and R-steps after the carry ----------------------------------------------
|
||||
|
||||
/** Every `Source finding:` id under ## Remediation Steps. */
|
||||
const sources = (text) => [...text.slice(text.indexOf('## Remediation Steps')).matchAll(/Source finding:\**\s*`?([0-9a-f]{40})/g)].map((m) => m[1]);
|
||||
|
||||
test('[RND] a finalized round-2 review with a fold and a carried BLOCKER has every id computed, the BLOCKER\'s R-step, and passes the validator and the gate', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [MAJ_A3, BLK_B3]);
|
||||
const blkPrev = prev.findings.find((x) => x.severity === 'BLOCKER').id;
|
||||
// Round 2 re-raises the MAJOR (one line lower: a fold) and not the BLOCKER.
|
||||
roundTwo(fx, [{ ...MAJ_A3, line: 4 }], { outs: { [CCR]: [], [BCR]: [] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const text = fx.read();
|
||||
const payload = jsonOf(text);
|
||||
assert.equal(payload.findings.length, 2, JSON.stringify(payload.findings));
|
||||
assert.deepEqual(payload.findings.map((x) => x.id), computeFindingIds(payload.findings), 'every id is the computed one');
|
||||
const blk = payload.findings.find((x) => x.severity === 'BLOCKER');
|
||||
assert.equal(blk.carried_from, blkPrev);
|
||||
assert.equal(payload.verdict, 'BLOCK', 'the verdict is raised by the carried BLOCKER');
|
||||
assert.match(fmOf(text), /^verdict: BLOCK$/m);
|
||||
assert.deepEqual(payload.counts, { BLOCKER: 1, MAJOR: 1, MINOR: 0, SUGGESTION: 0 });
|
||||
assert.match(fmOf(text), /^round: 2$/m);
|
||||
assert.match(fmOf(text), new RegExp(`^round_base: ${fx.c1}$`, 'm'));
|
||||
assert.match(fmOf(text), /^finalized: true$/m);
|
||||
const fm = fmOf(text);
|
||||
for (const x of payload.findings) {
|
||||
assert.ok(fm.includes(` - ${x.id}\n`), `the frontmatter lists ${x.id}`);
|
||||
assert.ok(text.includes(`### ${x.id}\n`), `the body has an entry for ${x.id}`);
|
||||
}
|
||||
assert.ok(sources(text).includes(blk.id), `the carried BLOCKER has its R-step with the recomputed id:\n${text}`);
|
||||
const v = fx.validate();
|
||||
assert.equal(v.status, 0, `review-validator --strict: ${v.stdout}${v.stderr}`);
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.equal(parse(r).reason, 'verdict BLOCK', r.stdout);
|
||||
});
|
||||
|
||||
test('[RND] a BLOCKER re-raised with the coordinator\'s own R-step keeps exactly that one R-step; an orphan R-step makes finalize exit 1', () => {
|
||||
const fx = rndRepo();
|
||||
const { prev } = roundOne(fx, [BLK_B3]);
|
||||
const blkPrev = prev.findings[0].id;
|
||||
const r2 = roundTwo(fx, [{ ...BLK_B3, line: 4, title: 'Build output never reaches disk' }], { outs: { [CCR]: [], [BCR]: [] } });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 0, f.stdout + f.stderr);
|
||||
const text = fx.read();
|
||||
const payload = jsonOf(text);
|
||||
const carried = payload.findings.find((x) => x.carried_from === blkPrev);
|
||||
assert.ok(carried, `the re-raised BLOCKER is the carried one: ${JSON.stringify(payload.findings)}`);
|
||||
assert.equal((text.match(/^### Step R\d+/gm) ?? []).length, 1, text);
|
||||
assert.ok(text.includes('### Step R1: fix Build output never reaches disk'), 'the one R-step is the coordinator\'s');
|
||||
assert.deepEqual(sources(text), [carried.id]);
|
||||
const ids = new Set(payload.findings.map((x) => x.id));
|
||||
assert.deepEqual(sources(text).filter((s) => !ids.has(s)), [], 'no Source finding names an absent id');
|
||||
assert.equal(fx.validate().status, 0);
|
||||
// A planted orphan R-step.
|
||||
const orphan = '\n### Step R9: fix nothing\n\n- **Files:** `a.mjs`\n- **Source finding:** `' + 'e'.repeat(40) + '`\n- **On failure:** escalate\n';
|
||||
roundTwo(fx, [{ ...BLK_B3, line: 4, title: 'Build output never reaches disk' }], { outs: { [CCR]: [], [BCR]: [] }, extraSteps: orphan });
|
||||
const planted = fx.read();
|
||||
assert.equal(r2.ids.length, 1);
|
||||
const o = fx.finalize2();
|
||||
assert.equal(o.status, 1, o.stdout + o.stderr);
|
||||
assert.equal(fx.read(), planted, 'a refused finalize leaves the file as it was');
|
||||
});
|
||||
|
||||
test('[RND] a round-2 review that finalize did not finalize is not measured: the gate exits 2', () => {
|
||||
const fx = rndRepo();
|
||||
roundOne(fx, [BLK_B3, MAJ_A3]);
|
||||
// Round 2 re-raises nothing; the coordinator's one R-step names an id the review does not hold.
|
||||
roundTwo(fx, [], { outs: { [CCR]: [], [BCR]: [] }, steps: [{ title: 'fix an old finding', source: 'f'.repeat(40) }] });
|
||||
const f = fx.finalize2();
|
||||
assert.equal(f.status, 1, f.stdout + f.stderr);
|
||||
assert.doesNotMatch(fmOf(fx.read()), /^finalized: true$/m, 'a refused finalize leaves the file unfinalized');
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
assert.equal(parse(r).reason, 'review-not-measured: round 2+ review not finalized', r.stdout);
|
||||
// An unfinalized round-2 review with no findings and no R-step: exit 2 as well.
|
||||
roundTwo(fx, [], { outs: { [CCR]: [], [BCR]: [] } });
|
||||
const n = fx.gate();
|
||||
assert.equal(n.status, 2, n.stdout + n.stderr);
|
||||
assert.equal(parse(n).reason, 'review-not-measured: round 2+ review not finalized', n.stdout);
|
||||
// Control: the same review finalized is decided on (the carried BLOCKER blocks).
|
||||
const ok = fx.finalize2();
|
||||
assert.equal(ok.status, 0, `Control: ${ok.stdout}${ok.stderr}`);
|
||||
const b = fx.gate();
|
||||
assert.equal(b.status, 1, `Control: ${b.stdout}${b.stderr}`);
|
||||
// Control: an unfinalized round-1 review with only another run's archive beside it is decided on.
|
||||
const other = rndRepo();
|
||||
other.write(rndReview({ start: other.start, end: other.c2, findings: [] }).text);
|
||||
other.write('another run\n', other.archive(1, 'f'.repeat(12)));
|
||||
const one = other.gate();
|
||||
assert.equal(one.status, 0, `Control: ${one.stdout}${one.stderr}`);
|
||||
});
|
||||
|
||||
test('[RND] a carried MAJOR whose line moved under the round\'s fix is never pre-existing: the gate keeps exit 2', () => {
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
fx.out(CCR);
|
||||
fx.out(BCR);
|
||||
const f1 = fx.finalize(1);
|
||||
assert.equal(f1.status, 0, f1.stdout + f1.stderr);
|
||||
// The round's fix puts two lines on top of a.mjs: the old line 3 is now line 5.
|
||||
const c3 = fx.commit({ 'a.mjs': `// header 1\n// header 2\n${readFileSync(join(fx.repo, 'a.mjs'), 'utf8')}` }, 'fix: round 2 work');
|
||||
const p = fx.prepare();
|
||||
assert.equal(p.status, 0, p.stdout + p.stderr);
|
||||
fx.write(rndReview({ start: fx.start, end: c3, findings: [] }).text);
|
||||
fx.out(CCR);
|
||||
fx.out(BCR);
|
||||
const f2 = fx.finalize(2, ['--round-base', fx.c1, '--previous', fx.archive(1)]);
|
||||
assert.equal(f2.status, 0, f2.stdout + f2.stderr);
|
||||
const text = fx.read();
|
||||
const carried = jsonOf(text).findings.find((x) => x.carried_from);
|
||||
assert.ok(carried, `the MAJOR is carried: ${JSON.stringify(jsonOf(text).findings)}`);
|
||||
assert.equal(carried.line, 3, 'a carried finding keeps its line');
|
||||
assert.equal(carried.pre_existing, false, 'finalize never marks a carried finding pre-existing');
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout + r.stderr);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered: 1 MAJOR/, r.stdout);
|
||||
// A hand-set mark does not take it out either.
|
||||
fx.write(editFindings(text, (x) => (x.carried_from ? { ...x, pre_existing: true } : x)));
|
||||
const h = fx.gate();
|
||||
assert.equal(h.status, 2, h.stdout + h.stderr);
|
||||
assert.match(String(parse(h).reason), /^review-unanswered: 1 MAJOR/, h.stdout);
|
||||
});
|
||||
|
||||
// ---- [CAP] the cap ----------------------------------------------------------------------
|
||||
|
||||
test('[CAP] MAX_REVIEW_ROUNDS is 3, and --cap prints it', () => {
|
||||
assert.equal(rounds.MAX_REVIEW_ROUNDS, 3);
|
||||
const r = run([ROUNDS, '--cap']);
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
assert.equal(r.stdout.trim(), '3');
|
||||
});
|
||||
|
||||
test('[CAP] prepare refuses round 4 with exit 1', () => {
|
||||
assert.ok(existsSync(ROUNDS), 'lib/review/review-rounds.mjs must exist');
|
||||
const fx = rndRepo();
|
||||
const r3 = rndReview({ start: fx.start, end: fx.c2, findings: [MAJ_A3] }).text.replace('\nfindings:', '\nround: 3\nfindings:');
|
||||
fx.write(r3);
|
||||
fx.write('round 1\n', fx.archive(1));
|
||||
fx.write('round 2\n', fx.archive(2));
|
||||
const before = fx.snapshot();
|
||||
const r = fx.prepare();
|
||||
assert.equal(r.status, 1, r.stdout + r.stderr);
|
||||
assert.match(String(parse(r).reason), /review round cap reached \(3\) — the operator re-plans/, r.stdout);
|
||||
assert.deepEqual(fx.snapshot(), before, 'a refused round writes nothing');
|
||||
// Control: round 3 is allowed.
|
||||
rmSync(fx.archive(2));
|
||||
fx.write(r3.replace('round: 3', 'round: 2'));
|
||||
const c = fx.prepare(['--dry-run']);
|
||||
assert.equal(c.status, 0, `Control: ${c.stdout}${c.stderr}`);
|
||||
assert.equal(parse(c).round, 3);
|
||||
});
|
||||
|
||||
// ---- [CRY] carried answers ---------------------------------------------------------------
|
||||
|
||||
/**
|
||||
* Round 1 finalized with a MAJOR on a.mjs:3, answered `accepted` through
|
||||
* review-answers.mjs record (K2 record committed); prepare; round 2 re-raises
|
||||
* the MAJOR; finalize carries it; review-answers.mjs carry writes the row.
|
||||
*/
|
||||
function cryFlow() {
|
||||
const fx = rndRepo();
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [MAJ_A3] }).text);
|
||||
fx.out(CCR);
|
||||
const f1 = fx.finalize(1);
|
||||
assert.equal(f1.status, 0, f1.stdout + f1.stderr);
|
||||
const oldId = jsonOf(fx.read()).findings[0].id;
|
||||
const rec = run([ANSWERS, 'record', '--review', fx.reviewPath, '--finding', oldId, '--answer', 'accepted', '--repo', fx.repo,
|
||||
'--round', '1', '--store', 'local', '--trailer', TRAILER], { input: 'the parser keeps the old contract on purpose' });
|
||||
assert.equal(rec.status, 0, `record: ${rec.stdout}${rec.stderr}`);
|
||||
const record = parse(rec).record;
|
||||
const p = fx.prepare();
|
||||
assert.equal(p.status, 0, p.stdout + p.stderr);
|
||||
roundTwo(fx, [MAJ_A3], { outs: { [CCR]: [] } });
|
||||
const f2 = fx.finalize2();
|
||||
assert.equal(f2.status, 0, f2.stdout + f2.stderr);
|
||||
const carry = run([ANSWERS, 'carry', '--review', fx.reviewPath, '--previous', fx.archive(1), '--repo', fx.repo]);
|
||||
return { fx, oldId, record, carry, link: `review-round-${fx.run}-1.md#${oldId}` };
|
||||
}
|
||||
|
||||
/** The review with its one `## Answers` row's cells replaced by `fn(cells)`. */
|
||||
function editRow(text, fn) {
|
||||
return text.split('\n').map((l) => {
|
||||
if (!/^\| [0-9a-f]{40} \| /.test(l)) return l;
|
||||
return `| ${fn(l.slice(1, -1).split('|').map((c) => c.trim())).join(' | ')} |`;
|
||||
}).join('\n');
|
||||
}
|
||||
|
||||
/** The review with its JSON block's findings changed by `fn` (ids kept). */
|
||||
function editFindings(text, fn) {
|
||||
const at = text.lastIndexOf('```json\n') + 8;
|
||||
const end = text.lastIndexOf('```');
|
||||
const payload = JSON.parse(text.slice(at, end));
|
||||
payload.findings = payload.findings.map(fn);
|
||||
return `${text.slice(0, at)}${JSON.stringify(payload, null, 2)}\n${text.slice(end)}`;
|
||||
}
|
||||
|
||||
test('[CRY] a round-2 review that carries an accepted MAJOR with the link to its earlier answer exits 0', () => {
|
||||
const { fx, oldId, record, carry, link } = cryFlow();
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const text = fx.read();
|
||||
const newId = jsonOf(text).findings[0].id;
|
||||
assert.equal(jsonOf(text).findings[0].carried_from, oldId);
|
||||
assert.ok(text.includes(`| ${newId} | accepted | the parser keeps the old contract on purpose | ${record} | ${link} |`), text);
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 0, r.stdout + r.stderr);
|
||||
});
|
||||
|
||||
test('[CRY] the same answer without a carried finding and link keeps exit 2', () => {
|
||||
const { fx, carry } = cryFlow();
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const good = fx.read();
|
||||
assert.equal(fx.gate().status, 0, 'the carried row opens the gate');
|
||||
// Control: the row without its link.
|
||||
fx.write(editRow(good, (c) => [...c.slice(0, 4), '-']));
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered:/, r.stdout);
|
||||
// Control: the link kept, the finding not carried.
|
||||
fx.write(editFindings(good, ({ carried_from: _c, ...rest }) => rest));
|
||||
const n = fx.gate();
|
||||
assert.equal(n.status, 2, n.stdout);
|
||||
assert.match(String(parse(n).reason), /^review-unanswered:/, n.stdout);
|
||||
});
|
||||
|
||||
test('[CRY] a carried row whose finding no longer matches is unanswered', () => {
|
||||
const { fx, carry } = cryFlow();
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const good = fx.read();
|
||||
assert.equal(fx.gate().status, 0, 'the carried row opens the gate');
|
||||
// Control: the finding (same id, same carried_from) now says something else under another key.
|
||||
fx.write(editFindings(good, (f) => ({ ...f, rule_key: 'MISSING_ERROR_HANDLING', title: 'Unbounded retry storm', detail: 'renderer loops forever on timeout' })));
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered:/, r.stdout);
|
||||
});
|
||||
|
||||
test('[CRY] a row linking to another run\'s archive keeps exit 2, even with a valid K2 record and a content match', () => {
|
||||
const { fx, oldId, carry } = cryFlow();
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const good = fx.read();
|
||||
assert.equal(fx.gate().status, 0, 'the carried row opens the gate');
|
||||
// Control: the same archive bytes under another run's name.
|
||||
const other = 'f'.repeat(12);
|
||||
copyFileSync(fx.archive(1), fx.archive(1, other));
|
||||
fx.write(editRow(good, (c) => [...c.slice(0, 4), `review-round-${other}-1.md#${oldId}`]));
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered:/, r.stdout);
|
||||
});
|
||||
|
||||
test('[CRY] a row linking to an archive whose <n> is not lower than the review\'s round keeps exit 2', () => {
|
||||
const { fx, oldId, carry } = cryFlow();
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const good = fx.read();
|
||||
assert.equal(fx.gate().status, 0, 'the carried row opens the gate');
|
||||
// Control: the same archive bytes as round 2 of this run, in a round-2 review.
|
||||
copyFileSync(fx.archive(1), fx.archive(2));
|
||||
fx.write(editRow(good, (c) => [...c.slice(0, 4), `review-round-${fx.run}-2.md#${oldId}`]));
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered:/, r.stdout);
|
||||
// Control: a review with no round field accepts no carried link.
|
||||
fx.write(good.replace(/^round: 2\n/m, ''));
|
||||
const n = fx.gate();
|
||||
assert.equal(n.status, 2, n.stdout);
|
||||
assert.match(String(parse(n).reason), /^review-unanswered:/, n.stdout);
|
||||
});
|
||||
|
||||
test('[CRY] an answer to one carried occurrence, linked from another carried occurrence of the same class, keeps exit 2', () => {
|
||||
const fx = rndRepo();
|
||||
const Y1 = { severity: 'MAJOR', file: 'a.mjs', line: 3, title: 'Parser drops the last token', detail: 'parse() stops one token early' };
|
||||
const Y2 = { severity: 'MAJOR', file: 'a.mjs', line: 5, title: 'Cache never expires entries', detail: 'ttl ignored in get()' };
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c1, findings: [Y1, Y2] }).text);
|
||||
fx.out(CCR);
|
||||
const f1 = fx.finalize(1);
|
||||
assert.equal(f1.status, 0, f1.stdout + f1.stderr);
|
||||
const [id1, id2] = jsonOf(fx.read()).findings.map((x) => x.id);
|
||||
const rec = run([ANSWERS, 'record', '--review', fx.reviewPath, '--finding', id1, '--answer', 'accepted', '--repo', fx.repo,
|
||||
'--round', '1', '--store', 'local', '--trailer', TRAILER], { input: 'the parser keeps the old contract on purpose' });
|
||||
assert.equal(rec.status, 0, `record: ${rec.stdout}${rec.stderr}`);
|
||||
const p = fx.prepare();
|
||||
assert.equal(p.status, 0, p.stdout + p.stderr);
|
||||
roundTwo(fx, [Y1, Y2], { outs: { [CCR]: [] } });
|
||||
const f2 = fx.finalize2();
|
||||
assert.equal(f2.status, 0, f2.stdout + f2.stderr);
|
||||
const carry = run([ANSWERS, 'carry', '--review', fx.reviewPath, '--previous', fx.archive(1), '--repo', fx.repo]);
|
||||
assert.equal(carry.status, 0, `review-answers carry: ${carry.stdout}${carry.stderr}`);
|
||||
const good = fx.read();
|
||||
const g1 = jsonOf(good).findings.find((x) => x.carried_from === id1);
|
||||
const g2 = jsonOf(good).findings.find((x) => x.carried_from === id2);
|
||||
assert.ok(g1 && g2, `both MAJORs are carried: ${JSON.stringify(jsonOf(good).findings)}`);
|
||||
const row = good.split('\n').find((l) => l.startsWith(`| ${g1.id} | accepted |`));
|
||||
assert.ok(row && row.endsWith(`| review-round-${fx.run}-1.md#${id1} |`), `the carried row of the answered MAJOR:\n${good}`);
|
||||
// Control: only the answered occurrence is answered.
|
||||
const c = fx.gate();
|
||||
assert.equal(c.status, 2, c.stdout);
|
||||
assert.deepEqual(parse(c).unanswered, [g2.id], c.stdout);
|
||||
// The same row copied to the other occurrence: its link names id1, the finding was carried from id2.
|
||||
fx.write(good.replace(row, `${row}\n${row.replace(`| ${g1.id} |`, `| ${g2.id} |`)}`));
|
||||
const r = fx.gate();
|
||||
assert.equal(r.status, 2, r.stdout);
|
||||
assert.match(String(parse(r).reason), /^review-unanswered:/, r.stdout);
|
||||
assert.deepEqual(parse(r).unanswered, [g2.id], r.stdout);
|
||||
// Control: round 3 carries the answer through the round-2 archive; a copy for the other occurrence does not.
|
||||
fx.write(good);
|
||||
const p3 = fx.prepare();
|
||||
assert.equal(p3.status, 0, `Control: ${p3.stdout}${p3.stderr}`);
|
||||
assert.equal(parse(p3).round, 3, p3.stdout);
|
||||
fx.write(rndReview({ start: fx.start, end: fx.c2, findings: [Y1, Y2] }).text);
|
||||
fx.out(CCR, []);
|
||||
const f3 = fx.finalize(3, ['--round-base', fx.c2, '--previous', fx.archive(2)]);
|
||||
assert.equal(f3.status, 0, `Control: ${f3.stdout}${f3.stderr}`);
|
||||
const carry3 = run([ANSWERS, 'carry', '--review', fx.reviewPath, '--previous', fx.archive(2), '--repo', fx.repo]);
|
||||
assert.equal(carry3.status, 0, `Control: ${carry3.stdout}${carry3.stderr}`);
|
||||
const third = fx.read();
|
||||
const h1 = jsonOf(third).findings.find((x) => x.carried_from === g1.id);
|
||||
const h2 = jsonOf(third).findings.find((x) => x.carried_from === g2.id);
|
||||
assert.ok(h1 && h2, `Control: both MAJORs are carried into round 3: ${JSON.stringify(jsonOf(third).findings)}`);
|
||||
const row3 = third.split('\n').find((l) => l.startsWith(`| ${h1.id} | accepted |`));
|
||||
assert.ok(row3 && row3.endsWith(`| review-round-${fx.run}-1.md#${id1} |`), `Control: the round-3 row links where the answer was given:\n${third}`);
|
||||
const c3 = fx.gate();
|
||||
assert.deepEqual(parse(c3).unanswered, [h2.id], `Control: ${c3.stdout}`);
|
||||
fx.write(third.replace(row3, `${row3}\n${row3.replace(`| ${h1.id} |`, `| ${h2.id} |`)}`));
|
||||
const r3 = fx.gate();
|
||||
assert.equal(r3.status, 2, r3.stdout);
|
||||
assert.deepEqual(parse(r3).unanswered, [h2.id], r3.stdout);
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue