fix(docs): the put-back, the research triggers and Pass 3's high-effort skip say what the code does
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
8a2a989f79
commit
f6bdc2de59
6 changed files with 143 additions and 15 deletions
|
|
@ -95,6 +95,9 @@ Fix round 4.5a (decisions 57, 67 and 68): the round's 30 findings, where a Voyag
|
|||
**Stats**
|
||||
|
||||
- **No command promises a stats key its record does not write** (finding #46). The "Stats records emit `profile` …" sentences left `/trekbrief`, `/trekplan`, `/trekexecute`, `/trekreview` and `/trekcontinue`; `/trekresearch` keeps only `engine`, which its record writes; `docs/profiles.md`, `docs/operations.md` and README no longer say a profile reaches the stats records.
|
||||
|
||||
**Across the round**
|
||||
|
||||
- **The texts next to the fixes say the same** (siblings found during the round): `/trekreview`, the review reference and the Arm B coordinator prompt no longer drop a finding whose rule key is unknown; `/trekbrief` writes its stats record on every exit of its manual and auto paths, says that `--gates` adds one pause before its auto chain, and that the chain ends at the plan; `/trekexecute` says `--gates` changes nothing there, checks the wave marker before a low-effort single session, and names no validator check that does not exist; plan mode goes through the plan approval before it prints the `/trekexecute` line; the gates shim reads `--gates=true` as on.
|
||||
|
||||
### Known issues
|
||||
|
|
|
|||
|
|
@ -804,7 +804,7 @@ flowchart TB
|
|||
end
|
||||
subgraph RES["/trekresearch · Phase 4 — parallel"]
|
||||
RL["LOCAL: architecture-mapper · dependency-tracer<br/>task-finder · git-historian · convention-scanner*"]
|
||||
RE["EXTERNAL: docs-researcher · community-researcher<br/>security-researcher* · contrarian-researcher*"]
|
||||
RE["EXTERNAL: docs-researcher · community-researcher<br/>security-researcher · contrarian-researcher*"]
|
||||
end
|
||||
subgraph PL["/trekplan"]
|
||||
PLG["Phase 4b · brief-reviewer (gate)"]
|
||||
|
|
@ -823,7 +823,7 @@ flowchart TB
|
|||
BR --> RES --> PL --> EX --> RV
|
||||
```
|
||||
|
||||
`* = conditional`: convention-scanner / test-strategist on medium+ codebases (50+ files); research-scout for unknown external tech; security-researcher when a leading recommendation forms (or always at `effort=high`); contrarian-researcher, when selected, runs in wave 2 after the other external agents return, with their leading answer to attack (always at `effort=high`); brief-conformance-reviewer skipped under `--quick`.
|
||||
`* = conditional`: convention-scanner / test-strategist on medium+ codebases (50+ files); research-scout for unknown external tech; contrarian-researcher, when selected, runs in wave 2 after wave 1 returns, with the leading answer of wave 1's external findings to attack (with `--engine deep-research`, after the engine returns, from the leading answer of its report), always at `effort=high`; brief-conformance-reviewer skipped under `--quick`.
|
||||
|
||||
> **Which Claude Code primitive each phase uses — and the alternatives considered (Workflow substrate, delegated orchestrator, dormant synthesis-agent)** → see [docs/architecture.md §Primitives per step](docs/architecture.md#primitives-per-step-decision-matrix).
|
||||
|
||||
|
|
|
|||
|
|
@ -191,6 +191,10 @@ The coordinator runs a 4-pass process:
|
|||
`RULE_CATALOGUE` (`unknown-rule_key`) or whose file cannot be checked
|
||||
(`file-existence:indeterminate`) to `unverified`, which forbids `ALLOW`;
|
||||
reset a severity that does not match its catalogue tier.
|
||||
Skipped at high effort, as in `quick` mode (below); at high effort an
|
||||
unknown `rule_key` is normalized to `PLAN_EXECUTE_DRIFT` instead
|
||||
(`agents/review-coordinator.md`, "High-effort normalization"); the
|
||||
brief-conflict rule still applies in every mode.
|
||||
4. **Compute verdict** — `BLOCK` if `BLOCKER ≥ 1`, `WARN` if `MAJOR ≥ 1`,
|
||||
else `ALLOW`.
|
||||
|
||||
|
|
@ -199,8 +203,8 @@ The coordinator's output is the full review.md content — frontmatter
|
|||
and `## Remediation Steps` with one numbered step per BLOCKER) + trailing
|
||||
JSON block — ready to write.
|
||||
|
||||
In `quick` mode, skip pass 3 (reasonableness filter). Passes 1, 2, 4
|
||||
still run.
|
||||
In `quick` mode, skip pass 3 (reasonableness filter), except the
|
||||
brief-conflict rule, which applies in every mode. Passes 1, 2, 4 still run.
|
||||
|
||||
### Phase 7 — Write review.md
|
||||
|
||||
|
|
|
|||
|
|
@ -516,6 +516,9 @@ The coordinator runs the 4-pass process documented in
|
|||
`rule_key` is not in `RULE_CATALOGUE` (`unknown-rule_key`) or whose file
|
||||
cannot be checked (`file-existence:indeterminate`) to `unverified`, which
|
||||
forbids `ALLOW`; reset a severity that does not match its catalogue tier.
|
||||
Skipped at high effort too, where an unknown `rule_key` is normalized to
|
||||
`PLAN_EXECUTE_DRIFT` instead (`### High-effort behavior (v5.1.1)` below);
|
||||
the brief-conflict rule still applies in every mode.
|
||||
4. **Verdict** — BLOCK / WARN / ALLOW per the threshold table.
|
||||
|
||||
**Fail-closed.** Every removal in Pass 2 and Pass 3 is either
|
||||
|
|
@ -563,11 +566,8 @@ Workflow({
|
|||
|
||||
A reviewer that returns nothing, an engine listed in `engineNotRun`, or a coordinator that returns no verdict and findings forbids `ALLOW` on this path too (`missing_reviewers` in the result), and the result's `allow_blocked_by` names each one (`missing-reviewer:<name>`, the coordinator as `missing-reviewer:review-coordinator`);
|
||||
Phase 7 writes that list as the trailing JSON block's `allow_blocked_by`, so the verdict gate reads such a WARN review as not performed in full (exit 2; a BLOCK stays exit 1).
|
||||
On this path dedup ranks as Pass 1 does (both paths), every engine finding the
|
||||
coordinator did not return at its exact `(file, line, rule_key)` is put back at its own
|
||||
severity (operator decision 23), and a returned finding is never below the lower of its
|
||||
reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path
|
||||
is stricter than the prose path.
|
||||
On this path dedup ranks as Pass 1 does (both paths), and after the coordinator each finding is matched to its own row, its `(file, line, rule_key)` triplet plus its title, trimmed and in lower case: of the engine findings in a row no returned finding is on, the one with the highest severity (on a tie, the first seen) is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)); its counterpart is its own row, else the strictest row on its triplet.
|
||||
So the workflow path is stricter than the prose path.
|
||||
|
||||
Contract (verified in S10 part B — follow exactly):
|
||||
- **Pass `args` as a JSON object.** The script defensively re-parses a JSON *string*, but
|
||||
|
|
@ -903,8 +903,8 @@ readable.
|
|||
### High-effort behavior (v5.1.1)
|
||||
|
||||
When `phase_signal_result.effort == 'high'` for the `review` phase,
|
||||
skip Pass 3 (Cloudflare reasonableness filter) in
|
||||
`agents/review-coordinator.md`. Passes 1, 2, and 4 still run.
|
||||
skip Pass 3 (Cloudflare reasonableness filter) in `agents/review-coordinator.md`, except the brief-conflict rule, which applies in every mode.
|
||||
Passes 1, 2, and 4 still run.
|
||||
Rationale: high-effort review trusts the operator to weigh borderline
|
||||
findings rather than have the coordinator drop them. To prevent
|
||||
unknown `rule_key` values from polluting downstream remediation plans
|
||||
|
|
|
|||
|
|
@ -484,7 +484,7 @@ The gate cannot check an `engine` label on the prose path: there the coordinator
|
|||
the workflow path (`--workflow`) takes `engine` from the payload source and writes it into the returned finding.
|
||||
`runContract` (`lib/review/coordinator-contract.mjs`) likewise takes `engine` from the payload a finding arrived in, never from the finding's own field,
|
||||
and in dedup the survivor is ranked by catalogue tier (for a catalogue key it always ties inside one triplet), so a reviewer-agent finding survives over an engine finding and carries the highest severity the group asserted; an engine's mark never lowers what a reviewer agent asserted.
|
||||
On the workflow path dedup ranks as Pass 1 does (both paths), every engine finding the coordinator did not return at its exact `(file, line, rule_key)` is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)), so the workflow path is stricter than the prose path; a coordinator that returns no verdict and findings counts as a missing reviewer there (`missing-reviewer:review-coordinator` in `allow_blocked_by`), and its findings are read at their catalogue tier or higher, so the review is at least WARN and the verdict gate exits 2 on such a WARN (a BLOCK stays exit 1).
|
||||
On the workflow path dedup ranks as Pass 1 does (both paths), and after the coordinator each finding is matched to its own row, its `(file, line, rule_key)` triplet plus its title, trimmed and in lower case: of the engine findings in a row no returned finding is on, the one with the highest severity (on a tie, the first seen) is put back at its own severity (operator decision 23), and a returned finding is never below the lower of its reviewer counterpart's severity and its catalogue tier (floor (b)); its counterpart is its own row, else the strictest row on its triplet, so the workflow path is stricter than the prose path; a coordinator that returns no verdict and findings counts as a missing reviewer there (`missing-reviewer:review-coordinator` in `allow_blocked_by`), and its findings are read at their catalogue tier or higher, so the review is at least WARN and the verdict gate exits 2 on such a WARN (a BLOCK stays exit 1).
|
||||
A plan made from the review skips a lowered finding: it is MINOR, and `/trekplan` takes BLOCKER and MAJOR findings only.
|
||||
|
||||
**Validation strategy:**
|
||||
|
|
|
|||
|
|
@ -3,8 +3,12 @@
|
|||
// .claude/projects/2026-10-20-fix-round-serious-findings): the fixes that are
|
||||
// text a session or a helper reads. The name is the brief's;
|
||||
// lib/util/test-census.mjs counts it as a behaviour test (it does not match
|
||||
// PIN_FILE_RE). Every test carries its id tag in brackets, and every tagged
|
||||
// test fails on an assertion at ab6b32c (the round's start point).
|
||||
// PIN_FILE_RE). A test that carries a round id in brackets fails on an
|
||||
// assertion at ab6b32c (the round's start point), except DJ's pins: they pass
|
||||
// there by design, and tests/fix-round-4-5a-red.sh proves them red by mutation.
|
||||
// Untagged tests (prefixes `helpers:`, `step 25:`, `keep:`, `control:`,
|
||||
// `arch:`, `e2e script:`, `red script:`, `CHANGELOG:`, `found in execution:`)
|
||||
// are outside the round's id list.
|
||||
// #23 code-correctness-reviewer cites a "MANDATORY OVERRIDE rule above"
|
||||
// that is no longer there; the sentence says the rule itself.
|
||||
// #24 research-scout says "Try Tavily first" but its tools held no Tavily
|
||||
|
|
@ -1387,7 +1391,7 @@ test('[#113] no conditional trigger is left, and high effort keeps contrarian-re
|
|||
test('[#113] sibling: README.md', () => {
|
||||
const text = flat(read('README.md'));
|
||||
assert.doesNotMatch(text, /contrarian-researcher when a leading recommendation forms/);
|
||||
assert.ok(text.includes('contrarian-researcher, when selected, runs in wave 2 after the other external agents return, with their leading answer to attack (always at `effort=high`)'),
|
||||
assert.ok(text.includes('contrarian-researcher, when selected, runs in wave 2 after wave 1 returns, with the leading answer of wave 1\'s external findings to attack (with `--engine deep-research`, after the engine returns, from the leading answer of its report), always at `effort=high`;'),
|
||||
'README.md does not describe wave 2');
|
||||
});
|
||||
|
||||
|
|
@ -1727,3 +1731,120 @@ test('CHANGELOG: the released v6.5.0 section is byte-equal to its text at ab6b32
|
|||
assert.equal(createHash('sha256').update(part, 'utf8').digest('hex'), V650_SHA256,
|
||||
'the released v6.5.0 section changed; edit only the round\'s section');
|
||||
});
|
||||
|
||||
// Found in execution (untagged, outside the round's id list): texts next to
|
||||
// the round's fixes that still said what the code no longer does. Each test
|
||||
// fails on an assertion at 8a2a989, before the text changed; its presence
|
||||
// assertion holds there.
|
||||
|
||||
// D1: the workflow matches each finding to its own row (`rowOf` in
|
||||
// scripts/trekreview-armB.workflow.mjs), not to its exact triplet.
|
||||
const PUT_BACK_TEXTS = [
|
||||
['commands/trekreview.md', () => partAt('commands/trekreview.md', 'On this path dedup ranks as Pass 1 does', toBlank)],
|
||||
['docs/HANDOVER-CONTRACTS.md', () => partAt('docs/HANDOVER-CONTRACTS.md', 'On the workflow path dedup ranks as Pass 1 does', () => true)],
|
||||
];
|
||||
const ROW_IS = 'its `(file, line, rule_key)` triplet plus its title, trimmed and in lower case';
|
||||
const COUNTERPART_IS = 'its counterpart is its own row, else the strictest row on its triplet';
|
||||
const PUT_BACK_IS = 'of the engine findings in a row no returned finding is on, the one with the highest severity (on a tie, the first seen) is put back at its own severity (operator decision 23)';
|
||||
|
||||
for (const [rel, part] of PUT_BACK_TEXTS) {
|
||||
test(`found in execution: ${rel} no longer puts back what the coordinator did not return "at its exact (file, line, rule_key)"`, () => {
|
||||
const text = part();
|
||||
assert.ok(text.includes('put back at its own severity'), `${rel}: the put-back sentence is missing: ${text}`);
|
||||
assert.doesNotMatch(text, /at its exact `\(file, line, rule_key\)`/, `${rel}: the put-back still matches on the exact triplet`);
|
||||
});
|
||||
|
||||
test(`found in execution: ${rel} says a finding is matched to its row, the triplet plus the normalized title`, () => {
|
||||
const text = part();
|
||||
assert.ok(text.includes('put back at its own severity'), `${rel}: the put-back sentence is missing: ${text}`);
|
||||
assert.ok(text.includes(`each finding is matched to its own row, ${ROW_IS}`), `${rel}: the put-back does not say the row: ${text}`);
|
||||
});
|
||||
|
||||
test(`found in execution: ${rel} puts back one copy per row, the row's engine finding with the highest severity`, () => {
|
||||
const text = part();
|
||||
assert.ok(text.includes('put back at its own severity'), `${rel}: the put-back sentence is missing: ${text}`);
|
||||
assert.ok(text.includes(PUT_BACK_IS), `${rel}: the put-back does not say which one copy of a row is put back: ${text}`);
|
||||
});
|
||||
|
||||
test(`found in execution: ${rel} says floor (b)'s counterpart is the finding's own row, else the strictest row on its triplet`, () => {
|
||||
const text = part();
|
||||
assert.ok(text.includes('(floor (b))'), `${rel}: floor (b) is missing: ${text}`);
|
||||
assert.ok(text.includes(COUNTERPART_IS), `${rel}: floor (b) does not say which row is the counterpart: ${text}`);
|
||||
});
|
||||
}
|
||||
|
||||
// D2: the README's conditional line says what /trekresearch says.
|
||||
const conditionalLine = () => partAt('README.md', '`* = conditional`:', () => true);
|
||||
|
||||
test('found in execution: README.md names no "leading recommendation" trigger for security-researcher', () => {
|
||||
const text = conditionalLine();
|
||||
assert.ok(text.includes('contrarian-researcher'), `the conditional line does not name contrarian-researcher: ${text}`);
|
||||
assert.doesNotMatch(text, /security-researcher when a leading recommendation forms/, 'README.md still names a trigger /trekresearch does not define');
|
||||
});
|
||||
|
||||
test('found in execution: README.md stars security-researcher nowhere, as docs-researcher and community-researcher (selected by scope alike)', () => {
|
||||
const diagram = partAt('README.md', 'RE["EXTERNAL: docs-researcher', () => true);
|
||||
assert.ok(diagram.includes('security-researcher'), `the diagram's external line does not name security-researcher: ${diagram}`);
|
||||
assert.ok(diagram.includes('security-researcher · contrarian-researcher*'), `the diagram still stars security-researcher: ${diagram}`);
|
||||
});
|
||||
|
||||
test('found in execution: README.md lists security-researcher as no conditional agent', () => {
|
||||
const text = conditionalLine();
|
||||
assert.ok(text.includes('contrarian-researcher'), `the conditional line does not name contrarian-researcher: ${text}`);
|
||||
assert.ok(!text.includes('security-researcher'), `the conditional line still lists security-researcher: ${text}`);
|
||||
});
|
||||
|
||||
test('found in execution: README.md names the deep-research case of wave 2', () => {
|
||||
const text = conditionalLine();
|
||||
assert.ok(text.includes('contrarian-researcher, when selected, runs in wave 2'), `the conditional line does not name wave 2: ${text}`);
|
||||
assert.ok(text.includes('contrarian-researcher, when selected, runs in wave 2 after wave 1 returns, with the leading answer of wave 1\'s external findings to attack (with `--engine deep-research`, after the engine returns, from the leading answer of its report), always at `effort=high`;'), `README.md does not name the deep-research case of wave 2: ${text}`);
|
||||
});
|
||||
|
||||
test('found in execution: README.md runs contrarian-researcher after wave 1 returns', () => {
|
||||
const text = conditionalLine();
|
||||
assert.ok(text.includes('contrarian-researcher, when selected, runs in wave 2'), `the conditional line does not name wave 2: ${text}`);
|
||||
assert.ok(text.includes('runs in wave 2 after wave 1 returns'), `README.md does not say "after wave 1 returns": ${text}`);
|
||||
});
|
||||
|
||||
// D3: item 3 of both 4-pass lists names the high-effort skip and points at the normalization.
|
||||
const HIGH_EFFORT_SKIP = {
|
||||
'commands/trekreview.md': /Skipped at high effort too, where an unknown `rule_key` is normalized to `PLAN_EXECUTE_DRIFT` instead \(.*High-effort.*\); the brief-conflict rule still applies in every mode\./,
|
||||
// This item 3 names no quick mode, so no "too": it points at the quick-mode line below the list.
|
||||
'agents/review-orchestrator.md': /Skipped at high effort, as in `quick` mode \(below\); at high effort an unknown `rule_key` is normalized to `PLAN_EXECUTE_DRIFT` instead \(.*High-effort.*\); the brief-conflict rule still applies in every mode\./,
|
||||
};
|
||||
|
||||
for (const rel of Object.keys(HIGH_EFFORT_SKIP)) {
|
||||
test(`found in execution: ${rel} item 3 names Pass 3's high-effort skip and the PLAN_EXECUTE_DRIFT normalization`, () => {
|
||||
const item = pass3Item(rel);
|
||||
assert.ok(item.includes(PASS3_ITEM.slice(PASS3_ITEM.indexOf(' — ') + 3)), `${rel}: item 3 is not the Pass 3 sentence: ${item}`);
|
||||
assert.match(item, HIGH_EFFORT_SKIP[rel], `${rel}: item 3 does not name the high-effort skip: ${item}`);
|
||||
});
|
||||
}
|
||||
|
||||
// D4: the siblings entry has its own group label.
|
||||
test('found in execution: CHANGELOG ### Fixed has the **Across the round** label directly above the siblings entry', () => {
|
||||
const lines = section(roundPart(), '### Fixed').split('\n');
|
||||
const at = lines.map((l, i) => (l.startsWith('- **The texts next to the fixes say the same**') ? i : -1)).filter((i) => i !== -1);
|
||||
assert.equal(at.length, 1, `### Fixed must hold the siblings entry once; found ${at.length}`);
|
||||
assert.deepEqual(lines.slice(at[0] - 2, at[0]), ['**Across the round**', ''],
|
||||
`the line pair above the siblings entry is not the **Across the round** label: ${JSON.stringify(lines.slice(at[0] - 2, at[0]))}`);
|
||||
});
|
||||
|
||||
// The skip sentences outside the 4-pass lists keep the brief-conflict rule,
|
||||
// as agents/review-coordinator.md does ("still applies under high-effort mode",
|
||||
// "still applies in quick mode").
|
||||
const BRIEF_CONFLICT_KEPT = ', except the brief-conflict rule, which applies in every mode.';
|
||||
|
||||
test('found in execution: commands/trekreview.md high-effort section skips Pass 3 except the brief-conflict rule', () => {
|
||||
const text = flat(section(read('commands/trekreview.md'), '### High-effort behavior (v5.1.1)'));
|
||||
assert.ok(text.includes('skip Pass 3 (Cloudflare reasonableness filter)'), `the high-effort section does not skip Pass 3: ${text}`);
|
||||
assert.ok(text.includes(`skip Pass 3 (Cloudflare reasonableness filter) in \`agents/review-coordinator.md\`${BRIEF_CONFLICT_KEPT}`),
|
||||
`the high-effort skip keeps no brief-conflict rule: ${text}`);
|
||||
});
|
||||
|
||||
test('found in execution: agents/review-orchestrator.md quick mode skips pass 3 except the brief-conflict rule', () => {
|
||||
const text = partAt('agents/review-orchestrator.md', 'In `quick` mode, skip pass 3', toBlank);
|
||||
assert.ok(text.includes('skip pass 3 (reasonableness filter)'), `the quick-mode line does not skip pass 3: ${text}`);
|
||||
assert.ok(text.includes(`In \`quick\` mode, skip pass 3 (reasonableness filter)${BRIEF_CONFLICT_KEPT}`),
|
||||
`the quick-mode skip keeps no brief-conflict rule: ${text}`);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue