fix(verification): a section the runner cannot read SAYS so, and trekplan is pinned to the format it reads
Measured 2026-09-18: nine ordinary shapes of a plan's `## Verification` section parsed to zero criteria - an untagged fence, a ```text fence, a markdown table, `## Verification (acceptance)`, `## Verification:`, `### Verification`, an unclosed fence earlier in the document. Every one came out as `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed` without anyone being told that the FORMAT, not the code, was the problem. "The section is empty" and "I cannot read this format" are different facts. Three changes, one hole: - The runner reports `NO_CRITERIA` with a source line (`plan.md:NN`) when the section is there and nothing in it parsed, and names the two forms it does read. Same for a brief's `## Success Criteria`, so the evidence block the conformance reviewer gets says which of the two it is looking at rather than showing an empty table. - Phase 7 says it out loud instead of failing silently: report the source line and the two forms, and say that the plan is what failed there, not the run. - `/trekplan` now pins what it produces to what the runner reads: the heading is exactly `## Verification`, the criteria are a bullet whose first backticked span is the command or a shell-tagged fence, and the command must be one the allowlist runs. A doc-consistency test holds the writer and the reader together, so a runner that learns a new form must update the source. Honest about the round trip: the two round-trip tests were GREEN on arrival - the template already writes the bullet form the runner reads. What was missing was not the format but the PIN: `/trekplan` mandated neither the heading string nor the format, so a plan could satisfy the command's own instructions and still parse to nothing. The tests now hold that. Red first: 5 of the 7 new tests failed before the change (3 NO_CRITERIA, 2 doc-consistency); the 2 round-trip tests are guards, and said so above. Suite 1161 (1159/0/2). Gate unchanged: defects 0 of 7, intact, exit 1. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
83d82f121d
commit
f90e1cf02d
5 changed files with 195 additions and 9 deletions
|
|
@ -1310,6 +1310,18 @@ A plan with no `## Verification` section exits 1 with
|
|||
`error.code = NO_VERIFICATION_SECTION`. That is deliberate: a plan that
|
||||
promises no end-to-end check cannot be reported as verified.
|
||||
|
||||
A plan whose `## Verification` section IS there but declares nothing the runner
|
||||
can read exits 1 with `error.code = NO_CRITERIA` and a **source line**
|
||||
(`{plan_path}:NN`). Say that in the report in plain words — "the plan's
|
||||
`## Verification` at {plan_path}:NN declares no criterion the runner reads; the
|
||||
plan is what failed here, not the run" — and name the two forms it does read
|
||||
(a bullet whose first backticked span is the command; a shell-tagged fence).
|
||||
Measured 2026-09-18: nine ordinary shapes of that section — an untagged fence,
|
||||
a ```text fence, a table, `## Verification (acceptance)`, `### Verification` —
|
||||
all parsed to zero criteria, and Phase 7 then forbade `result: completed`
|
||||
without anyone being told that the format, not the code, was the problem.
|
||||
Silence is what made an unreadable section indistinguishable from a broken run.
|
||||
|
||||
Record in the progress file (additive-optional; unknown keys are tolerated by
|
||||
`progress-validator.mjs`, so a legacy progress file still validates):
|
||||
|
||||
|
|
|
|||
|
|
@ -616,6 +616,28 @@ Write the plan following the template structure. The plan MUST include:
|
|||
What tests to write and which patterns to follow.
|
||||
8. **Verification** — Reuse the brief's **Success Criteria** as the baseline.
|
||||
Each criterion must be an executable command or observable condition.
|
||||
The heading is **exactly `## Verification`** and the criteria are written in
|
||||
one of two forms. This is not style: `lib/verification/criteria-runner.mjs`
|
||||
is what runs this section in trekexecute Phase 7, and it reads those two
|
||||
forms and nothing else — `## Verification (acceptance)`, `### Verification`,
|
||||
an untagged fence or a table all parse to zero criteria, which fells the
|
||||
run without verifying anything.
|
||||
- a bullet whose FIRST backticked span is the command:
|
||||
```markdown
|
||||
- [ ] `npm test` -> expected: exit 0
|
||||
```
|
||||
- or a shell-tagged fence, one command per line:
|
||||
```bash
|
||||
npm test
|
||||
node --test tests/lib/x.test.mjs
|
||||
```
|
||||
The command must also be one the criteria runner is allowed to run: a test
|
||||
runner (`npm test`, `npm run <script>`, `node --test <file>`, `vitest`,
|
||||
`jest`, `pytest`, `python -m pytest`, `uv run pytest`, `cargo test`,
|
||||
`go test`, `make test`), `bash <script under tests/>`, or a read-only `git`
|
||||
subcommand — with no pipe, redirect or `&&`. Anything else is reported NOT
|
||||
RUN, never executed and never failed. A check that needs more than that is
|
||||
wrapped in a script under `tests/` and declared as `bash tests/<script>.sh`.
|
||||
9. **Estimated Scope** — File counts and complexity rating.
|
||||
|
||||
### Quality standards
|
||||
|
|
|
|||
|
|
@ -83,8 +83,10 @@ function scanLines(markdown) {
|
|||
return out;
|
||||
}
|
||||
|
||||
// The lines from `heading` up to the next `## ` heading (exclusive). Headings
|
||||
// inside a fence do not count on either end.
|
||||
// The lines from `heading` up to the next `## ` heading (exclusive), plus the
|
||||
// 1-based line the heading itself is on — a section the runner cannot read has
|
||||
// to be reported with a place to look. Headings inside a fence do not count on
|
||||
// either end.
|
||||
function sectionLines(markdown, heading) {
|
||||
const lines = scanLines(markdown);
|
||||
const start = lines.findIndex((l) => !l.fenced && l.text.trim() === heading);
|
||||
|
|
@ -93,7 +95,7 @@ function sectionLines(markdown, heading) {
|
|||
for (let i = start + 1; i < lines.length; i++) {
|
||||
if (!lines[i].fenced && lines[i].text.startsWith('## ')) { end = i; break; }
|
||||
}
|
||||
return lines.slice(start + 1, end);
|
||||
return { line: start + 1, lines: lines.slice(start + 1, end) };
|
||||
}
|
||||
|
||||
// `- [ ] text`, `- [x] text`, `- text`, `* text`, `1. text` -> text
|
||||
|
|
@ -154,8 +156,9 @@ function fencedCommand(text) {
|
|||
}
|
||||
|
||||
function parseSection(markdown, heading, prefix) {
|
||||
const lines = sectionLines(markdown, heading);
|
||||
if (lines === null) return [];
|
||||
const section = sectionLines(markdown, heading);
|
||||
if (section === null) return [];
|
||||
const lines = section.lines;
|
||||
const criteria = [];
|
||||
const push = (text, command, reason) =>
|
||||
criteria.push({ label: `${prefix}${criteria.length + 1}`, text, command, reason });
|
||||
|
|
@ -408,6 +411,27 @@ function report(kind, source, criteria, opts, error) {
|
|||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* The section is THERE and nothing in it parsed as a check.
|
||||
*
|
||||
* "The section is empty" and "I cannot read this format" are different facts,
|
||||
* and reporting both as `0 of 0` collapsed them into one bare exit 1. Measured
|
||||
* 2026-09-18: nine ordinary shapes of a `## Verification` section — an untagged
|
||||
* fence, a ```text fence, a table, a heading with a suffix, an unclosed fence
|
||||
* earlier in the file — all came out that way, and the run was felled without
|
||||
* anyone being told which format the runner had failed to read.
|
||||
*/
|
||||
function noCriteria(source, heading, line) {
|
||||
return {
|
||||
code: 'NO_CRITERIA',
|
||||
message:
|
||||
`${source}:${line} — the "${heading}" section declares nothing the runner can read. `
|
||||
+ 'It reads two forms: a bullet whose first backticked span is the command '
|
||||
+ '(`- [ ] `npm test` -> expected: exit 0`), and a shell-tagged fence (```bash) '
|
||||
+ 'whose lines are the commands. An untagged fence, a table or prose parses to nothing.',
|
||||
};
|
||||
}
|
||||
|
||||
function readOrThrow(path) {
|
||||
try {
|
||||
return readFileSync(path, 'utf8');
|
||||
|
|
@ -425,13 +449,18 @@ function readOrThrow(path) {
|
|||
*/
|
||||
export function runPlanVerification(planPath, opts = {}) {
|
||||
const md = readOrThrow(planPath);
|
||||
if (sectionLines(md, PLAN_HEADING) === null) {
|
||||
const section = sectionLines(md, PLAN_HEADING);
|
||||
if (section === null) {
|
||||
return report('plan', planPath, [], opts, {
|
||||
code: 'NO_VERIFICATION_SECTION',
|
||||
message: `${planPath} has no "${PLAN_HEADING}" section — nothing to verify, so the run cannot be called verified`,
|
||||
});
|
||||
}
|
||||
return report('plan', planPath, parsePlanVerification(md), opts);
|
||||
const criteria = parsePlanVerification(md);
|
||||
if (criteria.length === 0) {
|
||||
return report('plan', planPath, [], opts, noCriteria(planPath, PLAN_HEADING, section.line));
|
||||
}
|
||||
return report('plan', planPath, criteria, opts);
|
||||
}
|
||||
|
||||
/**
|
||||
|
|
@ -440,13 +469,18 @@ export function runPlanVerification(planPath, opts = {}) {
|
|||
*/
|
||||
export function runSuccessCriteriaChecks(briefPath, opts = {}) {
|
||||
const md = readOrThrow(briefPath);
|
||||
if (sectionLines(md, BRIEF_HEADING) === null) {
|
||||
const section = sectionLines(md, BRIEF_HEADING);
|
||||
if (section === null) {
|
||||
return report('brief', briefPath, [], opts, {
|
||||
code: 'NO_SUCCESS_CRITERIA_SECTION',
|
||||
message: `${briefPath} has no "${BRIEF_HEADING}" section`,
|
||||
});
|
||||
}
|
||||
return report('brief', briefPath, parseSuccessCriteria(md), opts);
|
||||
const criteria = parseSuccessCriteria(md);
|
||||
if (criteria.length === 0) {
|
||||
return report('brief', briefPath, [], opts, noCriteria(briefPath, BRIEF_HEADING, section.line));
|
||||
}
|
||||
return report('brief', briefPath, criteria, opts);
|
||||
}
|
||||
|
||||
// --- rendering --------------------------------------------------------------
|
||||
|
|
|
|||
|
|
@ -740,3 +740,84 @@ test('formatCriteriaEvidence: a criterion outside the allowlist is NOT RUN, neve
|
|||
'the reviewer must be told in-band that an unrun criterion is an absent measurement',
|
||||
);
|
||||
});
|
||||
|
||||
// --- a section it cannot read must SAY so ----------------------------------
|
||||
//
|
||||
// Measured 2026-09-18: nine perfectly ordinary `## Verification` shapes parsed
|
||||
// to zero criteria — an untagged fence, a ```text fence, a markdown table,
|
||||
// `## Verification (acceptance)`, `## Verification:`, `### Verification`, an
|
||||
// unclosed fence earlier in the document. Every one of them came out as
|
||||
// `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed`
|
||||
// without ever saying that the runner had not understood the section. "The
|
||||
// section is empty" and "I cannot read this format" are different facts and
|
||||
// the runner must not collapse them into one exit code.
|
||||
|
||||
test('runPlanVerification: a ## Verification section with nothing runnable in it is NO_CRITERIA, with the source line', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
|
||||
const p = join(dir, 'plan.md');
|
||||
writeFileSync(p, [
|
||||
'# Plan',
|
||||
'',
|
||||
'## Verification',
|
||||
'',
|
||||
'```',
|
||||
'npm test',
|
||||
'```',
|
||||
'',
|
||||
'## Estimated Scope',
|
||||
'',
|
||||
].join('\n'));
|
||||
const report = runPlanVerification(p, { cwd: ROOT });
|
||||
assert.equal(report.summary.ok, false);
|
||||
assert.equal(report.error.code, 'NO_CRITERIA');
|
||||
assert.match(report.error.message, /plan\.md:3/, 'the error must cite the line the heading is on');
|
||||
assert.notEqual(report.error.code, 'NO_VERIFICATION_SECTION', 'the section IS there — that is the point');
|
||||
});
|
||||
|
||||
test('runSuccessCriteriaChecks: an empty Success Criteria section is NO_CRITERIA too', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
|
||||
const p = join(dir, 'brief.md');
|
||||
writeFileSync(p, '# Brief\n\n## Success Criteria\n\n(to be written)\n\n## Non-Goals\n\n- none\n');
|
||||
const report = runSuccessCriteriaChecks(p, { cwd: ROOT });
|
||||
assert.equal(report.error.code, 'NO_CRITERIA');
|
||||
assert.match(report.error.message, /brief\.md:3/);
|
||||
assert.match(formatCriteriaEvidence(report), /NO_CRITERIA/, 'the reviewer must be told which of the two it is');
|
||||
});
|
||||
|
||||
test('render: NO_CRITERIA is named in the report, not left as a bare exit code', () => {
|
||||
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
|
||||
const p = join(dir, 'plan.md');
|
||||
writeFileSync(p, '# Plan\n\n## Verification\n\n| check | expected |\n|---|---|\n| npm test | exit 0 |\n');
|
||||
const text = render(runPlanVerification(p, { cwd: ROOT }));
|
||||
assert.match(text, /NO_CRITERIA/);
|
||||
});
|
||||
|
||||
// --- the round trip: what /trekplan PRODUCES, the runner READS --------------
|
||||
//
|
||||
// The runner and the plan template are two halves of one contract, and nothing
|
||||
// tied them together: `/trekplan` pinned neither the heading string nor the
|
||||
// format inside it, so a plan could be perfectly correct by the command's own
|
||||
// instructions and still parse to zero criteria. This reads the template the
|
||||
// command tells the planner to follow — it does not restate it.
|
||||
|
||||
const PLAN_TEMPLATE = join(ROOT, 'templates', 'plan-template.md');
|
||||
|
||||
test('round trip: the ## Verification section the plan template PRODUCES parses to at least one criterion', () => {
|
||||
const template = readFileSync(PLAN_TEMPLATE, 'utf8');
|
||||
const criteria = parsePlanVerification(template);
|
||||
assert.ok(criteria.length >= 1, `the template's own Verification section parses to ${criteria.length} criteria`);
|
||||
});
|
||||
|
||||
test('round trip: the template with its placeholders filled in yields RUNNABLE criteria', () => {
|
||||
const filled = readFileSync(PLAN_TEMPLATE, 'utf8')
|
||||
.replace(/`\{exact command\}`/g, '`bash tests/fixtures/criteria-exit-0.sh`')
|
||||
.replace(/`\{exact output or behavior\}`/g, '`exit 0`');
|
||||
const criteria = parsePlanVerification(filled);
|
||||
assert.ok(criteria.length >= 1, 'a filled-in template declares criteria');
|
||||
for (const c of criteria) {
|
||||
assert.equal(c.reason, '', `a filled-in criterion is still unrunnable: ${c.text} (${c.reason})`);
|
||||
assert.equal(c.command, 'bash tests/fixtures/criteria-exit-0.sh');
|
||||
}
|
||||
const results = runCriteria(criteria, { cwd: ROOT });
|
||||
assert.ok(results.every((r) => r.status === 'passed'), JSON.stringify(results.map((r) => r.status)));
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1917,3 +1917,40 @@ test('D-04: trekreview runs the success-criteria commands and hands the reviewer
|
|||
'the reviewer-launch phase must hand the block over — a block nobody passes is not evidence',
|
||||
);
|
||||
});
|
||||
|
||||
// MAJOR from the 2026-09-18 PM checkpoint: the plan's `## Verification` section
|
||||
// is the one artifact BOTH /trekplan (writer) and lib/verification/criteria-runner.mjs
|
||||
// (reader) touch, and nothing pinned them to the same format. Measured: nine
|
||||
// ordinary shapes of that section parsed to zero criteria, which fells the
|
||||
// single-session run. The writer must pin the heading string and the two forms
|
||||
// the reader reads. Fix the SOURCE: if the runner learns a new form, say so here.
|
||||
test('trekplan pins the ## Verification heading and a format the criteria runner reads', () => {
|
||||
const t = read('commands/trekplan.md');
|
||||
const section = (t.split('8. **Verification**')[1] || '').split('\n9. ')[0];
|
||||
assert.ok(section.length > 0, 'trekplan.md must still carry the Verification instruction');
|
||||
assert.match(
|
||||
section, /exactly `## Verification`/,
|
||||
'the heading string is load-bearing: `## Verification (acceptance)` and `### Verification` parse to nothing',
|
||||
);
|
||||
assert.match(
|
||||
section, /criteria runner|criteria-runner/,
|
||||
'the instruction must name the reader it is writing for',
|
||||
);
|
||||
assert.match(
|
||||
section, /- \[ \] `/,
|
||||
'the bullet form the runner reads must be shown, not described',
|
||||
);
|
||||
assert.match(section, /```bash/, 'the shell-tagged fence form must be shown too');
|
||||
});
|
||||
|
||||
test('trekexecute Phase 7 says NO_CRITERIA out loud instead of failing silently', () => {
|
||||
const phase7 = (read('commands/trekexecute.md').split('\n## Phase 7 — ')[1] || '').split('\n## ')[0];
|
||||
assert.match(
|
||||
phase7, /NO_CRITERIA/,
|
||||
'a section the runner could not read must be reported as such, not as a bare exit 1',
|
||||
);
|
||||
assert.match(
|
||||
phase7, /source line|`\{plan_path\}:/,
|
||||
'the report must carry the line the unreadable section starts on',
|
||||
);
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue