fix(verification): a section the runner cannot read SAYS so, and trekplan is pinned to the format it reads

Measured 2026-09-18: nine ordinary shapes of a plan's `## Verification`
section parsed to zero criteria - an untagged fence, a ```text fence, a
markdown table, `## Verification (acceptance)`, `## Verification:`,
`### Verification`, an unclosed fence earlier in the document. Every one came
out as `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed`
without anyone being told that the FORMAT, not the code, was the problem.
"The section is empty" and "I cannot read this format" are different facts.

Three changes, one hole:

- The runner reports `NO_CRITERIA` with a source line (`plan.md:NN`) when the
  section is there and nothing in it parsed, and names the two forms it does
  read. Same for a brief's `## Success Criteria`, so the evidence block the
  conformance reviewer gets says which of the two it is looking at rather than
  showing an empty table.
- Phase 7 says it out loud instead of failing silently: report the source line
  and the two forms, and say that the plan is what failed there, not the run.
- `/trekplan` now pins what it produces to what the runner reads: the heading
  is exactly `## Verification`, the criteria are a bullet whose first
  backticked span is the command or a shell-tagged fence, and the command must
  be one the allowlist runs. A doc-consistency test holds the writer and the
  reader together, so a runner that learns a new form must update the source.

Honest about the round trip: the two round-trip tests were GREEN on arrival -
the template already writes the bullet form the runner reads. What was missing
was not the format but the PIN: `/trekplan` mandated neither the heading string
nor the format, so a plan could satisfy the command's own instructions and
still parse to nothing. The tests now hold that.

Red first: 5 of the 7 new tests failed before the change (3 NO_CRITERIA, 2
doc-consistency); the 2 round-trip tests are guards, and said so above.
Suite 1161 (1159/0/2). Gate unchanged: defects 0 of 7, intact, exit 1.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 03:00:13 +02:00
commit f90e1cf02d
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 195 additions and 9 deletions

View file

@ -740,3 +740,84 @@ test('formatCriteriaEvidence: a criterion outside the allowlist is NOT RUN, neve
'the reviewer must be told in-band that an unrun criterion is an absent measurement',
);
});
// --- a section it cannot read must SAY so ----------------------------------
//
// Measured 2026-09-18: nine perfectly ordinary `## Verification` shapes parsed
// to zero criteria — an untagged fence, a ```text fence, a markdown table,
// `## Verification (acceptance)`, `## Verification:`, `### Verification`, an
// unclosed fence earlier in the document. Every one of them came out as
// `0 of 0`, NOT OK, exit 1, and Phase 7 then forbade `result: completed`
// without ever saying that the runner had not understood the section. "The
// section is empty" and "I cannot read this format" are different facts and
// the runner must not collapse them into one exit code.
test('runPlanVerification: a ## Verification section with nothing runnable in it is NO_CRITERIA, with the source line', () => {
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
const p = join(dir, 'plan.md');
writeFileSync(p, [
'# Plan',
'',
'## Verification',
'',
'```',
'npm test',
'```',
'',
'## Estimated Scope',
'',
].join('\n'));
const report = runPlanVerification(p, { cwd: ROOT });
assert.equal(report.summary.ok, false);
assert.equal(report.error.code, 'NO_CRITERIA');
assert.match(report.error.message, /plan\.md:3/, 'the error must cite the line the heading is on');
assert.notEqual(report.error.code, 'NO_VERIFICATION_SECTION', 'the section IS there — that is the point');
});
test('runSuccessCriteriaChecks: an empty Success Criteria section is NO_CRITERIA too', () => {
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
const p = join(dir, 'brief.md');
writeFileSync(p, '# Brief\n\n## Success Criteria\n\n(to be written)\n\n## Non-Goals\n\n- none\n');
const report = runSuccessCriteriaChecks(p, { cwd: ROOT });
assert.equal(report.error.code, 'NO_CRITERIA');
assert.match(report.error.message, /brief\.md:3/);
assert.match(formatCriteriaEvidence(report), /NO_CRITERIA/, 'the reviewer must be told which of the two it is');
});
test('render: NO_CRITERIA is named in the report, not left as a bare exit code', () => {
const dir = mkdtempSync(join(tmpdir(), 'criteria-runner-'));
const p = join(dir, 'plan.md');
writeFileSync(p, '# Plan\n\n## Verification\n\n| check | expected |\n|---|---|\n| npm test | exit 0 |\n');
const text = render(runPlanVerification(p, { cwd: ROOT }));
assert.match(text, /NO_CRITERIA/);
});
// --- the round trip: what /trekplan PRODUCES, the runner READS --------------
//
// The runner and the plan template are two halves of one contract, and nothing
// tied them together: `/trekplan` pinned neither the heading string nor the
// format inside it, so a plan could be perfectly correct by the command's own
// instructions and still parse to zero criteria. This reads the template the
// command tells the planner to follow — it does not restate it.
const PLAN_TEMPLATE = join(ROOT, 'templates', 'plan-template.md');
test('round trip: the ## Verification section the plan template PRODUCES parses to at least one criterion', () => {
const template = readFileSync(PLAN_TEMPLATE, 'utf8');
const criteria = parsePlanVerification(template);
assert.ok(criteria.length >= 1, `the template's own Verification section parses to ${criteria.length} criteria`);
});
test('round trip: the template with its placeholders filled in yields RUNNABLE criteria', () => {
const filled = readFileSync(PLAN_TEMPLATE, 'utf8')
.replace(/`\{exact command\}`/g, '`bash tests/fixtures/criteria-exit-0.sh`')
.replace(/`\{exact output or behavior\}`/g, '`exit 0`');
const criteria = parsePlanVerification(filled);
assert.ok(criteria.length >= 1, 'a filled-in template declares criteria');
for (const c of criteria) {
assert.equal(c.reason, '', `a filled-in criterion is still unrunnable: ${c.text} (${c.reason})`);
assert.equal(c.command, 'bash tests/fixtures/criteria-exit-0.sh');
}
const results = runCriteria(criteria, { cwd: ROOT });
assert.ok(results.every((r) => r.status === 'passed'), JSON.stringify(results.map((r) => r.status)));
});

View file

@ -1917,3 +1917,40 @@ test('D-04: trekreview runs the success-criteria commands and hands the reviewer
'the reviewer-launch phase must hand the block over — a block nobody passes is not evidence',
);
});
// MAJOR from the 2026-09-18 PM checkpoint: the plan's `## Verification` section
// is the one artifact BOTH /trekplan (writer) and lib/verification/criteria-runner.mjs
// (reader) touch, and nothing pinned them to the same format. Measured: nine
// ordinary shapes of that section parsed to zero criteria, which fells the
// single-session run. The writer must pin the heading string and the two forms
// the reader reads. Fix the SOURCE: if the runner learns a new form, say so here.
test('trekplan pins the ## Verification heading and a format the criteria runner reads', () => {
const t = read('commands/trekplan.md');
const section = (t.split('8. **Verification**')[1] || '').split('\n9. ')[0];
assert.ok(section.length > 0, 'trekplan.md must still carry the Verification instruction');
assert.match(
section, /exactly `## Verification`/,
'the heading string is load-bearing: `## Verification (acceptance)` and `### Verification` parse to nothing',
);
assert.match(
section, /criteria runner|criteria-runner/,
'the instruction must name the reader it is writing for',
);
assert.match(
section, /- \[ \] `/,
'the bullet form the runner reads must be shown, not described',
);
assert.match(section, /```bash/, 'the shell-tagged fence form must be shown too');
});
test('trekexecute Phase 7 says NO_CRITERIA out loud instead of failing silently', () => {
const phase7 = (read('commands/trekexecute.md').split('\n## Phase 7 — ')[1] || '').split('\n## ')[0];
assert.match(
phase7, /NO_CRITERIA/,
'a section the runner could not read must be reported as such, not as a bare exit 1',
);
assert.match(
phase7, /source line|`\{plan_path\}:/,
'the report must carry the line the unreadable section starts on',
);
});