feat(skills): writing-criteria, one source for runnable criteria (order 7c)

A brief success criterion and a plan's Verify, Regression, End-to-end and
Verification lines are each one command the criteria runner accepts, written
as the line's first backticked span; anything more is bash tests/<name>.sh,
and each must be able to go red (the runner reads the exit code only).

- skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red:
  examples that a pin runs through the criteria runner
- brief-reviewer and plan-critic preload writing-criteria and
  reviewing-adversarially (S3 + S4); the copied runner rules leave them, the
  two templates and /trekplan, which names the skill where the plan is written
  and where its three reviewers start
- eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins
- review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref
  re-measured

Baseline without the skill: 24 of 27 written checks refused by the runner,
and 0 of 3 reviewers saw that git status/git log can never go red. With it:
0 of 27 refused, 3 of 3 reviewers flagged both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

evals/skill-writing-criteria-effect/case.yaml

@@ -0,0 +1,4 @@
+schema_version: "1.1"
+name: skill-writing-criteria-effect
+context:
+  scaffold_script: scaffold.sh

evals/skill-writing-criteria-effect/graders/all-runnable.md

@@ -0,0 +1,6 @@
+---
+type: llm
+focus: last_message
+---
+
+Look at every success criterion and every Verify and End-to-end line in the reply. PASS only if, for each one, the first backticked span on the line is the command (not the name of a feature such as `shelf count`), and that command is one plain invocation of `npm test`, `npm run <script>`, `node --test <test files>` or `bash tests/<name>.sh` (or `/bin/bash tests/<name>.sh`): no `&&`, `;`, `|`, `>`, `<` or `$(`, no quotes, no environment prefix such as `SHELF_FILE=... `, no `node -e`, and not the product's own binary (`node bin/shelf.mjs ...` or `shelf ...`). Also FAIL if a check can never fail whatever the code does (for example `git status`, `git log`, or a test file that already passes before the change). FAIL otherwise, and name the first line that breaks the rule.

evals/skill-writing-criteria-effect/graders/no-and-chain.md

@@ -0,0 +1,5 @@
+---
+type: regex
+pattern: '&&'
+match: not_contains
+---

evals/skill-writing-criteria-effect/prompt.md

@@ -0,0 +1,34 @@
+---
+description: "Effect case for writing-criteria (two arms: with and without the plugin). The scenario where Voyage's baseline helpers failed without the skill: 24 of 27 criteria and check lines were refused by the criteria runner (&& chains, $(...), redirects, node -e, the product's own binary, the feature named in the first backticked span). The operator grants Read, Glob and Grep, and sets the judge model explicitly (never Haiku)."
+expected_outcome: "Every success criterion and every Verify and End-to-end line is one command the criteria runner accepts, written as the first backticked span, and each can go red."
+tags: [skill-effect, skill=writing-criteria, from=baseline-wc-chained-verify]
+runs: 3
+max_turns: 20
+timeout_seconds: 600
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+You are writing the acceptance checks for a small change to the Node project in the current directory (a home-library CLI: `bin/shelf.mjs`, `src/books.mjs`, `tests/books.test.mjs`, `package.json`). Work fast: the maintainer wants the brief and plan checks in ten minutes so the pipeline can start. Do not change any file.
+
+The change has three parts:
+1. `shelf list --json` prints the whole catalogue as a JSON array (title, author, added).
+2. `shelf count` prints the number of books; `shelf count --author <name>` prints how many are by that author.
+3. `shelf add` refuses a title that is already on the shelf (case-insensitive): exit code 1 and the message `already on the shelf`.
+
+Reply with two things and nothing else.
+
+A. A brief section with 4 to 6 success criteria, in exactly this form:
+
+## Success Criteria
+
+- <what must hold>: `<command>` exits 0
+
+B. Three plan steps (one per part above), each with the step's test and its verify line, plus one end-to-end check on the last step, in exactly this form:
+
+### Step 1: <title>
+- **Test first:** <test file> — <what it checks>
+- **Verify:** `<command>` → expected: <result>
+
+(Step 3 also gets: `- **End-to-end:** \`<command>\` → expected: <what a user sees>`)
+
+Every backticked command will be run automatically by a checker in the project root (as a command, its exit code decides pass or fail), so each must be a real command that checks the thing.

evals/skill-writing-criteria-effect/scaffold.sh

@@ -0,0 +1,60 @@
+#!/bin/bash
+# Seeds the empty run workspace with a small home-library CLI (Node, node --test).
+# Self-contained on purpose: the run cannot read the eval directory, and the
+# fixture is written inline, with no path outside the workspace and no git.
+set -eu
+mkdir -p src bin tests
+cat > package.json <<'J'
+{ "name": "shelf", "version": "0.2.0", "type": "module", "private": true,
+  "bin": { "shelf": "bin/shelf.mjs" },
+  "scripts": { "test": "node --test", "lint": "node --check src/books.mjs" } }
+J
+cat > src/books.mjs <<'J'
+// A tiny home-library catalogue kept in a JSON file.
+import { readFileSync, writeFileSync, existsSync } from 'node:fs';
+
+export function load(path) {
+  return existsSync(path) ? JSON.parse(readFileSync(path, 'utf8')) : [];
+}
+
+export function save(path, books) {
+  writeFileSync(path, JSON.stringify(books, null, 2) + '\n');
+}
+
+export function add(books, { title, author }) {
+  if (!title) throw new Error('title is required');
+  return [...books, { title, author: author ?? 'unknown', added: new Date().toISOString().slice(0, 10) }];
+}
+J
+cat > bin/shelf.mjs <<'J'
+#!/usr/bin/env node
+// shelf add "<title>" [--author <name>] | shelf list
+import { load, save, add } from '../src/books.mjs';
+
+const file = process.env.SHELF_FILE ?? 'shelf.json';
+const [cmd, ...rest] = process.argv.slice(2);
+
+if (cmd === 'add') {
+  const i = rest.indexOf('--author');
+  const author = i >= 0 ? rest[i + 1] : undefined;
+  save(file, add(load(file), { title: rest[0], author }));
+} else if (cmd === 'list') {
+  for (const b of load(file)) console.log(`${b.title} — ${b.author}`);
+} else {
+  console.error('usage: shelf add "<title>" [--author <name>] | shelf list');
+  process.exit(2);
+}
+J
+cat > tests/books.test.mjs <<'J'
+import { test } from 'node:test';
+import assert from 'node:assert/strict';
+import { add } from '../src/books.mjs';
+
+test('add appends a book', () => {
+  const out = add([], { title: 'Dune', author: 'Herbert' });
+  assert.equal(out.length, 1);
+  assert.equal(out[0].author, 'Herbert');
+});
+
+test('add needs a title', () => assert.throws(() => add([], {}), /title is required/));
+J

evals/skill-writing-criteria-fires-1/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=skillsanalyse-4.4]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Write the success criteria for this brief so that the review can run each criterion as a command and see it pass or fail.

evals/skill-writing-criteria-fires-2/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=baseline-wc-chained-verify]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+The plan validator says PLAN_STEP_VERIFY_NOT_RUNNABLE on step 2: its Verify line chains the test file, a temp folder and a CLI check with &&. Rewrite that Verify line so the criteria runner accepts it.

evals/skill-writing-criteria-fires-3/graders/skill-fired.md

@@ -0,0 +1,5 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+---

evals/skill-writing-criteria-fires-3/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill should fire on this request (the reserve path outside the pipeline)."
+expected_outcome: "Claude calls the Skill tool for voyage:writing-criteria at least once."
+tags: [skill-trigger, skill=writing-criteria, from=baseline-crit-never-red]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+The plan's Verification section lists `git status --porcelain` with "expected: no output". Is that a criterion that can go red, and what should the Verification line be instead?

evals/skill-writing-criteria-quiet-1/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-writing-criteria-quiet-1/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill must stay quiet here; test-first owns this request."
+expected_outcome: "Claude never calls the Skill tool for voyage:writing-criteria."
+tags: [skill-no-trigger, skill=writing-criteria, owner=test-first, from=order-7c-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Write a failing test for the new duplicate-title rule first, see it fail, then write the code that makes it pass.

evals/skill-writing-criteria-quiet-2/graders/skill-quiet.md

@@ -0,0 +1,8 @@
+---
+type: tool_used
+tool: Skill
+input_match: '"skill"\s*:\s*"(?:voyage:)?writing-criteria"'
+min: 0
+max: 0
+arm: both
+---

evals/skill-writing-criteria-quiet-2/prompt.md

@@ -0,0 +1,10 @@
+---
+description: "Routing case: the writing-criteria skill must stay quiet here. Starting the app to look at a change is Claude Code's own run skill, which the model-free routing corpus (Voyage's skills, commands and agents) does not hold; there /trekexecute, the Voyage command that runs things, stands in as the owner. The model-run grader only checks that writing-criteria is not called."
+expected_outcome: "Claude never calls the Skill tool for voyage:writing-criteria."
+tags: [skill-no-trigger, skill=writing-criteria, owner=trekexecute, from=order-7c-negatives]
+runs: 3
+max_turns: 4
+allowed_tools: [Read, Glob, Grep, Skill]
+---
+
+Start the app and click through the new settings screen so I can see that it works.