voyage/evals/skill-writing-criteria-effect/scaffold.sh
Kjell Tore Guttormsen 837e92545d
feat(skills): writing-criteria, one source for runnable criteria (order 7c)
A brief success criterion and a plan's Verify, Regression, End-to-end and
Verification lines are each one command the criteria runner accepts, written
as the line's first backticked span; anything more is bash tests/<name>.sh,
and each must be able to go red (the runner reads the exit code only).

- skills/writing-criteria/SKILL.md (S1 form), with Runs:/Refused:/Never red:
  examples that a pin runs through the criteria runner
- brief-reviewer and plan-critic preload writing-criteria and
  reviewing-adversarially (S3 + S4); the copied runner rules leave them, the
  two templates and /trekplan, which names the skill where the plan is written
  and where its three reviewers start
- eval cases 3 fire / 2 quiet / 1 effect; tests/lib/skills-criteria-prose-pins
- review-630 @~1 pin follows the text into the skill; trekplan-stats line_ref
  re-measured

Baseline without the skill: 24 of 27 written checks refused by the runner,
and 0 of 3 reviewers saw that git status/git log can never go red. With it:
0 of 27 refused, 3 of 3 reviewers flagged both.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 07:39:04 +02:00

60 lines
2.1 KiB
Bash
Executable file

#!/bin/bash
# Seeds the empty run workspace with a small home-library CLI (Node, node --test).
# Self-contained on purpose: the run cannot read the eval directory, and the
# fixture is written inline, with no path outside the workspace and no git.
set -eu
mkdir -p src bin tests
cat > package.json <<'J'
{ "name": "shelf", "version": "0.2.0", "type": "module", "private": true,
"bin": { "shelf": "bin/shelf.mjs" },
"scripts": { "test": "node --test", "lint": "node --check src/books.mjs" } }
J
cat > src/books.mjs <<'J'
// A tiny home-library catalogue kept in a JSON file.
import { readFileSync, writeFileSync, existsSync } from 'node:fs';
export function load(path) {
return existsSync(path) ? JSON.parse(readFileSync(path, 'utf8')) : [];
}
export function save(path, books) {
writeFileSync(path, JSON.stringify(books, null, 2) + '\n');
}
export function add(books, { title, author }) {
if (!title) throw new Error('title is required');
return [...books, { title, author: author ?? 'unknown', added: new Date().toISOString().slice(0, 10) }];
}
J
cat > bin/shelf.mjs <<'J'
#!/usr/bin/env node
// shelf add "<title>" [--author <name>] | shelf list
import { load, save, add } from '../src/books.mjs';
const file = process.env.SHELF_FILE ?? 'shelf.json';
const [cmd, ...rest] = process.argv.slice(2);
if (cmd === 'add') {
const i = rest.indexOf('--author');
const author = i >= 0 ? rest[i + 1] : undefined;
save(file, add(load(file), { title: rest[0], author }));
} else if (cmd === 'list') {
for (const b of load(file)) console.log(`${b.title} — ${b.author}`);
} else {
console.error('usage: shelf add "<title>" [--author <name>] | shelf list');
process.exit(2);
}
J
cat > tests/books.test.mjs <<'J'
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { add } from '../src/books.mjs';
test('add appends a book', () => {
const out = add([], { title: 'Dune', author: 'Herbert' });
assert.equal(out.length, 1);
assert.equal(out[0].author, 'Herbert');
});
test('add needs a title', () => assert.throws(() => add([], {}), /title is required/));
J