feat(skills): interviewing - one source for the brief's question round (order 7e)
/trekbrief Phase 3 carried the interview form inline. It now asks for the new skill voyage:interviewing by name (allowed-tools gains Skill) and keeps only its own contract: the size class and budget, the three fixed questions (framing first, class, storage), where each answer lands in the brief, and the answers that decide nothing. Step 3a and the form points of Step 3c move into the skill with their credits (Pocock, Osmani, Spec Kit; licences and line ranges checked at the pinned commits). Red baseline first (3 helper interviews against a fixed answer sheet, Opus 5.5): 10-11 questions in round 1 (3 of 3), 15 of 31 without a recommendation, a hollow answer written in as a decision (2 of 3), unasked scope as a requirement (2 of 3). With the skill: 4, 0 of 12, 0 of 3, 0 of 3. Fact questions and stopping did not fail without it. Also: pointers to voyage:writing-criteria (S3) where criteria are asked and written, both review skill names (S3 + S4) in the brief-reviewer prompt; the intent pins that read moved text now read the skill, same assertions; the Phase 7 line_ref moves with the shorter Phase 3; 7 eval cases (3/2/2). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
parent
31c77fd8d6
commit
f6e18a1d09
25 changed files with 569 additions and 51 deletions
4
evals/skill-interviewing-effect-round/case.yaml
Normal file
4
evals/skill-interviewing-effect-round/case.yaml
Normal file
|
|
@ -0,0 +1,4 @@
|
|||
schema_version: "1.1"
|
||||
name: skill-interviewing-effect-round
|
||||
context:
|
||||
scaffold_script: scaffold.sh
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
---
|
||||
|
||||
PASS only if the reply asks the person at most four questions, and every question gives a recommended answer before or as its first option (marked "Recommended", "Suggested", "I recommend" or similar). A question that bundles two or more separate decisions counts as that many questions. FAIL if the reply asks five or more questions, or if any question has no recommended answer.
|
||||
|
|
@ -0,0 +1,6 @@
|
|||
---
|
||||
type: llm
|
||||
focus: last_message
|
||||
---
|
||||
|
||||
PASS only if the reply writes down what it did not ask: assumptions it made (marked as assumed, not decided by the person) or a deferred list of open points, or says plainly that nothing else is open. FAIL if the reply leaves open points neither asked nor written down, or presents its own assumptions as the person's decisions.
|
||||
15
evals/skill-interviewing-effect-round/prompt.md
Normal file
15
evals/skill-interviewing-effect-round/prompt.md
Normal file
|
|
@ -0,0 +1,15 @@
|
|||
---
|
||||
description: "Effect case for interviewing (two arms: with and without the plugin). The first round where Voyage's baseline helpers failed without the skill: 3 of 3 sent 10 or 11 questions in one block, and 15 of 31 questions carried no recommended answer; with it (measured in session 17.10), 3 of 3 sent 4, and 0 of 12 lacked one. None asked a fact the project could answer, so that is not graded here. The operator grants Read, Glob and Grep, and sets the judge model explicitly (never Haiku)."
|
||||
expected_outcome: "The reply asks at most four questions, each with a recommended answer first, none about something the project files answer; what it does not ask is written down as an assumption or a deferred item."
|
||||
tags: [skill-effect, skill=interviewing, from=baseline-ivw-round-size]
|
||||
runs: 3
|
||||
max_turns: 20
|
||||
timeout_seconds: 600
|
||||
allowed_tools: [Read, Glob, Grep, Skill]
|
||||
---
|
||||
|
||||
I run a small seed library for my garden club. The app is the project in the current directory.
|
||||
|
||||
Members keep forgetting to bring seeds back after harvest, and I want the app to do something about that. Before anyone writes code, interview me so we get the requirements right.
|
||||
|
||||
How this works: you cannot reach me directly. Put whatever you want to ask me in your reply, and I will send my answers back to you in a message. Do not change any files in the project.
|
||||
92
evals/skill-interviewing-effect-round/scaffold.sh
Executable file
92
evals/skill-interviewing-effect-round/scaffold.sh
Executable file
|
|
@ -0,0 +1,92 @@
|
|||
#!/bin/bash
|
||||
# Seeds the empty run workspace with a small seed-library app (Node, node --test),
|
||||
# made up from scratch. Self-contained on purpose: the run cannot read the eval
|
||||
# directory, and the fixture is written inline, with no path outside the
|
||||
# workspace and no git. The same project the red baseline ran against.
|
||||
set -eu
|
||||
mkdir -p src tests data
|
||||
cat > package.json <<'J'
|
||||
{ "name": "seedswap", "version": "0.4.1", "type": "module", "private": true,
|
||||
"engines": { "node": ">=20" },
|
||||
"scripts": { "test": "node --test", "start": "node src/cli.mjs" } }
|
||||
J
|
||||
cat > README.md <<'J'
|
||||
# seedswap
|
||||
|
||||
A seed library for a garden club. Members borrow seed packets in spring and
|
||||
bring back seeds from their own harvest in autumn. One volunteer runs this
|
||||
command-line tool on the club laptop.
|
||||
|
||||
node src/cli.mjs borrow <member> <variety>
|
||||
node src/cli.mjs return <member> <variety>
|
||||
node src/cli.mjs list
|
||||
|
||||
Data lives in `data/*.json`. Messages to members go to `data/outbox.txt`,
|
||||
which the volunteer prints and hands out at the monthly meeting.
|
||||
J
|
||||
cat > src/store.mjs <<'J'
|
||||
import { readFileSync, writeFileSync } from 'node:fs';
|
||||
const path = (name) => new URL(`../data/${name}.json`, import.meta.url);
|
||||
export const load = (name) => JSON.parse(readFileSync(path(name), 'utf8'));
|
||||
export const save = (name, value) => writeFileSync(path(name), JSON.stringify(value, null, 2) + '\n');
|
||||
J
|
||||
cat > src/members.mjs <<'J'
|
||||
import { load } from './store.mjs';
|
||||
// A member: { id, name, joined }. No email or phone is stored.
|
||||
export const members = () => load('members');
|
||||
export const byId = (id) => members().find((m) => m.id === id);
|
||||
J
|
||||
cat > src/loans.mjs <<'J'
|
||||
import { load, save } from './store.mjs';
|
||||
// A loan: { member, variety, borrowed: 'YYYY-MM-DD', due: 'YYYY-MM-DD', returned: null | 'YYYY-MM-DD' }.
|
||||
// `due` is set to 15 October of the borrowing year (end of harvest).
|
||||
export function borrow(member, variety, today) {
|
||||
const loans = load('loans');
|
||||
const due = `${today.slice(0, 4)}-10-15`;
|
||||
loans.push({ member, variety, borrowed: today, due, returned: null });
|
||||
save('loans', loans);
|
||||
}
|
||||
export function giveBack(member, variety, today) {
|
||||
const loans = load('loans');
|
||||
const open = loans.find((l) => l.member === member && l.variety === variety && !l.returned);
|
||||
if (!open) throw new Error(`no open loan of ${variety} for ${member}`);
|
||||
open.returned = today;
|
||||
save('loans', loans);
|
||||
}
|
||||
export const openLoans = () => load('loans').filter((l) => !l.returned);
|
||||
J
|
||||
cat > src/notify.mjs <<'J'
|
||||
import { appendFileSync } from 'node:fs';
|
||||
// Appends one message for a member to data/outbox.txt (printed at meetings).
|
||||
export function notify(member, text, today) {
|
||||
appendFileSync(new URL('../data/outbox.txt', import.meta.url), `${today}\t${member.name}\t${text}\n`);
|
||||
}
|
||||
J
|
||||
cat > src/cli.mjs <<'J'
|
||||
import { borrow, giveBack, openLoans } from './loans.mjs';
|
||||
const [cmd, member, variety] = process.argv.slice(2);
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
if (cmd === 'borrow') borrow(member, variety, today);
|
||||
else if (cmd === 'return') giveBack(member, variety, today);
|
||||
else if (cmd === 'list') for (const l of openLoans()) console.log(`${l.member}\t${l.variety}\tdue ${l.due}`);
|
||||
else { console.error('usage: borrow|return|list'); process.exit(2); }
|
||||
J
|
||||
cat > tests/loans.test.mjs <<'J'
|
||||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
test('loans module loads', async () => {
|
||||
const m = await import('../src/loans.mjs');
|
||||
assert.equal(typeof m.borrow, 'function');
|
||||
});
|
||||
J
|
||||
cat > data/members.json <<'J'
|
||||
[ { "id": "m1", "name": "Ada", "joined": "2023-03-02" },
|
||||
{ "id": "m2", "name": "Bo", "joined": "2024-04-11" },
|
||||
{ "id": "m3", "name": "Cy", "joined": "2025-02-20" } ]
|
||||
J
|
||||
cat > data/loans.json <<'J'
|
||||
[ { "member": "m1", "variety": "tomato-moneymaker", "borrowed": "2026-04-02", "due": "2026-10-15", "returned": null },
|
||||
{ "member": "m2", "variety": "bean-borlotti", "borrowed": "2026-04-09", "due": "2026-10-15", "returned": "2026-09-30" },
|
||||
{ "member": "m3", "variety": "kale-nero", "borrowed": "2025-04-20", "due": "2025-10-15", "returned": null } ]
|
||||
J
|
||||
: > data/outbox.txt
|
||||
Loading…
Add table
Add a link
Reference in a new issue