fix(end-state): the gate can no longer be turned green by editing its own ledger
An independent adversarial review of the end-state gate reproduced a GREEN result, exit 0, with the gate's 18 tests passing. Only the registry and the STATE markers were edited; every other file stayed byte-identical. The judged party owned the denominator, and nothing guarded it. This change closes that finding and three more, and reopens a defect the gate had closed too early. Registry integrity (the blocker): - tests/fixtures/end-state-frozen.json freezes the denominator. It holds the 7 defect ids and 3 experiment ids, each with a sha256 signature of its check (canonical JSON, key order irrelevant), plus the exact agents / decisions / freeze configuration. - The rule: an entry may close; it may never disappear or have its check changed. New entries are allowed. spawnSites may never include agents/. - The test file pins the manifest literally, with a comment that the denominator was frozen on 2026-09-17 and that changing it is a decision. The gate verifies the registry against the manifest on every run. - A violation makes the gate red and names what moved. A missing manifest makes integrity n/a, which is also red. Dormant means "never spawned", not "never mentioned": - An agent counts as spawned only when a command names it in a spawn instruction: a row of a table headed `Agent`, a line that starts with Launch/Spawn, or a **name** block followed by a Prompt: line. - Prose, negations, HTML comments and fenced code never count. Block quotes cannot match either, because every form is anchored at the start of the line. - Re-measured under this definition: still 1 of 20 (synthesis-agent). Per agent, every other spawnable agent has a spawn instruction in at least one command. A fifth tally, feat-after-freeze: - It counts commits after the `end-state-freeze` tag whose subject matches ^feat(\(|!|:). - The row is n/a, and therefore red, when there is no tag yet, the root is not a git work tree, or the root is not the top of its work tree. It is never 0. Honest output: - Every registry entry now declares its probe kind (phrase or byte). - Open phrase probes are labelled as such, and closed ids are listed. The output states that a closed phrase probe is evidence, not proof of behaviour. D-03 and D-04 stay open and labelled; how they are fixed is still to be decided. - The decisions row now says it counts ticks, not verified decisions. D-07 reopened: - Its check now also covers README.md and commands/trekresearch.md, where the "orchestrator runs on opus" claim still lives. README even ships a sed recipe for `model: opus` lines no command has. - Defects are 3 of 7 until the separate docs fix lands. Three surviving review mutants were killed with tests: - an empty glob over an existing dir is silent; - the header counts n/a rows as zero; - a dormant-row error reads as 0. Measured on a clean export of the index: - Mutation harness, adapted from the review's: review code mutants 18 of 18 killed. Four of them were re-targeted at the equivalent new code. M02 survived until a test with an intact registry plus a missing freeze tag was added. - New-logic mutants: 12 of 12 killed. One equivalent mutant (block-quote stripping) was removed together with the dead clause it targeted. - Registry attacks: 7 of 7 killed. - The review's combined green attack (registry + STATE only) is now RED, exit 1: "defects: D-03 is frozen but missing from the registry". - Gate tests 38/38. Suite 1059 -> 1079 (1077/0/2). - Gate: RED, defects 3 of 7, experiments 3 of 3, dormant 1 of 20; decisions and feat-after-freeze are n/a in a clean export. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e43153c3c4
commit
09feb415b3
4 changed files with 762 additions and 144 deletions
|
|
@ -1,27 +1,32 @@
|
|||
#!/usr/bin/env node
|
||||
// scripts/end-state-gate.mjs
|
||||
// End-state gate — counts the distance to "Voyage is finished" as four tallies:
|
||||
// End-state gate — counts the distance to "Voyage is finished" as five tallies:
|
||||
//
|
||||
// defects open pipeline defects (listed in the registry, each with a runnable check)
|
||||
// experiments unresolved experiments/opt-ins (listed in the registry, each with a runnable check)
|
||||
// dormant-agents spawnable agents no command references (measured structurally from agents/)
|
||||
// decisions open operator decisions (counted in STATE.md by a fixed marker)
|
||||
// defects open pipeline defects (listed in the registry, each with a runnable check)
|
||||
// experiments unresolved experiments/opt-ins (listed in the registry, each with a runnable check)
|
||||
// dormant-agents spawnable agents no command spawns (measured structurally, see spawnedNames)
|
||||
// decisions open operator decisions (counted in STATE.md by a fixed marker)
|
||||
// feat-after-freeze feat commits after the freeze tag (measured from git)
|
||||
//
|
||||
// "Finished" = all four at 0. Exit 0 = green, 1 = red (any tally > 0 or not measurable),
|
||||
// 2 = usage or registry error. The denominators come from the tracked registry and the
|
||||
// tracked agents/ directory — never from free-text greps over prose.
|
||||
// "Finished" = all five at 0 AND the registry is intact against its frozen denominator
|
||||
// (tests/fixtures/end-state-frozen.json: an entry may close, never disappear or change its
|
||||
// check). Exit 0 = green, 1 = red (a tally > 0, a tally not measurable, or the registry
|
||||
// tampered with), 2 = usage or registry error.
|
||||
//
|
||||
// A registry entry is OPEN while ALL of its conditions hold. An entry whose check is null,
|
||||
// or whose check cannot run (missing file, empty glob, missing section), is NOT FELLABLE
|
||||
// and is counted as open — a check that cannot fire must never read as "fixed".
|
||||
// or whose check cannot run (missing file, empty glob, missing section), is NOT FELLABLE and
|
||||
// is counted as open — a check that cannot fire must never read as "fixed".
|
||||
//
|
||||
// Usage: node scripts/end-state-gate.mjs [--json] [--root <dir>]
|
||||
|
||||
import { readFileSync, existsSync, readdirSync, statSync } from 'node:fs';
|
||||
import { readFileSync, existsSync, readdirSync, statSync, realpathSync } from 'node:fs';
|
||||
import { join, resolve, dirname, relative, sep, basename } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { spawnSync } from 'node:child_process';
|
||||
|
||||
export const REGISTRY_PATH = 'scripts/end-state-registry.json';
|
||||
export const FROZEN_PATH = 'tests/fixtures/end-state-frozen.json';
|
||||
const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '..');
|
||||
|
||||
const toPosix = (p) => p.split(sep).join('/');
|
||||
|
|
@ -101,52 +106,166 @@ export function evaluateCheck(root, check) {
|
|||
}
|
||||
}
|
||||
|
||||
// --- the frozen denominator -------------------------------------------------
|
||||
|
||||
// JSON with object keys sorted, so a signature does not depend on key order.
|
||||
function canonical(value) {
|
||||
if (Array.isArray(value)) return `[${value.map(canonical).join(',')}]`;
|
||||
if (value && typeof value === 'object') {
|
||||
return `{${Object.keys(value).sort().map((k) => `${JSON.stringify(k)}:${canonical(value[k])}`).join(',')}}`;
|
||||
}
|
||||
return JSON.stringify(value ?? null);
|
||||
}
|
||||
|
||||
export function checkSignature(check) {
|
||||
return createHash('sha256').update(canonical(check)).digest('hex');
|
||||
}
|
||||
|
||||
export function loadFrozen(root) {
|
||||
return JSON.parse(readFileSync(join(root, FROZEN_PATH), 'utf8'));
|
||||
}
|
||||
|
||||
// The judged party must not own the ledger: every frozen entry must still be present with
|
||||
// the same check, and the counting configuration must not move. New entries are allowed.
|
||||
export function verifyIntegrity(registry, frozen) {
|
||||
const violations = [];
|
||||
for (const kind of ['defects', 'experiments']) {
|
||||
const byId = new Map();
|
||||
for (const e of registry[kind] || []) {
|
||||
if (byId.has(e.id)) violations.push(`${kind}: duplicate id ${e.id}`);
|
||||
byId.set(e.id, e);
|
||||
}
|
||||
for (const [id, sig] of Object.entries(frozen[kind] || {})) {
|
||||
const e = byId.get(id);
|
||||
if (!e) {
|
||||
violations.push(`${kind}: ${id} is frozen but missing from the registry (an entry may close, never disappear)`);
|
||||
} else if (checkSignature(e.check) !== sig) {
|
||||
violations.push(`${kind}: ${id} has a changed check (signature ${checkSignature(e.check).slice(0, 12)}, frozen ${sig.slice(0, 12)})`);
|
||||
}
|
||||
}
|
||||
}
|
||||
for (const key of ['agents', 'decisions', 'freeze']) {
|
||||
if (canonical(registry[key]) !== canonical(frozen[key])) {
|
||||
violations.push(`${key}: configuration differs from the frozen manifest`);
|
||||
}
|
||||
}
|
||||
const sites = [].concat(registry.agents?.spawnSites ?? []);
|
||||
if (sites.some((s) => /(^|\/)agents\//.test(String(s)))) {
|
||||
violations.push('agents: spawnSites may never include agents/ (an agent file would reference itself)');
|
||||
}
|
||||
return { ok: violations.length === 0, violations };
|
||||
}
|
||||
|
||||
// --- tallies ----------------------------------------------------------------
|
||||
|
||||
const notMeasured = (detail) => ({ open: null, total: null, items: [], detail });
|
||||
|
||||
function tallyEntries(root, entries) {
|
||||
const items = entries.map((e) => ({ id: e.id, summary: e.summary, ...evaluateCheck(root, e.check) }));
|
||||
const items = entries.map((e) => ({ id: e.id, summary: e.summary, probe: e.probe, ...evaluateCheck(root, e.check) }));
|
||||
const phrase = items.some((i) => i.probe === 'phrase');
|
||||
return {
|
||||
open: items.filter((i) => i.status !== 'closed').length,
|
||||
total: items.length,
|
||||
items,
|
||||
detail: '',
|
||||
detail: phrase ? 'a phrase probe closes on rewording: a closed phrase probe is evidence, not proof of behaviour' : '',
|
||||
};
|
||||
}
|
||||
|
||||
const escapeRe = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&');
|
||||
// Lines that can carry an instruction: HTML comments and fenced code are blanked (line numbers
|
||||
// are kept so "the next line" still means the next line). Block quotes need no stripping: every
|
||||
// spawn form below is anchored at the start of the line, and a quoted line starts with `>`.
|
||||
function instructionLines(markdown) {
|
||||
const uncommented = markdown.replace(/<!--[\s\S]*?-->/g, (m) => m.replace(/[^\n]/g, ''));
|
||||
const out = [];
|
||||
let fenced = false;
|
||||
for (const line of uncommented.split('\n')) {
|
||||
if (/^\s*(```|~~~)/.test(line)) { fenced = !fenced; out.push(''); continue; }
|
||||
out.push(fenced ? '' : line);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const NAME = '[a-z0-9][a-z0-9-]*';
|
||||
const ROSTER_ROW = new RegExp(`^\\s*\\|\\s*\`(${NAME})\`\\s*\\|`);
|
||||
const LAUNCH_LINE = /^\s*(?:[-*]\s+|\d+\.\s+)?(?:\*\*)?(?:launch|spawn)\b(?:\*\*)?(.*)$/i;
|
||||
const LAUNCH_NAME = new RegExp(`\`(?:voyage:)?(${NAME})\`|\\*\\*(${NAME})\\*\\*|voyage:(${NAME})`, 'g');
|
||||
const AGENT_BLOCK = new RegExp(`^\\s*\\*\\*(${NAME})\\*\\*`);
|
||||
|
||||
// The agent names a command markdown file actually spawns. A spawn instruction is one of:
|
||||
// 1. a row of a table whose header's first cell is `Agent` (a roster the step launches);
|
||||
// 2. an imperative line that STARTS with Launch/Spawn and names the agent as `name`,
|
||||
// **name** or voyage:name;
|
||||
// 3. a line starting with **name** whose next non-empty line starts with `Prompt:`.
|
||||
// Prose, negations, HTML comments, fenced code and block quotes never count.
|
||||
export function spawnedNames(markdown) {
|
||||
const lines = instructionLines(markdown);
|
||||
const names = new Set();
|
||||
let inTable = false;
|
||||
let roster = false;
|
||||
for (let i = 0; i < lines.length; i++) {
|
||||
const line = lines[i];
|
||||
if (/^\s*\|/.test(line)) {
|
||||
if (!inTable) {
|
||||
inTable = true;
|
||||
roster = /^\s*\|\s*Agent\s*\|/.test(line);
|
||||
continue;
|
||||
}
|
||||
const row = roster && line.match(ROSTER_ROW);
|
||||
if (row) names.add(row[1]);
|
||||
continue;
|
||||
}
|
||||
inTable = false;
|
||||
const launch = line.match(LAUNCH_LINE);
|
||||
if (launch) {
|
||||
for (const m of launch[1].matchAll(LAUNCH_NAME)) names.add(m[1] || m[2] || m[3]);
|
||||
continue;
|
||||
}
|
||||
const block = line.match(AGENT_BLOCK);
|
||||
if (block) {
|
||||
let j = i + 1;
|
||||
while (j < lines.length && lines[j].trim() === '') j++;
|
||||
if (j < lines.length && /^\s*Prompt:/.test(lines[j])) names.add(block[1]);
|
||||
}
|
||||
}
|
||||
return names;
|
||||
}
|
||||
|
||||
function tallyDormantAgents(root, cfg) {
|
||||
try {
|
||||
const agentFiles = expandPath(root, `${cfg.dir}/*.md`);
|
||||
const sites = expandPath(root, cfg.spawnSites).map((f) => readFileSync(f, 'utf8')).join('\n');
|
||||
const spawned = new Set();
|
||||
for (const f of expandPath(root, cfg.spawnSites)) {
|
||||
for (const n of spawnedNames(readFileSync(f, 'utf8'))) spawned.add(n);
|
||||
}
|
||||
const items = [];
|
||||
for (const file of agentFiles) {
|
||||
const text = readFileSync(file, 'utf8');
|
||||
const fm = text.startsWith('---') ? text.split('\n---')[0] : '';
|
||||
if (cfg.referenceMarker && fm.includes(cfg.referenceMarker)) continue;
|
||||
const name = (fm.match(/^name:\s*(\S+)\s*$/m) || [])[1] || basename(file, '.md');
|
||||
const ref = new RegExp(`(?<![A-Za-z0-9-])${escapeRe(name)}(?![A-Za-z0-9-])`);
|
||||
const referenced = ref.test(sites);
|
||||
const isSpawned = spawned.has(name);
|
||||
items.push({
|
||||
id: name,
|
||||
summary: toPosix(relative(root, file)),
|
||||
status: referenced ? 'closed' : 'open',
|
||||
detail: referenced ? '' : `not referenced by any ${cfg.spawnSites}`,
|
||||
status: isSpawned ? 'closed' : 'open',
|
||||
detail: isSpawned ? '' : `no spawn instruction in ${cfg.spawnSites}`,
|
||||
});
|
||||
}
|
||||
return {
|
||||
open: items.filter((i) => i.status === 'open').length,
|
||||
total: items.length,
|
||||
items,
|
||||
detail: 'spawnable = agent files without the reference-document marker; referenced = name appears (word-bounded) in a spawn site',
|
||||
detail: 'spawnable = agent files without the reference-document marker; spawned = named in a spawn instruction (an `Agent` table row, a line starting with Launch/Spawn, or a **name** block followed by Prompt:); prose, comments, fences and quotes do not count',
|
||||
};
|
||||
} catch (err) {
|
||||
return { open: null, total: null, items: [], detail: `not measurable: ${err.message}` };
|
||||
return notMeasured(`not measurable: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
function tallyDecisions(root, cfg) {
|
||||
const abs = join(root, cfg.file);
|
||||
if (!existsSync(abs)) {
|
||||
return { open: null, total: null, items: [], detail: `not measurable: ${cfg.file} not found (it is local-only)` };
|
||||
return notMeasured(`not measurable: ${cfg.file} not found (it is local-only)`);
|
||||
}
|
||||
try {
|
||||
const section = sectionOf(readFileSync(abs, 'utf8'), cfg.section, cfg.file);
|
||||
|
|
@ -158,33 +277,60 @@ function tallyDecisions(root, cfg) {
|
|||
// Format drift guard: a list item that carries neither marker would silently read as "0 open".
|
||||
const unmarked = lines.filter((l) => /^\s*(\d+\.|[-*]) /.test(l) && !openRe.test(l) && !closedRe.test(l));
|
||||
if (unmarked.length > 0) {
|
||||
return {
|
||||
open: null,
|
||||
total: null,
|
||||
items: [],
|
||||
detail: `not measurable: ${unmarked.length} unmarked list item(s) in ${cfg.section} — mark each with the open/decided marker`,
|
||||
};
|
||||
return notMeasured(`not measurable: ${unmarked.length} unmarked list item(s) in ${cfg.section} — mark each with the open/decided marker`);
|
||||
}
|
||||
return {
|
||||
open: open.length,
|
||||
total: open.length + closed.length,
|
||||
items: open.map((l, i) => ({ id: `#${i + 1}`, summary: l.replace(openRe, '').slice(0, 100), status: 'open', detail: '' })),
|
||||
detail: `marker: open = /${cfg.open}/, decided = /${cfg.closed}/`,
|
||||
detail: `marker: open = /${cfg.open}/, decided = /${cfg.closed}/; this counts ticks — a ticked line is not verified to be an actual decision`,
|
||||
};
|
||||
} catch (err) {
|
||||
return { open: null, total: null, items: [], detail: `not measurable: ${err.message}` };
|
||||
return notMeasured(`not measurable: ${err.message}`);
|
||||
}
|
||||
}
|
||||
|
||||
export function measure(root, registry) {
|
||||
function tallyFeatAfterFreeze(root, cfg) {
|
||||
const git = (...args) => spawnSync('git', ['-C', root, ...args], { encoding: 'utf8' });
|
||||
const top = git('rev-parse', '--show-toplevel');
|
||||
if (top.status !== 0) return notMeasured(`not measured: ${toPosix(root)} is not a git work tree`);
|
||||
let atTop = false;
|
||||
try { atTop = realpathSync(top.stdout.trim()) === realpathSync(root); } catch { atTop = false; }
|
||||
if (!atTop) return notMeasured('not measured: the gate root is not the top of its git work tree');
|
||||
const ref = `refs/tags/${cfg.tag}`;
|
||||
if (git('rev-parse', '-q', '--verify', ref).status !== 0) {
|
||||
return notMeasured(`not measured: no freeze tag "${cfg.tag}" yet — open until the freeze is declared`);
|
||||
}
|
||||
const log = git('log', '--format=%s', `${ref}..HEAD`);
|
||||
if (log.status !== 0) return notMeasured(`not measured: git log failed: ${log.stderr.trim()}`);
|
||||
const subjects = log.stdout.split('\n').filter(Boolean);
|
||||
const featRe = new RegExp(cfg.featPattern);
|
||||
const feats = subjects.filter((s) => featRe.test(s));
|
||||
return {
|
||||
open: feats.length,
|
||||
total: subjects.length,
|
||||
items: feats.map((s, i) => ({ id: `#${i + 1}`, summary: s, status: 'open', detail: '' })),
|
||||
detail: `commits after ${cfg.tag} whose subject matches /${cfg.featPattern}/`,
|
||||
};
|
||||
}
|
||||
|
||||
export function measure(root, registry, frozen) {
|
||||
const d = registry.decisions;
|
||||
const f = registry.freeze;
|
||||
const rows = [
|
||||
{ id: 'defects', label: 'open pipeline defects', source: `${REGISTRY_PATH}#defects`, ...tallyEntries(root, registry.defects) },
|
||||
{ id: 'experiments', label: 'unresolved experiments/opt-ins', source: `${REGISTRY_PATH}#experiments`, ...tallyEntries(root, registry.experiments) },
|
||||
{ id: 'dormant-agents', label: 'dormant agents', source: `${registry.agents.dir}/*.md vs ${registry.agents.spawnSites}`, ...tallyDormantAgents(root, registry.agents) },
|
||||
{ id: 'decisions', label: 'open operator decisions', source: `${d.file} ${d.section.replace(/^#+\s*/, '§ ')}`, ...tallyDecisions(root, d) },
|
||||
{ id: 'feat-after-freeze', label: 'feat commits after the freeze tag', source: `git log ${f.tag}..HEAD`, ...tallyFeatAfterFreeze(root, f) },
|
||||
].map((r) => ({ ...r, target: 0 }));
|
||||
return { green: rows.every((r) => r.open === 0), rows };
|
||||
let integrity;
|
||||
try {
|
||||
integrity = { ...verifyIntegrity(registry, frozen ?? loadFrozen(root)), detail: '' };
|
||||
} catch (err) {
|
||||
integrity = { ok: null, violations: [], detail: `not measurable: cannot read ${FROZEN_PATH}: ${err.message}` };
|
||||
}
|
||||
return { green: rows.every((r) => r.open === 0) && integrity.ok === true, rows, integrity };
|
||||
}
|
||||
|
||||
export function loadRegistry(root) {
|
||||
|
|
@ -198,7 +344,7 @@ export function loadRegistry(root) {
|
|||
for (const key of ['defects', 'experiments']) {
|
||||
if (!Array.isArray(reg[key])) throw new Error(`${REGISTRY_PATH}: "${key}" must be a list`);
|
||||
}
|
||||
for (const key of ['agents', 'decisions']) {
|
||||
for (const key of ['agents', 'decisions', 'freeze']) {
|
||||
if (!reg[key] || typeof reg[key] !== 'object') throw new Error(`${REGISTRY_PATH}: "${key}" must be an object`);
|
||||
}
|
||||
return reg;
|
||||
|
|
@ -208,6 +354,15 @@ export function render(result) {
|
|||
const zero = result.rows.filter((r) => r.open === 0).length;
|
||||
const out = [];
|
||||
out.push(`Voyage end-state gate: ${result.green ? 'GREEN' : 'RED'} (${zero} of ${result.rows.length} tallies at 0)`);
|
||||
const integrity = result.integrity;
|
||||
if (integrity?.ok === true) {
|
||||
out.push(`registry integrity: intact against ${FROZEN_PATH}`);
|
||||
} else if (integrity?.ok === false) {
|
||||
out.push('registry integrity: VIOLATED — the gate cannot be green while the ledger differs from its frozen denominator');
|
||||
for (const v of integrity.violations) out.push(` - ${v}`);
|
||||
} else if (integrity) {
|
||||
out.push(`registry integrity: n/a — ${integrity.detail}`);
|
||||
}
|
||||
out.push('');
|
||||
out.push('| tally | open | of | target | source |');
|
||||
out.push('|---|---|---|---|---|');
|
||||
|
|
@ -221,9 +376,14 @@ export function render(result) {
|
|||
out.push(`${r.id} — ${r.label}${r.detail ? ` (${r.detail})` : ''}`);
|
||||
for (const i of r.items) {
|
||||
if (i.status === 'closed') continue;
|
||||
const tag = i.status === 'not-fellable' ? 'NOT FELLABLE, counted open' : 'open';
|
||||
let tag = i.status === 'not-fellable' ? 'NOT FELLABLE, counted open' : 'open';
|
||||
if (i.probe === 'phrase') tag += ' · phrase probe';
|
||||
out.push(` [${tag}] ${i.id}: ${i.summary}${i.detail ? ` — ${i.detail}` : ''}`);
|
||||
}
|
||||
const closed = r.items.filter((i) => i.status === 'closed' && i.probe);
|
||||
if (closed.length > 0) {
|
||||
out.push(` closed: ${closed.map((i) => (i.probe === 'phrase' ? `${i.id} (phrase probe)` : i.id)).join(', ')}`);
|
||||
}
|
||||
}
|
||||
return out.join('\n');
|
||||
}
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue