open/voyage
1
0
Fork 0
Code Issues Pull requests Releases Activity

fix(voyage): S17 — downgrade NW2 bake-off verdict + drop orchestrator Agent-grants

Browse source 
Survivor #3 (bake-off data un-archived): per-run JSON a1..b3.json was never
committed (git diff-filter=A empty), so the medians/jaccard ladder cannot be
re-derived or audited. Commit-path would require fabricated data (forbidden) →
downgraded "POSITIVE" to "opt-in-defensible (single un-archived run)" in
docs/T2-bakeoff-results.md (verdict line + §5 header + new §Reproducibility
caveat) and the v5.5.0 CHANGELOG entry. New doc-consistency pin fails if the
doc re-asserts bare POSITIVE or drops the un-archived disclosure.

Survivor #5 (latent dispatch risk): planning/research/review orchestrators are
inline reference docs no command invokes, yet shipped tools:["Agent",...]. The
harness cannot spawn sub-agents with Agent, so the grant was pure latent risk.
Dropped "Agent" from all three frontmatters; flipped the agent-frontmatter
invariant (its canonical home) from "must include Agent" to "must NOT include
Agent". Removed the duplicate orchestrator pin from doc-consistency to avoid a
double guard (S19 anti-bloat).

Tests 690 → 691 (689 pass / 2 skip / 0 fail). claude plugin validate passes
(1 accepted CLAUDE.md-at-root warning).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-19 06:24:27 +02:00
commit eb5a7324a4
7 changed files with 48 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

12
tests/lib/agent-frontmatter.test.mjs
View file

@ -90,7 +90,11 @@ test('every agents/*.md frontmatter name matches its filename', () => {
}
});
test('orchestrator agents are model: opus and include the Agent tool', () => {
test('orchestrator agents are model: opus and do NOT include the Agent tool (reference-only, never spawned)', () => {
// S17 / Survivor #5: these orchestrator .md files are inline reference docs.
// No command invokes them, and the harness does not expose the Agent tool to
// sub-agents, so an Agent grant here is purely a latent dispatch risk. They
// must NOT carry it — the real orchestration runs in the main session.
for (const f of ORCHESTRATORS) {
const path = `agents/${f}`;
const fm = extractFrontmatter(read(path));
@ -99,12 +103,12 @@ test('orchestrator agents are model: opus and include the Agent tool', () => {
assert.equal(
model,
'opus',
`${path}: orchestrator must be model: opus (drives multi-agent swarm reasoning) — got "${model}"`,
`${path}: orchestrator must be model: opus (documents multi-agent swarm reasoning) — got "${model}"`,
);
const tools = getTopLevelValue(fm, 'tools');
assert.ok(
tools && /\bAgent\b/.test(tools),
`${path}: orchestrator tools: must include "Agent" so it can spawn the swarm — got ${tools}`,
tools && !/\bAgent\b/.test(tools),
`${path}: orchestrator is reference-only — tools: must NOT include "Agent" (Survivor #5) — got ${tools}`,
);
}
});

20
tests/lib/doc-consistency.test.mjs
View file

@ -849,3 +849,23 @@ test('S15: README cost prose does not resurrect the false Sonnet-swarm claim', (
);
}
});
// NOTE (S17): the "reference-only orchestrators must not grant the Agent tool"
// invariant (Survivor #5) is guarded in tests/lib/agent-frontmatter.test.mjs
// (its canonical frontmatter home), not duplicated here.
test('S17: NW2 bake-off verdict is not labeled unqualified POSITIVE (raw data un-archived)', () => {
// Survivor #3: the per-run JSON (a1..b3.json) was never committed
// (git log --diff-filter=A is empty), so the medians/jaccards cannot be
// re-derived, audited, or falsified. The verdict must be qualified as
// opt-in-defensible, and the un-reproducibility must be disclosed.
const doc = read('docs/T2-bakeoff-results.md');
assert.ok(
!/VERDICT:\s*\*\*POSITIVE\*\*/.test(doc),
'T2-bakeoff-results.md still labels the verdict bare POSITIVE — downgrade to opt-in-defensible (Survivor #3).',
);
assert.ok(
/un-archived|not committed|cannot be re-derived/i.test(doc),
'T2-bakeoff-results.md must disclose the raw per-run data is un-archived / un-reproducible.',
);
});