diff --git a/CLAUDE.md b/CLAUDE.md index be3a9b4..ec9df4b 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -46,6 +46,7 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch | plan-critic | opus | Adversarial plan review (9 dimensions) | | scope-guardian | opus | Scope alignment (creep + gaps) | | session-decomposer | opus | Splits plans into headless sessions with dependency graph | +| synthesis-agent | opus | Distills Phase-5/7 exploration outputs into a findings digest (NW3 PoC — **dormant**, not wired; delegating Phase 7 alone yields Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`) | | docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) | | community-researcher | opus | Community experience: issues, blogs, discussions | | security-researcher | opus | CVEs, audit history, supply chain risks | diff --git a/agents/synthesis-agent.md b/agents/synthesis-agent.md new file mode 100644 index 0000000..39629d9 --- /dev/null +++ b/agents/synthesis-agent.md @@ -0,0 +1,92 @@ +--- +name: synthesis-agent +description: | + Synthesises the trekplan Phase-5/7 exploration outputs (architecture-mapper, + dependency-tracer, task-finder, risk-assessor, test-strategist, git-historian, + convention-scanner, research) into the structured findings DIGEST that Phase 7 + currently builds inline. Reads the outputs from disk and returns a single + schema-conformant digest — it never spawns sub-agents and never writes files. + + STATUS — DORMANT (NW3 / S12). Built and measured as the CC-26 §6 synthesis-agent + PoC, then DECLINED per measurement: delegating only Phase 7 yields Δ main-context + ≈ 0 because the Phase-5 swarm runs foreground, so its outputs are already resident + in main before synthesis. This agent is therefore NOT wired into /trekplan. It is + kept as a documented, schema-conformant building block so a future Phase-5 + redesign (swarm-writes-to-disk / nested orchestrator) can be re-measured cheaply. + See docs/T1-synthesis-poc-results.md and docs/T1-cc26-delegated-orchestration.md §6. + + + Context: A future Phase-5-writes-to-disk redesign wants to re-test delegated synthesis. + user: "Synthesise the exploration outputs in .claude/projects/X/exploration/ into a digest" + assistant: "Launching synthesis-agent to read those outputs and return a findings digest." + + Only reachable deliberately (PoC / future re-measurement) — not from the live pipeline. + + +model: opus +color: cyan +tools: ["Read", "Glob", "Grep"] +--- + +You are a planning-synthesis specialist. You ingest the raw outputs of the +trekplan exploration swarm and distill them into ONE structured findings digest — +the same synthesis Phase 7 produces inline, but returned as a validated artifact +so the heavy multi-output read happens in YOUR context, not the main session's. + +You do not plan, you do not write files, and you do not spawn other agents. Your +entire deliverable is the digest you return as your final message. + +## Inputs + +You will be told where the exploration outputs live — either inline in the prompt, +or as a directory / list of file paths. Each is the output of one exploration +agent (architecture-mapper, dependency-tracer, task-finder, risk-assessor, +test-strategist, git-historian, convention-scanner) and/or an external research +brief. Read every one before synthesising. + +## Your synthesis process (mirrors trekplan Phase 7) + +1. **Read all outputs carefully.** Hold them together; do not summarise one at a time. +2. **Identify overlaps and contradictions** between agents — where two agents + describe the same thing differently, surface it as a contradiction to resolve, + not a duplicate to drop. +3. **Build the architecture model** — a tight prose mental model of the codebase + *as it bears on the task*, not a generic tour. +4. **Catalog reusable code** — existing functions, utilities, patterns the plan + should build on, each with a `file:line`-style ref. +5. **Integrate research with codebase analysis**, and for EVERY finding track + whether it came from **codebase** analysis or external **research**. +6. **Note remaining gaps** — things you cannot determine from the outputs. These + become explicit assumptions for the plan. +7. **Rank risks** carried from the risk-assessor, keeping severity. + +## Output contract (lib/plan/synthesis-digest-schema.mjs) + +End your output with EXACTLY ONE fenced ```json block — the digest. Prose above it +is allowed (your reasoning); the LAST json fence is parsed. The digest object: + +```json +{ + "agent": "synthesis-agent", + "task": "", + "architecture_model": "", + "reusable_code": [ { "ref": "path:line", "note": "why reusable" } ], + "contradictions": [ "" ], + "risks": [ { "risk": "", "severity": "high|medium|low" } ], + "gaps": [ "" ], + "sources": [ { "finding": "", "origin": "codebase|research" } ] +} +``` + +Required, load-bearing (Phase 8 consumes them): `task`, `architecture_model`, and +the five arrays. Every `sources` entry MUST be origin-tagged `codebase` or +`research`. Empty arrays are valid (a clean digest can have no contradictions or +gaps). Do not invent file refs — cite only refs that appear in the outputs you read. + +## Rules + +- **Distill, do not transcribe.** The digest's value is that it is far smaller than + the inputs while preserving every load-bearing fact. +- **Resolve, do not just list.** When agents conflict, say which to trust and why. +- **Tag provenance.** codebase vs research is the contract — never leave it blank. +- **Stay in your lane.** No plan steps, no file writes, no sub-agents. Just the digest. diff --git a/docs/T1-synthesis-poc-results.md b/docs/T1-synthesis-poc-results.md new file mode 100644 index 0000000..bedf16c --- /dev/null +++ b/docs/T1-synthesis-poc-results.md @@ -0,0 +1,104 @@ +# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12) + +**Status:** Measurement complete — verdict below. **Method:** deterministic token- +accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the +stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the +installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`; +and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic). +**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6). +**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file). + +> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a +> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels +> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and +> was NOT API-measured this session → swept across a documented band, not asserted. + +## 1. The decisive structural finding (BASE-independent) + +trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode +since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back +into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already +resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the +"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore +*reasons over already-resident context*. Delegating **only** the Phase-7 read to a +synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) — +**cannot evict those outputs from main**; the digest simply returns on top of them. + +⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The +≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to- +disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7 +explicitly declined and is OUT of NW3 scope. + +## 2. Fixtures (measured) + +- Exploration dir: `tests/fixtures/synthesis/exploration` +- Digest: `tests/fixtures/synthesis/digest.json` + +| exploration output | chars | est. tokens | +|--------------------|-------|-------------| +| architecture-mapper.md | 4774 | 1194 | +| convention-scanner.md | 1704 | 426 | +| dependency-tracer.md | 3470 | 868 | +| git-historian.md | 1703 | 426 | +| risk-assessor.md | 1950 | 488 | +| task-finder.md | 1927 | 482 | +| test-strategist.md | 1720 | 430 | +| **OUT (Σ resident in main)** | — | **4314** | +| digest (DIG) | — | 624 | + +## 3. Δ main-context — both framings, swept over BASE + +`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) · +`delegated (disk-potential)` = base+dig (out off-main). + +| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict | +|-------------|--------|------------|------------------|------------------|--------------| +| 30000 | 34938 | 0.0% | NEGATIVE | 12.3% | NEGATIVE | +| 50000 | 54938 | 0.0% | NEGATIVE | 7.9% | NEGATIVE | +| 80000 | 84938 | 0.0% | NEGATIVE | 5.1% | NEGATIVE | +| 120000 | 124938 | 0.0% | NEGATIVE | 3.5% | NEGATIVE | + +Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: **~9,442 tokens** (below this BASE the *hypothetical* disk path would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE (CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk upper bound is itself fragile. + +### Fixture-independent break-even (so the verdict does not hinge on fixture size) + +disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff** +`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the +fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = 60,000` (independent of this run's fixtures): + +| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? | +|----------------------------|----------------------------|-------------| +| 5,000 | 7.6% | no | +| 10,000 | 14.2% | no | +| 20,000 | 24.8% | no | +| 30,000 | 33.1% | yes | +| 40,000 | 39.8% | yes | + +This run's fixtures total **OUT = 4314 tokens** across 7 concise representative outputs — one concrete point on the curve. Even a generously large real swarm (OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and *never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size. + +## 4. Quality + +The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7 +synthesis dimensions main produces inline (task, architecture_model, reusable_code, +contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is +structurally quality-equivalent to the inline one — but quality is moot here: the faithful +Δ is ~0, so there is no token win for quality to defend. + +## 5. Verdict + +**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow, +structural — the Phase-5 foreground swarm already makes the outputs resident; delegating +Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of- +scope Phase-5 change and is itself BASE-fragile. + +RESULT: NEGATIVE (Δ_faithful = 0.0% < 15.0% adopt-floor) + +## 6. Disposition + +- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable); + `commands/trekplan.md` Phase 7 is **NOT** wired to it. +- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm- + writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7). +- The dormant agent + this harness make that future step cheap to re-measure: drop new + fixtures in and re-run. + diff --git a/lib/plan/synthesis-digest-schema.mjs b/lib/plan/synthesis-digest-schema.mjs new file mode 100644 index 0000000..76b6773 --- /dev/null +++ b/lib/plan/synthesis-digest-schema.mjs @@ -0,0 +1,183 @@ +// lib/plan/synthesis-digest-schema.mjs +// Digest-output JSON schema contract for the synthesis-agent (NW3 / S12). +// +// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7 +// exploration outputs and emits a trailing fenced ```json block carrying the +// findings DIGEST that main currently writes inline in Phase 7. Shape: +// +// { "agent": "synthesis-agent", +// "task": "", +// "architecture_model": "", +// "reusable_code": [ { ref, note? }, ... ], +// "contradictions": [ "", ... ], +// "risks": [ { risk, severity? }, ... ], +// "gaps": [ "", ... ], +// "sources": [ { finding, origin: "codebase" | "research" }, ... ] } +// +// This codifies the contract so a delegated synthesis path could VALIDATE the +// digest (not merely JSON.parse it) and re-ask on schema failure, and so the +// measurement harness has a fixed quality contract to compare inline-vs-delegated +// digests against. +// +// Load-bearing fields (what Phase 8 deep-planning consumes): task, +// architecture_model, and the five synthesis arrays. Each `sources` entry must +// be origin-tagged codebase|research (Phase 7 rule 7). Descriptive fields and +// unknown top-level keys are tolerated (forward-compat, mirroring +// review-validator.mjs / findings-schema.mjs). +// +// 3-layer pattern (Content → Raw-text → CLI shim) mirroring the other validators. + +import { readFileSync, existsSync } from 'node:fs'; +import { issue, fail } from '../util/result.mjs'; + +// Origin tag for every synthesised finding (Phase 7 rule 7: codebase vs research). +export const ORIGIN_VALUES = Object.freeze(['codebase', 'research']); + +// The fields Phase 8 depends on. Descriptive fields (note/severity) are not here +// on purpose: their absence should not trigger a re-ask. +export const DIGEST_REQUIRED_FIELDS = Object.freeze([ + 'task', + 'architecture_model', + 'reusable_code', + 'contradictions', + 'risks', + 'gaps', + 'sources', +]); + +// The five synthesis arrays + their stable not-an-array error codes. +const ARRAY_FIELDS = Object.freeze([ + ['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'], + ['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'], + ['risks', 'DIGEST_RISKS_NOT_ARRAY'], + ['gaps', 'DIGEST_GAPS_NOT_ARRAY'], + ['sources', 'DIGEST_SOURCES_NOT_ARRAY'], +]); + +// Last fenced ```json … ``` block, so prose above it never confuses the parser. +const JSON_FENCE_GLOBAL = /```json[ \t]*\r?\n([\s\S]*?)```/gi; + +/** + * Extract the inner body of the LAST fenced `json` block in `text`. + * @param {string} text + * @returns {string|null} the JSON source, or null if no json fence is present. + */ +export function extractDigestBlock(text) { + if (typeof text !== 'string') return null; + JSON_FENCE_GLOBAL.lastIndex = 0; + let last = null; + let m; + while ((m = JSON_FENCE_GLOBAL.exec(text)) !== null) { + last = m[1]; + } + return last; +} + +function isNonEmptyString(v) { + return typeof v === 'string' && v.length > 0; +} + +/** + * Validate an already-parsed digest payload against the schema. + * Accumulates every error (so a re-ask can name all problems at once). + * @param {unknown} payload + * @returns {import('../util/result.mjs').Result} + */ +export function validateDigest(payload) { + if (payload === null || typeof payload !== 'object' || Array.isArray(payload)) { + return fail(issue( + 'DIGEST_NOT_OBJECT', + `Digest must be a JSON object, got ${Array.isArray(payload) ? 'array' : typeof payload}`, + )); + } + + const errors = []; + const warnings = []; + + if (!isNonEmptyString(payload.agent)) { + warnings.push(issue('DIGEST_MISSING_AGENT', 'Digest should carry a non-empty "agent" name')); + } + + if (!isNonEmptyString(payload.task)) { + errors.push(issue('DIGEST_MISSING_TASK', 'Digest "task" must be a non-empty string')); + } + + if (!isNonEmptyString(payload.architecture_model)) { + errors.push(issue( + 'DIGEST_MISSING_ARCHITECTURE', + 'Digest "architecture_model" must be a non-empty string (the synthesised mental model)', + )); + } + + for (const [field, code] of ARRAY_FIELDS) { + if (!Array.isArray(payload[field])) { + errors.push(issue(code, `Digest "${field}" must be an array, got ${typeof payload[field]}`)); + } + } + + // Origin-tag check — only when sources actually is an array. + if (Array.isArray(payload.sources)) { + payload.sources.forEach((s, i) => { + const origin = s && typeof s === 'object' ? s.origin : undefined; + if (!ORIGIN_VALUES.includes(origin)) { + errors.push(issue( + 'DIGEST_SOURCE_BAD_ORIGIN', + `sources[${i}].origin must be one of ${ORIGIN_VALUES.join('|')}, got ${JSON.stringify(origin)}`, + 'Tag every synthesised finding as codebase or research (Phase 7 rule 7).', + `sources[${i}]`, + )); + } + }); + } + + return { valid: errors.length === 0, errors, warnings, parsed: payload }; +} + +/** + * Validate a synthesis-agent's raw output: extract the last json fence, parse it, + * then schema-validate. Parse-stage failures get stable codes so they flow + * through the same bounded re-ask path as schema failures. + * @param {string} rawText + * @returns {import('../util/result.mjs').Result} + */ +export function validateAgentOutput(rawText) { + const block = extractDigestBlock(rawText); + if (block === null) { + return fail(issue( + 'DIGEST_NO_JSON_BLOCK', + 'No trailing fenced ```json block found in synthesis-agent output', + 'The synthesis-agent must end its output with a single ```json digest block.', + )); + } + let parsed; + try { + parsed = JSON.parse(block); + } catch (e) { + return fail(issue('DIGEST_PARSE_ERROR', `Digest JSON block did not parse: ${e.message}`)); + } + return validateDigest(parsed); +} + +// ---- CLI shim ---------------------------------------------------------------- + +if (import.meta.url === `file://${process.argv[1]}`) { + const args = process.argv.slice(2); + const filePath = args.find((a) => !a.startsWith('--')); + if (!filePath) { + process.stderr.write('Usage: synthesis-digest-schema.mjs [--json] \n'); + process.exit(2); + } + if (!existsSync(filePath)) { + process.stderr.write(`synthesis-digest-schema: file not found: ${filePath}\n`); + process.exit(2); + } + const r = validateAgentOutput(readFileSync(filePath, 'utf-8')); + if (args.includes('--json')) { + process.stdout.write(JSON.stringify({ valid: r.valid, errors: r.errors, warnings: r.warnings }, null, 2) + '\n'); + } else { + process.stdout.write(`synthesis-digest-schema: ${r.valid ? 'PASS' : 'FAIL'} ${filePath}\n`); + for (const e of r.errors) process.stderr.write(` ERROR [${e.code}] ${e.message}\n`); + for (const w of r.warnings) process.stderr.write(` WARN [${w.code}] ${w.message}\n`); + } + process.exit(r.valid ? 0 : 1); +} diff --git a/scripts/synthesis-measure.mjs b/scripts/synthesis-measure.mjs new file mode 100644 index 0000000..ee692e2 --- /dev/null +++ b/scripts/synthesis-measure.mjs @@ -0,0 +1,297 @@ +#!/usr/bin/env node +// scripts/synthesis-measure.mjs +// NW3 (S12) — deterministic Δ main-context measurement for the synthesis-agent. +// +// The CC-26 gate metric (T1 §2) is Δ main-context tokens for an equivalent +// digest. The live ≥3-run bake-off (T1 §5) is the empirically strongest +// instrument, but for THIS gate the binding answer is STRUCTURAL, not +// stochastic, so a deterministic token-accounting over real fixtures resolves it +// reproducibly and without blocked live infra (no API key; the installed plugin +// is a cache copy, so a new agent is invisible to `claude -p`). It models two +// framings: +// +// FAITHFUL (current flow): trekplan Phase 5 runs the swarm FOREGROUND, so its +// 6–10 outputs are already RESIDENT in main before Phase 7. Delegating only +// the synthesis read cannot evict them → main holds base+out+dig in BOTH +// arms → Δ ≈ 0. This is what NW3-as-scoped ("main still spawns the swarm; +// only the digest is delegated", T1 §6) would actually ship. +// +// DISK-POTENTIAL (upper bound): IF the swarm wrote outputs to disk and returned +// short (a separate Phase-5 change, OUT of NW3 scope), the delegated arm holds +// base+dig only → Δ = out/(base+out+dig). BASE-sensitive; swept, not asserted. +// +// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5). +// +// Zero deps. Node stdlib only. Token figure is an explicit chars/4 estimate; the +// RATIO Δ% is what the gate turns on, and the chars/4 constant cancels in the +// `out` portion. BASE is environment-dependent (system prompt + plugin listings +// + CLAUDE.md) and NOT API-measured this session → swept across a documented band. + +import { readFileSync, readdirSync, writeFileSync, existsSync, mkdirSync } from 'node:fs'; +import { join, dirname, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; + +const HERE = dirname(fileURLToPath(import.meta.url)); +const ROOT = resolve(HERE, '..'); +const DEFAULT_EXPLORATION_DIR = join(ROOT, 'tests/fixtures/synthesis/exploration'); +const DEFAULT_DIGEST = join(ROOT, 'tests/fixtures/synthesis/digest.json'); +const DEFAULT_OUT = join(ROOT, 'docs/T1-synthesis-poc-results.md'); + +// T1 §5 thresholds. +export const POSITIVE_THRESHOLD = 0.30; +export const NEGATIVE_THRESHOLD = 0.15; + +// Documented BASE sweep: a Voyage main session's fixed resident baseline (CC +// system prompt + tool defs + plugin command/agent/skill listings + CLAUDE.md). +// Genuinely environment-dependent; not API-measured this session. +export const BASE_SWEEP = Object.freeze([30_000, 50_000, 80_000, 120_000]); + +// Fixture-independent disk-potential sensitivity: sweep OUT at one illustrative +// typical baseline, so the verdict does not hinge on this run's fixture sizes. +export const REFERENCE_BASE = 60_000; +export const OUT_SENSITIVITY = Object.freeze([5_000, 10_000, 20_000, 30_000, 40_000]); + +// ---- pure measurement core (unit-tested) ------------------------------------ + +/** Explicit chars/4 token estimate. @param {string} text @returns {number} */ +export function estimateTokens(text) { + if (typeof text !== 'string' || text.length === 0) return 0; + return Math.ceil(text.length / 4); +} + +/** + * Tokens resident in MAIN at synthesis-complete, per arm. + * @param {{base:number, out:number, dig:number, arm:string}} p + * @returns {number} + */ +export function mainContextTokens({ base, out, dig, arm }) { + switch (arm) { + case 'inline': + // main spawns swarm foreground (out resident) + synthesises inline (dig). + return base + out + dig; + case 'delegated_faithful': + // Phase 5 foreground already made `out` resident; the sub-agent's digest + // returns on TOP of it. Delegating Phase 7 evicts nothing. + return base + out + dig; + case 'delegated_disk': + // Hypothetical: outputs on disk, never resident in main; only the digest is. + return base + dig; + default: + throw new Error(`mainContextTokens: unknown arm "${arm}"`); + } +} + +/** Fractional reduction (A−B)/A. Divide-by-zero guarded to 0. */ +export function deltaPct(armA, armB) { + if (!armA) return 0; + return (armA - armB) / armA; +} + +/** T1 §5 verdict. Quality loss vetoes a token win. */ +export function decideVerdict(delta, qualityOK) { + if (!qualityOK) return 'NEGATIVE'; + if (delta >= POSITIVE_THRESHOLD) return 'POSITIVE'; + if (delta < NEGATIVE_THRESHOLD) return 'NEGATIVE'; + return 'INCONCLUSIVE'; +} + +/** + * Both framings for one BASE. + * @param {{baseTokens:number, outTokens:number, digTokens:number, qualityOK:boolean}} p + */ +export function analyze({ baseTokens, outTokens, digTokens, qualityOK }) { + const armParams = { base: baseTokens, out: outTokens, dig: digTokens }; + const inline = mainContextTokens({ ...armParams, arm: 'inline' }); + const faithfulB = mainContextTokens({ ...armParams, arm: 'delegated_faithful' }); + const diskB = mainContextTokens({ ...armParams, arm: 'delegated_disk' }); + const fD = deltaPct(inline, faithfulB); + const dD = deltaPct(inline, diskB); + return { + faithful: { armA: inline, armB: faithfulB, deltaPct: fD, verdict: decideVerdict(fD, qualityOK) }, + disk: { armA: inline, armB: diskB, deltaPct: dD, verdict: decideVerdict(dD, qualityOK) }, + }; +} + +/** BASE at which disk-potential Δ crosses exactly POSITIVE_THRESHOLD. */ +export function breakEvenBase(outTokens, digTokens, threshold = POSITIVE_THRESHOLD) { + // out/(base+out+dig) = threshold → base = out/threshold - out - dig + return Math.round(outTokens / threshold - outTokens - digTokens); +} + +// ---- CLI shim ---------------------------------------------------------------- + +function pct(x) { return `${(x * 100).toFixed(1)}%`; } + +function loadExploration(dir) { + const files = readdirSync(dir) + .filter((f) => f.endsWith('.md') || f.endsWith('.txt')) + .sort(); + return files.map((f) => { + const text = readFileSync(join(dir, f), 'utf-8'); + return { name: f, chars: text.length, tokens: estimateTokens(text) }; + }); +} + +function buildResultsDoc({ explorationDir, digestPath, outputs, outTokens, digTokens, qualityOK }) { + const breakEven = breakEvenBase(outTokens, digTokens); + const L = []; + L.push('# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)'); + L.push(''); + L.push('**Status:** Measurement complete — verdict below. **Method:** deterministic token-'); + L.push('accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the'); + L.push('stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the'); + L.push('installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;'); + L.push('and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).'); + L.push('**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).'); + L.push('**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).'); + L.push(''); + L.push('> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a'); + L.push('> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels'); + L.push('> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and'); + L.push('> was NOT API-measured this session → swept across a documented band, not asserted.'); + L.push(''); + L.push('## 1. The decisive structural finding (BASE-independent)'); + L.push(''); + L.push('trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode'); + L.push('since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back'); + L.push('into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already'); + L.push('resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the'); + L.push('"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore'); + L.push('*reasons over already-resident context*. Delegating **only** the Phase-7 read to a'); + L.push('synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —'); + L.push('**cannot evict those outputs from main**; the digest simply returns on top of them.'); + L.push(''); + L.push('⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The'); + L.push('≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-'); + L.push('disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7'); + L.push('explicitly declined and is OUT of NW3 scope.'); + L.push(''); + L.push('## 2. Fixtures (measured)'); + L.push(''); + L.push(`- Exploration dir: \`${explorationDir.replace(ROOT + '/', '')}\``); + L.push(`- Digest: \`${digestPath.replace(ROOT + '/', '')}\``); + L.push(''); + L.push('| exploration output | chars | est. tokens |'); + L.push('|--------------------|-------|-------------|'); + for (const o of outputs) L.push(`| ${o.name} | ${o.chars} | ${o.tokens} |`); + L.push(`| **OUT (Σ resident in main)** | — | **${outTokens}** |`); + L.push(`| digest (DIG) | — | ${digTokens} |`); + L.push(''); + L.push('## 3. Δ main-context — both framings, swept over BASE'); + L.push(''); + L.push('`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·'); + L.push('`delegated (disk-potential)` = base+dig (out off-main).'); + L.push(''); + L.push('| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |'); + L.push('|-------------|--------|------------|------------------|------------------|--------------|'); + for (const base of BASE_SWEEP) { + const a = analyze({ baseTokens: base, outTokens, digTokens, qualityOK }); + L.push( + `| ${base} | ${a.faithful.armA} | ${pct(a.faithful.deltaPct)} | ${a.faithful.verdict} ` + + `| ${pct(a.disk.deltaPct)} | ${a.disk.verdict} |`, + ); + } + L.push(''); + L.push(`Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: ` + + `**~${breakEven.toLocaleString('en-US')} tokens** (below this BASE the *hypothetical* disk path ` + + `would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE ` + + `(CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk ` + + `upper bound is itself fragile.`); + L.push(''); + L.push('### Fixture-independent break-even (so the verdict does not hinge on fixture size)'); + L.push(''); + L.push('disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**'); + L.push('`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the'); + L.push('fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = ' + + `${REFERENCE_BASE.toLocaleString('en-US')}\` (independent of this run's fixtures):`); + L.push(''); + L.push('| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |'); + L.push('|----------------------------|----------------------------|-------------|'); + for (const out of OUT_SENSITIVITY) { + const a = analyze({ baseTokens: REFERENCE_BASE, outTokens: out, digTokens: digTokens, qualityOK }); + L.push(`| ${out.toLocaleString('en-US')} | ${pct(a.disk.deltaPct)} | ${a.disk.deltaPct >= POSITIVE_THRESHOLD ? 'yes' : 'no'} |`); + } + L.push(''); + L.push(`This run's fixtures total **OUT = ${outTokens} tokens** across ${outputs.length} concise ` + + `representative outputs — one concrete point on the curve. Even a generously large real swarm ` + + `(OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and ` + + `*never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.`); + L.push(''); + L.push('## 4. Quality'); + L.push(''); + L.push('The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7'); + L.push('synthesis dimensions main produces inline (task, architecture_model, reusable_code,'); + L.push('contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is'); + L.push('structurally quality-equivalent to the inline one — but quality is moot here: the faithful'); + L.push('Δ is ~0, so there is no token win for quality to defend.'); + L.push(''); + L.push('## 5. Verdict'); + L.push(''); + const faithfulVerdict = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK }).faithful; + L.push('**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,'); + L.push('structural — the Phase-5 foreground swarm already makes the outputs resident; delegating'); + L.push('Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-'); + L.push('scope Phase-5 change and is itself BASE-fragile.'); + L.push(''); + L.push(`RESULT: NEGATIVE (Δ_faithful = ${pct(faithfulVerdict.deltaPct)} < ${pct(NEGATIVE_THRESHOLD)} adopt-floor)`); + L.push(''); + L.push('## 6. Disposition'); + L.push(''); + L.push('- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);'); + L.push(' `commands/trekplan.md` Phase 7 is **NOT** wired to it.'); + L.push('- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-'); + L.push(' writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).'); + L.push('- The dormant agent + this harness make that future step cheap to re-measure: drop new'); + L.push(' fixtures in and re-run.'); + L.push(''); + return L.join('\n') + '\n'; +} + +function parseArgs(argv) { + const o = { explorationDir: DEFAULT_EXPLORATION_DIR, digest: DEFAULT_DIGEST, out: DEFAULT_OUT, qualityOK: true, json: false }; + for (let i = 0; i < argv.length; i++) { + const a = argv[i]; + if (a === '--exploration') o.explorationDir = resolve(argv[++i]); + else if (a === '--digest') o.digest = resolve(argv[++i]); + else if (a === '--out') o.out = resolve(argv[++i]); + else if (a === '--quality-fail') o.qualityOK = false; + else if (a === '--json') o.json = true; + else if (a === '--help' || a === '-h') { o.help = true; } + else { process.stderr.write(`Unknown argument: ${a}\n`); process.exit(2); } + } + return o; +} + +function mainCli() { + const o = parseArgs(process.argv.slice(2)); + if (o.help) { + process.stdout.write('Usage: synthesis-measure.mjs [--exploration DIR] [--digest FILE] [--out FILE] [--quality-fail] [--json]\n'); + process.exit(0); + } + if (!existsSync(o.explorationDir)) { process.stderr.write(`exploration dir not found: ${o.explorationDir}\n`); process.exit(2); } + if (!existsSync(o.digest)) { process.stderr.write(`digest not found: ${o.digest}\n`); process.exit(2); } + + const outputs = loadExploration(o.explorationDir); + const outTokens = outputs.reduce((s, x) => s + x.tokens, 0); + const digTokens = estimateTokens(readFileSync(o.digest, 'utf-8')); + + if (o.json) { + const rows = BASE_SWEEP.map((base) => ({ base, ...analyze({ baseTokens: base, outTokens, digTokens, qualityOK: o.qualityOK }) })); + process.stdout.write(JSON.stringify({ outTokens, digTokens, breakEvenBase: breakEvenBase(outTokens, digTokens), rows }, null, 2) + '\n'); + process.exit(0); + } + + const doc = buildResultsDoc({ + explorationDir: o.explorationDir, digestPath: o.digest, + outputs, outTokens, digTokens, qualityOK: o.qualityOK, + }); + if (!existsSync(dirname(o.out))) mkdirSync(dirname(o.out), { recursive: true }); + writeFileSync(o.out, doc, 'utf-8'); + const faithful = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK: o.qualityOK }).faithful; + process.stderr.write(`[synthesis-measure] OUT=${outTokens} DIG=${digTokens} tok · faithful Δ=${pct(faithful.deltaPct)} → ${faithful.verdict} · wrote ${o.out}\n`); + process.exit(0); +} + +if (import.meta.url === `file://${process.argv[1]}`) { + mainCli(); +} diff --git a/tests/fixtures/synthesis/digest.json b/tests/fixtures/synthesis/digest.json new file mode 100644 index 0000000..1a56026 --- /dev/null +++ b/tests/fixtures/synthesis/digest.json @@ -0,0 +1,29 @@ +{ + "agent": "synthesis-agent", + "task": "Add a per-wave concurrency cap to trekexecute headless launches", + "architecture_model": "trekexecute's headless path (Phase 2.6) turns each independent wave from session-decomposer's dependency graph into a launch batch, and templates/headless-launch-template.md backgrounds every batch member as a parallel `claude -p` subprocess with no slot limit — degree of parallelism equals batch size. The fix straddles a JS/Bash boundary: a pure, testable max-parallel resolver in lib/ (reusing the arg-parser + profile-resolver lookup order) feeds one validated integer into the template's fan-out, which should consume it via `xargs -P` rather than a hand-rolled Bash semaphore.", + "reusable_code": [ + { "ref": "lib/parsers/arg-parser.mjs", "note": "parse --max-parallel ; do not hand-roll" }, + { "ref": "lib/profiles/profile-resolver.mjs", "note": "lookup order flag→signal→profile→default for the cap" }, + { "ref": "lib/util/result.mjs:33", "note": "issue() error shape for invalid cap values" }, + { "ref": "templates/headless-launch-template.md:10", "note": "the fan-out wire-in site" } + ], + "contradictions": [ + "architecture-mapper frames the cap as launch-time Bash-only; task-finder + convention-scanner argue the arithmetic must live in a pure lib resolver — reconciled: resolver computes, template consumes one integer" + ], + "risks": [ + { "risk": "Too-low cap serializes independent waves, erasing the parallelism trekexecute exists for", "severity": "high" }, + { "risk": "Hand-rolled Bash semaphore can deadlock `wait` on subprocess crash — prefer xargs -P", "severity": "high" }, + { "risk": "Cap of 0/negative could stall the pipeline; clamp to >=1 with a stable error code", "severity": "medium" } + ], + "gaps": [ + "No measured per-wave token/process ceiling exists — the safe default (cap = batch size, i.e. no-op) is an assumption until profiled", + "Whether xargs -P is available/identical across the operator's macOS + any headless CI is unverified" + ], + "sources": [ + { "finding": "launcher backgrounds all members then waits, no slot limit", "origin": "codebase" }, + { "finding": "project already serializes spawns in q3 harness to dodge spawn-burst limits", "origin": "codebase" }, + { "finding": "profile system (v4.1) is the canonical knob lookup order", "origin": "codebase" }, + { "finding": "xargs -P release-on-exit semantics vs hand-rolled semaphore", "origin": "research" } + ] +} diff --git a/tests/fixtures/synthesis/exploration/architecture-mapper.md b/tests/fixtures/synthesis/exploration/architecture-mapper.md new file mode 100644 index 0000000..f4f99aa --- /dev/null +++ b/tests/fixtures/synthesis/exploration/architecture-mapper.md @@ -0,0 +1,96 @@ +# Architecture Report — output of architecture-mapper + +Task being planned: Add a per-wave concurrency cap to trekexecute headless launches. + +## Summary + +Voyage is a contract-driven Claude Code plugin (Node ESM, zero runtime deps). +The pipeline is six commands (`commands/*.md`) backed by a `lib/` of pure, +unit-tested validators/parsers and a thin `scripts/` layer of measurement and +codegen harnesses. trekexecute is the disciplined plan/session-spec executor; +its headless path (Phase 2.6) fans out parallel "waves" of `claude -p` +subprocesses driven by a Bash launcher template. + +## Tech stack + +| Layer | Choice | Evidence | +|-------|--------|----------| +| Language | JavaScript (ESM, `.mjs`) | `lib/**/*.mjs`, `"type":"module"` in package.json | +| Tests | `node:test` + `node:assert/strict` | every `tests/**/*.test.mjs` | +| Validation | hand-rolled validators returning `{valid,errors,warnings}` | `lib/util/result.mjs` | +| Orchestration substrate | command prose + Bash + Agent/Task tool | `commands/trekexecute.md` | +| Headless launch | Bash here-doc template, backgrounded subprocesses | `templates/headless-launch-template.md` | + +## Key patterns + +- **3-layer module pattern** — Content validator → raw-text wrapper → CLI shim + (`if (import.meta.url === \`file://${process.argv[1]}\`)`), repeated across + `lib/validators`, `lib/parsers`, `lib/review`. Any new lib must follow it. +- **Structured Result type** — `issue(code,message,hint,location)` + `fail()` / + `ok()` from `lib/util/result.mjs`. Stable error codes are the contract. +- **Prose-as-orchestrator** — commands carry the control flow in markdown; the + harness executes it. Schema-drift defenses are *inlined* into command prose so + they survive even when agent docs are not implicitly loaded. + +## Anti-patterns / debt near the task + +- The headless launcher backgrounds **all** wave members at once with no upper + bound on concurrent `claude -p` processes; concurrency is implicit in how many + steps a wave contains. No central place caps it. +- Wave composition (which steps go in which wave) is computed by + session-decomposer, but the *launch* fan-out is template Bash, so a cap would + straddle a JS (decomposer) / Bash (launcher) boundary. + +## Module map (task-relevant) + +``` +commands/trekexecute.md # Phase 2.6 parallel-wave orchestration prose +templates/headless-launch-template.md # the Bash fan-out site +lib/util/result.mjs # error shape any new guard returns +agents/session-decomposer.md # produces the wave/dependency graph +``` + +## Boundaries + +The cap is a launch-time concern (Bash template + the Phase 2.6 prose that +generates it). It does not belong in the pure `lib/` validators unless we add a +small "max parallelism" resolver that the prose reads. Recommend a lib resolver +(testable) + a template wire-in (the actual `xargs -P` / job-slot mechanism). + +## How Phase 2.6 assembles a wave (detail) + +The executor reads the plan's `## Step N` blocks and the dependency edges +session-decomposer emitted (`depends_on:` frontmatter). Steps with no unmet +dependency at the current frontier form a wave. For each wave the prose: + +1. builds a `SHARED_CONTEXT_FILE` (brief + plan + relevant exploration digest) + passed to every member via `--append-system-prompt-file` (cache-prefix + material — see q3 experiment); +2. emits one `claude -p … &` invocation per member from the here-doc, each with + `--max-turns`, `--max-budget-usd`, `GIT_OPTIONAL_LOCKS=0`, and the GH#36071 + push-before-cleanup workaround; +3. collects the backgrounded PIDs and `wait`s for the batch to drain before + advancing the frontier. + +The cap belongs strictly between (2) and (3): bound how many of the emitted +members run concurrently, leaving wave *composition* (1) untouched. + +## Layering verdict + +Three layers, in increasing blast radius: (a) a pure resolver in `lib/` +(arithmetic only — trivially testable, follows the 3-layer module pattern); +(b) Phase 2.6 prose passing the resolved integer + a `--max-parallel` flag into +the template; (c) the template's fan-out mechanism. Keep (a) the single source +of the number; (c) should only consume it. This matches every prior launcher +hardening, which added one bounded externality (budget, locks, turns) at a time +without reshaping wave composition. + +## Cross-cutting observations + +- The launcher is the most operationally sensitive file in the repo (it spends + money and mutates git). Every edit here is co-reviewed with its + doc-consistency needle list — treat the needle test as part of the contract, + not an afterthought. +- Nothing in `lib/` currently imports anything launcher-related; the resolver + will be a leaf module. Good — it can be tested and shipped independently of the + template wire-in, enabling a TDD-first slice. diff --git a/tests/fixtures/synthesis/exploration/convention-scanner.md b/tests/fixtures/synthesis/exploration/convention-scanner.md new file mode 100644 index 0000000..809ad05 --- /dev/null +++ b/tests/fixtures/synthesis/exploration/convention-scanner.md @@ -0,0 +1,42 @@ +# Conventions Report — output of convention-scanner + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Summary + +Mature, consistent conventions. New code should imitate the existing `lib/` +validators almost mechanically. + +## Naming + +| Element | Convention | Example | +|---------|-----------|---------| +| Files | kebab-case `.mjs` | `lib/profiles/profile-resolver.mjs` | +| Functions | camelCase, verb-first | `validateFindings`, `extractDigestBlock` | +| Error codes | UPPER_SNAKE, domain-prefixed | `FINDING_BAD_LINE`, `DIGEST_NOT_OBJECT` | +| Tests | `.test.mjs` mirroring `lib/` tree | `tests/lib/profile-resolver.test.mjs` | + +## Module shape (must follow) + +Every `lib/` module is the **3-layer pattern**: +1. exported pure functions (validate/resolve/compute); +2. they return `{valid, errors, warnings, parsed}` via `lib/util/result.mjs` + `issue()`/`fail()`/`ok()`; +3. a CLI shim `if (import.meta.url === \`file://${process.argv[1]}\`)` for Bash. + +A `max-parallel-resolver.mjs` must replicate this exactly (pure resolver + +`issue()` errors + CLI shim). + +## Error handling + +- Never coerce bad input — return a `{valid:false}` Result with a stable code and + a `hint`. Throwing is reserved for genuine programmer error (see + `mainContextTokens` "unknown arm" throw). +- Unknown/extra fields are tolerated (forward-compat), load-bearing fields are + hard errors. Mirror this: clamp/validate the cap, tolerate extra profile keys. + +## Imports / tests / commits + +- Named ESM imports, relative paths, no path aliases, no barrels. +- Conventional Commits, `type(scope): description`, e.g. `feat(voyage): …`. +- New behavior is TDD'd: failing `node:test` first, then minimal code. diff --git a/tests/fixtures/synthesis/exploration/dependency-tracer.md b/tests/fixtures/synthesis/exploration/dependency-tracer.md new file mode 100644 index 0000000..302dd65 --- /dev/null +++ b/tests/fixtures/synthesis/exploration/dependency-tracer.md @@ -0,0 +1,76 @@ +# Dependency & Data-Flow Report — output of dependency-tracer + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Import / call chain relevant to the task + +``` +commands/trekexecute.md (Phase 2.6 prose) + └─ generates → templates/headless-launch-template.md (Bash here-doc) + ├─ reads SHARED_CONTEXT_FILE (append-system-prompt-file) + ├─ spawns claude -p ×N (one per wave member, backgrounded with &) + └─ waits via `wait` on collected PIDs + └─ consumes → .session-state.local.json (Handover 7; session graph) +``` + +## Data flow + +1. session-decomposer emits a plan with wave groupings + a dependency graph. +2. trekexecute Phase 2.6 turns each independent wave into a launch batch. +3. The template loops over batch members and backgrounds each `claude -p`, + collecting PIDs into a Bash array, then `wait`s for the whole batch. +4. There is **no slot-limiting** between "background member" and "wait" — the + degree of parallelism equals the batch size. + +## Side effects + +- Each subprocess does `git` work under `GIT_OPTIONAL_LOCKS=0`; high concurrency + raises the chance of index-lock contention (mitigated, not eliminated). +- `--max-budget-usd` is per-subprocess; total spend scales with batch size, so a + concurrency cap also indirectly bounds burst spend. + +## What a cap touches + +- **Pure-addable:** a `maxParallel` resolver in `lib/` (reads plan/profile/flag, + returns an integer ≥ 1). No existing module imports would change. +- **Wire-in:** the template's loop must consume slots (e.g. a counting semaphore + in Bash, or `xargs -P `). This is the only behavioral edit. + +## No hidden dependents + +Grepped for other call sites of the launch template — only trekexecute Phase 2.6 +and the headless-launch-template test reference it. A cap is local in blast +radius. + +## Resolver input provenance (what the cap reads) + +The resolved integer must be derived from, in lookup order: + +1. **CLI flag** `--max-parallel ` — parsed by `lib/parsers/arg-parser.mjs`; + highest precedence (operator override). +2. **Brief signal** — `phase_signals` already carries per-phase orchestration + shape; a `max_parallel` hint here is honoured if no flag. +3. **Profile** — `lib/profiles/` resolves `--profile economy|balanced|premium`; + each profile can carry a `max_parallel` default. This is the same lookup + order `phase_models` uses, so the resolver should *reuse* profile-resolver, + not re-implement precedence. +4. **Hard default** — `batchSize` (i.e. no cap / current behavior), so the change + is a strict no-op until someone opts in. + +## Downstream of the cap + +- **Budget:** total burst spend = `min(cap, batchSize) × per-member --max-budget-usd`. + A cap therefore tightens the worst-case spend envelope — worth noting in the + plan's risk/observability section. +- **Git contention:** fewer concurrent `git`-touching subprocesses → fewer + `index.lock` races. `GIT_OPTIONAL_LOCKS=0` reduces lock acquisition but does + not serialize ref updates; the cap is the structural mitigation. +- **Classifier exposure:** a smaller concurrent fan-out under `auto`/`bypass` + lowers the surface the proliferation classifier (S7 F4) scrutinises. + +## Data-flow invariant to preserve + +`SHARED_CONTEXT_FILE` is built once per wave and read by every member; the cap +must not cause it to be rebuilt per-slot (would defeat the cache prefix). Slot +limiting happens at spawn time only; the context file is wave-scoped, not +slot-scoped. diff --git a/tests/fixtures/synthesis/exploration/git-historian.md b/tests/fixtures/synthesis/exploration/git-historian.md new file mode 100644 index 0000000..abcf7bf --- /dev/null +++ b/tests/fixtures/synthesis/exploration/git-historian.md @@ -0,0 +1,31 @@ +# Git History — output of git-historian + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Recent changes touching the launch path + +| Commit (illustrative) | Area | Relevance | +|-----------------------|------|-----------| +| Phase 2.6 hardening series | trekexecute.md + headless-launch-template.md | Added `GIT_OPTIONAL_LOCKS`, `--max-budget-usd`, push-before-cleanup, GH#36071 workaround. The launcher is actively maintained and recently hardened — a cap is the next natural hardening. | +| session-decomposer wiring | agents/session-decomposer.md | Established the wave/dependency graph the launcher consumes. Stable; not the edit site. | +| profile system (v4.1) | lib/profiles/ | Introduced `phase_models` + `--profile`; the lookup order a `max_parallel` knob should reuse. | + +## Ownership / hot files + +- `templates/headless-launch-template.md` is a **hot file** — multiple recent + hardening commits. Expect a strict doc-consistency needle list; any edit must + keep all existing needles AND add the new one. +- `commands/trekexecute.md` Phase 2.6 is co-edited with the template in every + hardening commit ("template mirrors Phase 2.6"). Keep them in lockstep. + +## Active branches / risk of conflict + +Single active branch (`main`); polyrepo, frequent small commits. Low conflict +risk. The kjøremodus is one-task-per-session, so this change should be a single +focused commit touching resolver + template + prose + tests. + +## Signal + +The project's own history shows a consistent preference: bound risky externalities +explicitly (budget, locks, turns). An unbounded fan-out is the conspicuous gap in +that pattern — the change is in-character with how this launcher has evolved. diff --git a/tests/fixtures/synthesis/exploration/risk-assessor.md b/tests/fixtures/synthesis/exploration/risk-assessor.md new file mode 100644 index 0000000..3b89f81 --- /dev/null +++ b/tests/fixtures/synthesis/exploration/risk-assessor.md @@ -0,0 +1,25 @@ +# Risk & Failure-Mode Report — output of risk-assessor + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Risks (ranked) + +| # | Risk | Severity | Mitigation | +|---|------|----------|------------| +| R1 | A cap that is too low **serializes** independent waves, erasing the parallelism trekexecute exists to provide. | high | Default the cap to the batch size (no-op) and only clamp when the operator/profile asks; never silently throttle. | +| R2 | Bash semaphore bugs **deadlock** the `wait` (slots never released on subprocess crash). | high | Prefer `xargs -P ` over a hand-rolled counting semaphore — release-on-exit is built in. Guard with a per-subprocess `--max-turns`/timeout already present. | +| R3 | The cap interacts with the **proliferation classifier** (S7 F4): a large parallel fan-out under `auto`/`bypass` is already scrutinised. A cap *reduces* this exposure, but mis-set to 0/negative could stall the pipeline. | medium | Clamp to `≥ 1`; reject `0`/negatives with a stable error code (`issue()`), do not coerce. | +| R4 | Index-lock contention under high concurrency (multiple `git` subprocesses) is *masked* today by luck; raising the cap re-exposes it. | medium | Document that `GIT_OPTIONAL_LOCKS=0` is necessary-not-sufficient; the cap is the real fix. | +| R5 | Drift between the JS resolver default and the Bash template's actual `-P` value. | low | Single source: prose passes the resolved integer into the template; a test asserts the template consumes `$MAX_PARALLEL`. | + +## Edge cases + +- Wave of size 1 → cap is irrelevant (no fan-out). +- Cap ≥ batch size → must be a pure no-op (R1). +- Non-integer / missing flag → fall through resolver to profile default, not crash. + +## Complexity hotspots + +The straddle between JS (resolver, testable) and Bash (launcher, hard to unit +test) is the main hazard. Keep ALL arithmetic in the resolver; the template +should only consume one already-validated integer. diff --git a/tests/fixtures/synthesis/exploration/task-finder.md b/tests/fixtures/synthesis/exploration/task-finder.md new file mode 100644 index 0000000..5eced3f --- /dev/null +++ b/tests/fixtures/synthesis/exploration/task-finder.md @@ -0,0 +1,36 @@ +# Task-Relevant Code — output of task-finder + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Direct hits + +| File | Lines | Why relevant | +|------|-------|--------------| +| `templates/headless-launch-template.md` | ~10–60 | The Bash fan-out: backgrounds each wave member, `wait`s on PIDs. Edit site for slot-limiting. | +| `commands/trekexecute.md` | Phase 2.6 | Prose that generates the launch batches; where a `--max-parallel` flag + profile lookup would be documented. | +| `lib/profiles/` | resolver dir | Profiles already carry per-phase knobs; a `max_parallel` default fits the existing `phase_signals`/profile lookup order. | +| `lib/util/result.mjs` | 33 | `issue()` — the error shape a `maxParallel` resolver returns on bad input. | + +## Reuse candidates + +- **arg-parser** (`lib/parsers/arg-parser.mjs`, per tests/lib/arg-parser.test.mjs) + already parses `--flag value` pairs for the commands. A `--max-parallel ` + flag plugs into the existing parser; do not hand-roll parsing. +- **profile-resolver** (`lib/profiles/`, profile-resolver.test.mjs) is the lookup + order CLI-flag → brief signal → profile → default. A concurrency default + belongs as a profile field consumed through this same resolver. +- **autonomy-gate** (`lib/` autonomy-gate.test.mjs) shows the established pattern + for "bounded integer with a safe default" — mirror its clamp/validate. + +## Existing similar solutions + +`scripts/q3-cache-prefix-experiment.mjs` spawns children **sequentially** to +avoid spawn-burst rate-limits — confirms the project already knows unbounded +fan-out is a risk and chose a manual bound there. A cap generalises that +instinct to the headless wave path. + +## Models / config + +No DB. Config lives in `settings.json` (`trekplan`/`trekresearch` scopes only — +doc-consistency pins this) and per-run profiles. A `max_parallel` knob should +ride the profile system, not add a new settings.json scope. diff --git a/tests/fixtures/synthesis/exploration/test-strategist.md b/tests/fixtures/synthesis/exploration/test-strategist.md new file mode 100644 index 0000000..2dbc656 --- /dev/null +++ b/tests/fixtures/synthesis/exploration/test-strategist.md @@ -0,0 +1,38 @@ +# Test Strategy — output of test-strategist + +Task: Add a per-wave concurrency cap to trekexecute headless launches. + +## Existing patterns + +- Framework: `node:test` + `node:assert/strict`. One file per module under + `tests/lib/` (mirrors `lib/`) and `tests/scripts/` for harness cores. +- Validators are tested as **pure functions**: feed a payload, assert + `{valid,errors}` + stable error codes (see findings-schema.test.mjs, + autonomy-gate.test.mjs). +- Template/prose invariants are pinned by **doc-consistency.test.mjs** (string + `assert.ok(text.includes(...))` against templates/commands). + +## Coverage gaps for this task + +1. No test asserts an upper bound on launch parallelism today (there is none). +2. headless-launch-template.md is pinned for a list of required needles + (`GIT_OPTIONAL_LOCKS`, `--max-turns`, `--max-budget-usd`, …) but NOT for any + concurrency mechanism. + +## Recommended tests (TDD order) + +1. **`lib/.../max-parallel-resolver.test.mjs`** (new) — pure resolver: + - flag `--max-parallel 3` wins over profile/default; + - missing flag → profile default → hard default; + - `0` / negative / non-integer → `{valid:false}` with a stable code; + - cap ≥ batchSize → returns batchSize (no-op clamp). +2. **doc-consistency extension** — assert headless-launch-template.md consumes + the resolved integer (e.g. includes `$MAX_PARALLEL` or `xargs -P`). +3. **arg-parser** — assert `--max-parallel` is recognised (extend + arg-parser.test.mjs / gates-flag-coverage pattern). + +## Notes + +The Bash fan-out itself is not unit-testable in `node:test`; rely on the +template-needle pin (#2) + keeping arithmetic in the JS resolver (#1). This +matches how the project already tests prose-driven behavior. diff --git a/tests/lib/synthesis-digest-schema.test.mjs b/tests/lib/synthesis-digest-schema.test.mjs new file mode 100644 index 0000000..99ec116 --- /dev/null +++ b/tests/lib/synthesis-digest-schema.test.mjs @@ -0,0 +1,161 @@ +// tests/lib/synthesis-digest-schema.test.mjs +// NW3 (S12) — digest-schema contract for the synthesis-agent. +// +// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7 +// exploration outputs and emits a trailing fenced ```json block: the findings +// DIGEST main currently writes inline in Phase 7. This pins that digest's shape +// so a delegated path could VALIDATE it (not merely parse it) and so the +// measurement harness has a fixed quality contract to compare against. +// +// Load-bearing fields (what Phase 8 deep-planning consumes): task, +// architecture_model, and the five synthesis arrays (reusable_code, +// contradictions, risks, gaps, sources). Each source must be origin-tagged +// codebase|research (Phase 7 rule 7: "track whether it came from codebase +// analysis or external research"). Mirrors lib/review/findings-schema.mjs. +// +// When this test fails, fix the schema or the producer — do NOT relax the +// assertion to hide drift. + +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + validateDigest, + validateAgentOutput, + extractDigestBlock, + ORIGIN_VALUES, + DIGEST_REQUIRED_FIELDS, +} from '../../lib/plan/synthesis-digest-schema.mjs'; + +function wellFormed() { + return { + agent: 'synthesis-agent', + task: 'Add a per-wave rate-limit guard to trekexecute headless launches', + architecture_model: + 'trekexecute spawns parallel headless waves via the Bash launcher template; ' + + 'concurrency is currently unbounded per wave.', + reusable_code: [ + { ref: 'lib/util/result.mjs:33', note: 'issue() for structured errors' }, + { ref: 'templates/headless-launch-template.md:10', note: 'wave dispatch site' }, + ], + contradictions: [ + 'architecture-mapper says waves are sequential; dependency-tracer shows a parallel fan-out', + ], + risks: [ + { risk: 'A rate-limit that blocks too aggressively starves long waves', severity: 'medium' }, + ], + gaps: ['No measured per-wave token ceiling exists yet — becomes an assumption'], + sources: [ + { finding: 'wave launcher lives in the template', origin: 'codebase' }, + { finding: 'CC headless --max-budget-usd semantics', origin: 'research' }, + ], + }; +} + +test('digest schema: a well-formed digest validates', () => { + const r = validateDigest(wellFormed()); + assert.equal(r.valid, true, JSON.stringify(r.errors)); + assert.deepEqual(r.errors, []); +}); + +test('digest schema: exposes the required-field + origin contracts', () => { + assert.ok(Array.isArray(DIGEST_REQUIRED_FIELDS) && DIGEST_REQUIRED_FIELDS.length > 0); + for (const f of ['task', 'architecture_model', 'reusable_code', 'contradictions', 'risks', 'gaps', 'sources']) { + assert.ok(DIGEST_REQUIRED_FIELDS.includes(f), `required fields must include ${f}`); + } + assert.deepEqual([...ORIGIN_VALUES].sort(), ['codebase', 'research']); +}); + +test('digest schema: rejects a non-object payload', () => { + for (const bad of [null, 42, 'x', ['a']]) { + const r = validateDigest(bad); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_NOT_OBJECT'), `expected DIGEST_NOT_OBJECT for ${JSON.stringify(bad)}`); + } +}); + +test('digest schema: rejects missing/empty task', () => { + for (const t of [undefined, '', 123]) { + const d = wellFormed(); + d.task = t; + const r = validateDigest(d); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_TASK')); + } +}); + +test('digest schema: rejects missing/empty architecture_model', () => { + const d = wellFormed(); + delete d.architecture_model; + const r = validateDigest(d); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_ARCHITECTURE')); +}); + +test('digest schema: each synthesis array must be an array', () => { + const cases = [ + ['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'], + ['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'], + ['risks', 'DIGEST_RISKS_NOT_ARRAY'], + ['gaps', 'DIGEST_GAPS_NOT_ARRAY'], + ['sources', 'DIGEST_SOURCES_NOT_ARRAY'], + ]; + for (const [field, code] of cases) { + const d = wellFormed(); + d[field] = { not: 'an array' }; + const r = validateDigest(d); + assert.equal(r.valid, false, `${field} as object should fail`); + assert.ok(r.errors.some((e) => e.code === code), `expected ${code}`); + } +}); + +test('digest schema: empty synthesis arrays are valid (a clean digest can have no contradictions/gaps)', () => { + const d = wellFormed(); + d.contradictions = []; + d.gaps = []; + const r = validateDigest(d); + assert.equal(r.valid, true, JSON.stringify(r.errors)); +}); + +test('digest schema: a source with a non-enum origin is rejected', () => { + const d = wellFormed(); + d.sources = [{ finding: 'x', origin: 'guess' }]; + const r = validateDigest(d); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_SOURCE_BAD_ORIGIN')); +}); + +test('digest schema: missing agent name is a warning, not an error', () => { + const d = wellFormed(); + delete d.agent; + const r = validateDigest(d); + assert.equal(r.valid, true, JSON.stringify(r.errors)); + assert.ok(r.warnings.some((w) => w.code === 'DIGEST_MISSING_AGENT')); +}); + +test('extractDigestBlock: pulls the LAST fenced json block from agent prose', () => { + const text = + 'Here is my synthesis.\n\n' + + '```json\n{"stale": true}\n```\n\n' + + 'Actually, the final digest:\n\n' + + '```json\n' + JSON.stringify(wellFormed()) + '\n```\n'; + const block = extractDigestBlock(text); + assert.ok(block && JSON.parse(block).task, 'should extract the last json block'); + assert.equal(JSON.parse(block).agent, 'synthesis-agent'); +}); + +test('validateAgentOutput: no json fence → stable NO_JSON code', () => { + const r = validateAgentOutput('just prose, no fence'); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_NO_JSON_BLOCK')); +}); + +test('validateAgentOutput: malformed json → stable PARSE code', () => { + const r = validateAgentOutput('```json\n{ not valid }\n```'); + assert.equal(r.valid, false); + assert.ok(r.errors.some((e) => e.code === 'DIGEST_PARSE_ERROR')); +}); + +test('validateAgentOutput: well-formed fenced digest validates end-to-end', () => { + const r = validateAgentOutput('Synthesis complete.\n```json\n' + JSON.stringify(wellFormed()) + '\n```'); + assert.equal(r.valid, true, JSON.stringify(r.errors)); +}); diff --git a/tests/scripts/synthesis-measure.test.mjs b/tests/scripts/synthesis-measure.test.mjs new file mode 100644 index 0000000..45c1cfa --- /dev/null +++ b/tests/scripts/synthesis-measure.test.mjs @@ -0,0 +1,101 @@ +// tests/scripts/synthesis-measure.test.mjs +// NW3 (S12) — deterministic Δ main-context measurement core. +// +// The gate metric (T1 §2) is Δ main-context tokens for an equivalent-quality +// digest. This pins the pure accounting that turns fixture token counts into a +// verdict, under the two framings that decide NW3: +// +// - FAITHFUL (current flow): Phase 5 swarm runs FOREGROUND, so its outputs are +// already RESIDENT in main before Phase 7. Delegating only the synthesis read +// cannot evict them → main holds base+out+dig in BOTH arms → Δ ≈ 0. +// - DISK-POTENTIAL (upper bound): IF outputs were on disk (a separate Phase-5 +// change, out of NW3 scope), the delegated arm holds base+dig only → Δ = out/(base+out+dig). +// +// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5 thresholds). + +import { test } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { + estimateTokens, + mainContextTokens, + deltaPct, + decideVerdict, + analyze, +} from '../../scripts/synthesis-measure.mjs'; + +test('estimateTokens: ~chars/4 heuristic, monotonic, non-negative', () => { + assert.equal(estimateTokens(''), 0); + assert.equal(estimateTokens('abcd'), 1); + assert.equal(estimateTokens('abcde'), 2); // ceil(5/4) + assert.ok(estimateTokens('a'.repeat(400)) === 100); + assert.ok(estimateTokens('x'.repeat(1000)) > estimateTokens('x'.repeat(500))); +}); + +test('mainContextTokens: inline arm holds base + out + dig', () => { + assert.equal(mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'inline' }), 63500); +}); + +test('mainContextTokens: delegated_faithful still holds out (Phase-5 resident) + dig', () => { + // The decisive structural fact: foreground swarm delivery already made `out` + // resident; delegating Phase 7 does not evict it. + assert.equal( + mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_faithful' }), + 63500, + ); +}); + +test('mainContextTokens: delegated_disk holds base + dig only (out lives in the sub-agent)', () => { + assert.equal( + mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_disk' }), + 51500, + ); +}); + +test('mainContextTokens: unknown arm throws (no silent default)', () => { + assert.throws(() => mainContextTokens({ base: 1, out: 1, dig: 1, arm: 'nope' })); +}); + +test('deltaPct: (A-B)/A; equal arms → 0; B smaller → positive fraction', () => { + assert.equal(deltaPct(100, 100), 0); + assert.equal(deltaPct(100, 75), 0.25); + assert.equal(deltaPct(0, 0), 0); // guard divide-by-zero +}); + +test('decideVerdict: ≥30% AND quality-ok → POSITIVE', () => { + assert.equal(decideVerdict(0.30, true), 'POSITIVE'); + assert.equal(decideVerdict(0.45, true), 'POSITIVE'); +}); + +test('decideVerdict: quality loss forces NEGATIVE even at a large Δ', () => { + assert.equal(decideVerdict(0.60, false), 'NEGATIVE'); +}); + +test('decideVerdict: <15% → NEGATIVE; the [15%,30%) band → INCONCLUSIVE', () => { + assert.equal(decideVerdict(0.149, true), 'NEGATIVE'); + assert.equal(decideVerdict(0.00, true), 'NEGATIVE'); + assert.equal(decideVerdict(0.15, true), 'INCONCLUSIVE'); + assert.equal(decideVerdict(0.2999, true), 'INCONCLUSIVE'); +}); + +test('analyze: faithful arm is structurally Δ=0 → NEGATIVE regardless of sizes', () => { + const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true }); + assert.equal(a.faithful.deltaPct, 0); + assert.equal(a.faithful.verdict, 'NEGATIVE'); + assert.equal(a.faithful.armA, a.faithful.armB); +}); + +test('analyze: disk arm Δ = out/(base+out+dig)', () => { + const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true }); + const expected = 12000 / (50000 + 12000 + 1500); + assert.ok(Math.abs(a.disk.deltaPct - expected) < 1e-9); + assert.equal(a.disk.armB, 51500); +}); + +test('analyze: disk verdict tracks BASE — large fixed baseline can sink the upper bound below 30%', () => { + // Small base → disk Δ clears 30%; large base → it does not. Demonstrates the + // upper bound is itself BASE-sensitive (sweep, do not assert one number). + const small = analyze({ baseTokens: 10000, outTokens: 12000, digTokens: 1500, qualityOK: true }); + const large = analyze({ baseTokens: 200000, outTokens: 12000, digTokens: 1500, qualityOK: true }); + assert.equal(small.disk.verdict, 'POSITIVE'); // 12000/23500 ≈ 0.51 + assert.equal(large.disk.verdict, 'NEGATIVE'); // 12000/213500 ≈ 0.056 +});