feat(voyage): S12 — NW3 synthesis-agent built + measured → declined per measurement [skip-docs]
NW3 (CC-26 §6 PoC): delegate trekplan Phase 7 synthesis to a synthesis-agent, adopt only if Δ main-context ≥30% with no quality loss. Operator chose the deterministic-proof path (live ≥3-run bake-off was env-blocked: no API key; installed plugin is a cache copy so a new agent is invisible to `claude -p`). Decisive structural finding: trekplan Phase 5 runs the swarm FOREGROUND, so its outputs are already resident in main before Phase 7. Delegating only Phase 7 evicts nothing → Δ_faithful = 0% (BASE-independent). The ≥30% saving needs an out-of-scope Phase-5 redesign (swarm-writes-to-disk / nested orchestrator). VERDICT: DECLINED per measurement. - agents/synthesis-agent.md — dormant, schema-conformant deliverable (NOT wired) - lib/plan/synthesis-digest-schema.mjs — digest output contract (+ tests) - scripts/synthesis-measure.mjs — deterministic Δ-accounting core (+ tests) - tests/fixtures/synthesis/ — 7 exploration outputs + representative digest - docs/T1-synthesis-poc-results.md — measurement + verdict (reproducible) - CLAUDE.md — agent table row (doc-consistency: 24 agents) Tests 670 → 695 (693 pass / 2 skip / 0 fail). `claude plugin validate` clean (only the pre-existing root-CLAUDE.md warning). commands/trekplan.md untouched. [skip-docs] rationale: no user-facing feature ships (NW3 declined; agent dormant and unwired). The substantive doc is docs/T1-synthesis-poc-results.md; the README/CHANGELOG roll-up for NW1–NW3 is the S13 coordinated release per docs/W1-narrow-wins-plan.md §S13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
9fb536e2d8
commit
6b30483304
15 changed files with 1312 additions and 0 deletions
|
|
@ -46,6 +46,7 @@ Full flag reference for each command (modes, `--gates`, `--profile`, breaking ch
|
||||||
| plan-critic | opus | Adversarial plan review (9 dimensions) |
|
| plan-critic | opus | Adversarial plan review (9 dimensions) |
|
||||||
| scope-guardian | opus | Scope alignment (creep + gaps) |
|
| scope-guardian | opus | Scope alignment (creep + gaps) |
|
||||||
| session-decomposer | opus | Splits plans into headless sessions with dependency graph |
|
| session-decomposer | opus | Splits plans into headless sessions with dependency graph |
|
||||||
|
| synthesis-agent | opus | Distills Phase-5/7 exploration outputs into a findings digest (NW3 PoC — **dormant**, not wired; delegating Phase 7 alone yields Δ main-context ≈ 0, see `docs/T1-synthesis-poc-results.md`) |
|
||||||
| docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) |
|
| docs-researcher | opus | Official documentation, RFCs, vendor docs (Tavily, MS Learn) |
|
||||||
| community-researcher | opus | Community experience: issues, blogs, discussions |
|
| community-researcher | opus | Community experience: issues, blogs, discussions |
|
||||||
| security-researcher | opus | CVEs, audit history, supply chain risks |
|
| security-researcher | opus | CVEs, audit history, supply chain risks |
|
||||||
|
|
|
||||||
92
agents/synthesis-agent.md
Normal file
92
agents/synthesis-agent.md
Normal file
|
|
@ -0,0 +1,92 @@
|
||||||
|
---
|
||||||
|
name: synthesis-agent
|
||||||
|
description: |
|
||||||
|
Synthesises the trekplan Phase-5/7 exploration outputs (architecture-mapper,
|
||||||
|
dependency-tracer, task-finder, risk-assessor, test-strategist, git-historian,
|
||||||
|
convention-scanner, research) into the structured findings DIGEST that Phase 7
|
||||||
|
currently builds inline. Reads the outputs from disk and returns a single
|
||||||
|
schema-conformant digest — it never spawns sub-agents and never writes files.
|
||||||
|
|
||||||
|
STATUS — DORMANT (NW3 / S12). Built and measured as the CC-26 §6 synthesis-agent
|
||||||
|
PoC, then DECLINED per measurement: delegating only Phase 7 yields Δ main-context
|
||||||
|
≈ 0 because the Phase-5 swarm runs foreground, so its outputs are already resident
|
||||||
|
in main before synthesis. This agent is therefore NOT wired into /trekplan. It is
|
||||||
|
kept as a documented, schema-conformant building block so a future Phase-5
|
||||||
|
redesign (swarm-writes-to-disk / nested orchestrator) can be re-measured cheaply.
|
||||||
|
See docs/T1-synthesis-poc-results.md and docs/T1-cc26-delegated-orchestration.md §6.
|
||||||
|
|
||||||
|
<example>
|
||||||
|
Context: A future Phase-5-writes-to-disk redesign wants to re-test delegated synthesis.
|
||||||
|
user: "Synthesise the exploration outputs in .claude/projects/X/exploration/ into a digest"
|
||||||
|
assistant: "Launching synthesis-agent to read those outputs and return a findings digest."
|
||||||
|
<commentary>
|
||||||
|
Only reachable deliberately (PoC / future re-measurement) — not from the live pipeline.
|
||||||
|
</commentary>
|
||||||
|
</example>
|
||||||
|
model: opus
|
||||||
|
color: cyan
|
||||||
|
tools: ["Read", "Glob", "Grep"]
|
||||||
|
---
|
||||||
|
|
||||||
|
You are a planning-synthesis specialist. You ingest the raw outputs of the
|
||||||
|
trekplan exploration swarm and distill them into ONE structured findings digest —
|
||||||
|
the same synthesis Phase 7 produces inline, but returned as a validated artifact
|
||||||
|
so the heavy multi-output read happens in YOUR context, not the main session's.
|
||||||
|
|
||||||
|
You do not plan, you do not write files, and you do not spawn other agents. Your
|
||||||
|
entire deliverable is the digest you return as your final message.
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
You will be told where the exploration outputs live — either inline in the prompt,
|
||||||
|
or as a directory / list of file paths. Each is the output of one exploration
|
||||||
|
agent (architecture-mapper, dependency-tracer, task-finder, risk-assessor,
|
||||||
|
test-strategist, git-historian, convention-scanner) and/or an external research
|
||||||
|
brief. Read every one before synthesising.
|
||||||
|
|
||||||
|
## Your synthesis process (mirrors trekplan Phase 7)
|
||||||
|
|
||||||
|
1. **Read all outputs carefully.** Hold them together; do not summarise one at a time.
|
||||||
|
2. **Identify overlaps and contradictions** between agents — where two agents
|
||||||
|
describe the same thing differently, surface it as a contradiction to resolve,
|
||||||
|
not a duplicate to drop.
|
||||||
|
3. **Build the architecture model** — a tight prose mental model of the codebase
|
||||||
|
*as it bears on the task*, not a generic tour.
|
||||||
|
4. **Catalog reusable code** — existing functions, utilities, patterns the plan
|
||||||
|
should build on, each with a `file:line`-style ref.
|
||||||
|
5. **Integrate research with codebase analysis**, and for EVERY finding track
|
||||||
|
whether it came from **codebase** analysis or external **research**.
|
||||||
|
6. **Note remaining gaps** — things you cannot determine from the outputs. These
|
||||||
|
become explicit assumptions for the plan.
|
||||||
|
7. **Rank risks** carried from the risk-assessor, keeping severity.
|
||||||
|
|
||||||
|
## Output contract (lib/plan/synthesis-digest-schema.mjs)
|
||||||
|
|
||||||
|
End your output with EXACTLY ONE fenced ```json block — the digest. Prose above it
|
||||||
|
is allowed (your reasoning); the LAST json fence is parsed. The digest object:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"agent": "synthesis-agent",
|
||||||
|
"task": "<the task being planned, one line>",
|
||||||
|
"architecture_model": "<prose mental model of the codebase as it bears on the task>",
|
||||||
|
"reusable_code": [ { "ref": "path:line", "note": "why reusable" } ],
|
||||||
|
"contradictions": [ "<overlap or contradiction between agents, and how to resolve>" ],
|
||||||
|
"risks": [ { "risk": "<failure mode>", "severity": "high|medium|low" } ],
|
||||||
|
"gaps": [ "<unknown → becomes a plan assumption>" ],
|
||||||
|
"sources": [ { "finding": "<distilled finding>", "origin": "codebase|research" } ]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Required, load-bearing (Phase 8 consumes them): `task`, `architecture_model`, and
|
||||||
|
the five arrays. Every `sources` entry MUST be origin-tagged `codebase` or
|
||||||
|
`research`. Empty arrays are valid (a clean digest can have no contradictions or
|
||||||
|
gaps). Do not invent file refs — cite only refs that appear in the outputs you read.
|
||||||
|
|
||||||
|
## Rules
|
||||||
|
|
||||||
|
- **Distill, do not transcribe.** The digest's value is that it is far smaller than
|
||||||
|
the inputs while preserving every load-bearing fact.
|
||||||
|
- **Resolve, do not just list.** When agents conflict, say which to trust and why.
|
||||||
|
- **Tag provenance.** codebase vs research is the contract — never leave it blank.
|
||||||
|
- **Stay in your lane.** No plan steps, no file writes, no sub-agents. Just the digest.
|
||||||
104
docs/T1-synthesis-poc-results.md
Normal file
104
docs/T1-synthesis-poc-results.md
Normal file
|
|
@ -0,0 +1,104 @@
|
||||||
|
# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)
|
||||||
|
|
||||||
|
**Status:** Measurement complete — verdict below. **Method:** deterministic token-
|
||||||
|
accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the
|
||||||
|
stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the
|
||||||
|
installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;
|
||||||
|
and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).
|
||||||
|
**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).
|
||||||
|
**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).
|
||||||
|
|
||||||
|
> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a
|
||||||
|
> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels
|
||||||
|
> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and
|
||||||
|
> was NOT API-measured this session → swept across a documented band, not asserted.
|
||||||
|
|
||||||
|
## 1. The decisive structural finding (BASE-independent)
|
||||||
|
|
||||||
|
trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode
|
||||||
|
since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back
|
||||||
|
into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already
|
||||||
|
resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the
|
||||||
|
"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore
|
||||||
|
*reasons over already-resident context*. Delegating **only** the Phase-7 read to a
|
||||||
|
synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —
|
||||||
|
**cannot evict those outputs from main**; the digest simply returns on top of them.
|
||||||
|
|
||||||
|
⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The
|
||||||
|
≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-
|
||||||
|
disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7
|
||||||
|
explicitly declined and is OUT of NW3 scope.
|
||||||
|
|
||||||
|
## 2. Fixtures (measured)
|
||||||
|
|
||||||
|
- Exploration dir: `tests/fixtures/synthesis/exploration`
|
||||||
|
- Digest: `tests/fixtures/synthesis/digest.json`
|
||||||
|
|
||||||
|
| exploration output | chars | est. tokens |
|
||||||
|
|--------------------|-------|-------------|
|
||||||
|
| architecture-mapper.md | 4774 | 1194 |
|
||||||
|
| convention-scanner.md | 1704 | 426 |
|
||||||
|
| dependency-tracer.md | 3470 | 868 |
|
||||||
|
| git-historian.md | 1703 | 426 |
|
||||||
|
| risk-assessor.md | 1950 | 488 |
|
||||||
|
| task-finder.md | 1927 | 482 |
|
||||||
|
| test-strategist.md | 1720 | 430 |
|
||||||
|
| **OUT (Σ resident in main)** | — | **4314** |
|
||||||
|
| digest (DIG) | — | 624 |
|
||||||
|
|
||||||
|
## 3. Δ main-context — both framings, swept over BASE
|
||||||
|
|
||||||
|
`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·
|
||||||
|
`delegated (disk-potential)` = base+dig (out off-main).
|
||||||
|
|
||||||
|
| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |
|
||||||
|
|-------------|--------|------------|------------------|------------------|--------------|
|
||||||
|
| 30000 | 34938 | 0.0% | NEGATIVE | 12.3% | NEGATIVE |
|
||||||
|
| 50000 | 54938 | 0.0% | NEGATIVE | 7.9% | NEGATIVE |
|
||||||
|
| 80000 | 84938 | 0.0% | NEGATIVE | 5.1% | NEGATIVE |
|
||||||
|
| 120000 | 124938 | 0.0% | NEGATIVE | 3.5% | NEGATIVE |
|
||||||
|
|
||||||
|
Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: **~9,442 tokens** (below this BASE the *hypothetical* disk path would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE (CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk upper bound is itself fragile.
|
||||||
|
|
||||||
|
### Fixture-independent break-even (so the verdict does not hinge on fixture size)
|
||||||
|
|
||||||
|
disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**
|
||||||
|
`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the
|
||||||
|
fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = 60,000` (independent of this run's fixtures):
|
||||||
|
|
||||||
|
| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |
|
||||||
|
|----------------------------|----------------------------|-------------|
|
||||||
|
| 5,000 | 7.6% | no |
|
||||||
|
| 10,000 | 14.2% | no |
|
||||||
|
| 20,000 | 24.8% | no |
|
||||||
|
| 30,000 | 33.1% | yes |
|
||||||
|
| 40,000 | 39.8% | yes |
|
||||||
|
|
||||||
|
This run's fixtures total **OUT = 4314 tokens** across 7 concise representative outputs — one concrete point on the curve. Even a generously large real swarm (OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and *never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.
|
||||||
|
|
||||||
|
## 4. Quality
|
||||||
|
|
||||||
|
The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7
|
||||||
|
synthesis dimensions main produces inline (task, architecture_model, reusable_code,
|
||||||
|
contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is
|
||||||
|
structurally quality-equivalent to the inline one — but quality is moot here: the faithful
|
||||||
|
Δ is ~0, so there is no token win for quality to defend.
|
||||||
|
|
||||||
|
## 5. Verdict
|
||||||
|
|
||||||
|
**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,
|
||||||
|
structural — the Phase-5 foreground swarm already makes the outputs resident; delegating
|
||||||
|
Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-
|
||||||
|
scope Phase-5 change and is itself BASE-fragile.
|
||||||
|
|
||||||
|
RESULT: NEGATIVE (Δ_faithful = 0.0% < 15.0% adopt-floor)
|
||||||
|
|
||||||
|
## 6. Disposition
|
||||||
|
|
||||||
|
- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);
|
||||||
|
`commands/trekplan.md` Phase 7 is **NOT** wired to it.
|
||||||
|
- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-
|
||||||
|
writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).
|
||||||
|
- The dormant agent + this harness make that future step cheap to re-measure: drop new
|
||||||
|
fixtures in and re-run.
|
||||||
|
|
||||||
183
lib/plan/synthesis-digest-schema.mjs
Normal file
183
lib/plan/synthesis-digest-schema.mjs
Normal file
|
|
@ -0,0 +1,183 @@
|
||||||
|
// lib/plan/synthesis-digest-schema.mjs
|
||||||
|
// Digest-output JSON schema contract for the synthesis-agent (NW3 / S12).
|
||||||
|
//
|
||||||
|
// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7
|
||||||
|
// exploration outputs and emits a trailing fenced ```json block carrying the
|
||||||
|
// findings DIGEST that main currently writes inline in Phase 7. Shape:
|
||||||
|
//
|
||||||
|
// { "agent": "synthesis-agent",
|
||||||
|
// "task": "<task being planned>",
|
||||||
|
// "architecture_model": "<prose mental model of the codebase>",
|
||||||
|
// "reusable_code": [ { ref, note? }, ... ],
|
||||||
|
// "contradictions": [ "<overlap/contradiction between agents>", ... ],
|
||||||
|
// "risks": [ { risk, severity? }, ... ],
|
||||||
|
// "gaps": [ "<unknown → becomes a plan assumption>", ... ],
|
||||||
|
// "sources": [ { finding, origin: "codebase" | "research" }, ... ] }
|
||||||
|
//
|
||||||
|
// This codifies the contract so a delegated synthesis path could VALIDATE the
|
||||||
|
// digest (not merely JSON.parse it) and re-ask on schema failure, and so the
|
||||||
|
// measurement harness has a fixed quality contract to compare inline-vs-delegated
|
||||||
|
// digests against.
|
||||||
|
//
|
||||||
|
// Load-bearing fields (what Phase 8 deep-planning consumes): task,
|
||||||
|
// architecture_model, and the five synthesis arrays. Each `sources` entry must
|
||||||
|
// be origin-tagged codebase|research (Phase 7 rule 7). Descriptive fields and
|
||||||
|
// unknown top-level keys are tolerated (forward-compat, mirroring
|
||||||
|
// review-validator.mjs / findings-schema.mjs).
|
||||||
|
//
|
||||||
|
// 3-layer pattern (Content → Raw-text → CLI shim) mirroring the other validators.
|
||||||
|
|
||||||
|
import { readFileSync, existsSync } from 'node:fs';
|
||||||
|
import { issue, fail } from '../util/result.mjs';
|
||||||
|
|
||||||
|
// Origin tag for every synthesised finding (Phase 7 rule 7: codebase vs research).
|
||||||
|
export const ORIGIN_VALUES = Object.freeze(['codebase', 'research']);
|
||||||
|
|
||||||
|
// The fields Phase 8 depends on. Descriptive fields (note/severity) are not here
|
||||||
|
// on purpose: their absence should not trigger a re-ask.
|
||||||
|
export const DIGEST_REQUIRED_FIELDS = Object.freeze([
|
||||||
|
'task',
|
||||||
|
'architecture_model',
|
||||||
|
'reusable_code',
|
||||||
|
'contradictions',
|
||||||
|
'risks',
|
||||||
|
'gaps',
|
||||||
|
'sources',
|
||||||
|
]);
|
||||||
|
|
||||||
|
// The five synthesis arrays + their stable not-an-array error codes.
|
||||||
|
const ARRAY_FIELDS = Object.freeze([
|
||||||
|
['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'],
|
||||||
|
['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'],
|
||||||
|
['risks', 'DIGEST_RISKS_NOT_ARRAY'],
|
||||||
|
['gaps', 'DIGEST_GAPS_NOT_ARRAY'],
|
||||||
|
['sources', 'DIGEST_SOURCES_NOT_ARRAY'],
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Last fenced ```json … ``` block, so prose above it never confuses the parser.
|
||||||
|
const JSON_FENCE_GLOBAL = /```json[ \t]*\r?\n([\s\S]*?)```/gi;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract the inner body of the LAST fenced `json` block in `text`.
|
||||||
|
* @param {string} text
|
||||||
|
* @returns {string|null} the JSON source, or null if no json fence is present.
|
||||||
|
*/
|
||||||
|
export function extractDigestBlock(text) {
|
||||||
|
if (typeof text !== 'string') return null;
|
||||||
|
JSON_FENCE_GLOBAL.lastIndex = 0;
|
||||||
|
let last = null;
|
||||||
|
let m;
|
||||||
|
while ((m = JSON_FENCE_GLOBAL.exec(text)) !== null) {
|
||||||
|
last = m[1];
|
||||||
|
}
|
||||||
|
return last;
|
||||||
|
}
|
||||||
|
|
||||||
|
function isNonEmptyString(v) {
|
||||||
|
return typeof v === 'string' && v.length > 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate an already-parsed digest payload against the schema.
|
||||||
|
* Accumulates every error (so a re-ask can name all problems at once).
|
||||||
|
* @param {unknown} payload
|
||||||
|
* @returns {import('../util/result.mjs').Result}
|
||||||
|
*/
|
||||||
|
export function validateDigest(payload) {
|
||||||
|
if (payload === null || typeof payload !== 'object' || Array.isArray(payload)) {
|
||||||
|
return fail(issue(
|
||||||
|
'DIGEST_NOT_OBJECT',
|
||||||
|
`Digest must be a JSON object, got ${Array.isArray(payload) ? 'array' : typeof payload}`,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
const errors = [];
|
||||||
|
const warnings = [];
|
||||||
|
|
||||||
|
if (!isNonEmptyString(payload.agent)) {
|
||||||
|
warnings.push(issue('DIGEST_MISSING_AGENT', 'Digest should carry a non-empty "agent" name'));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!isNonEmptyString(payload.task)) {
|
||||||
|
errors.push(issue('DIGEST_MISSING_TASK', 'Digest "task" must be a non-empty string'));
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!isNonEmptyString(payload.architecture_model)) {
|
||||||
|
errors.push(issue(
|
||||||
|
'DIGEST_MISSING_ARCHITECTURE',
|
||||||
|
'Digest "architecture_model" must be a non-empty string (the synthesised mental model)',
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
for (const [field, code] of ARRAY_FIELDS) {
|
||||||
|
if (!Array.isArray(payload[field])) {
|
||||||
|
errors.push(issue(code, `Digest "${field}" must be an array, got ${typeof payload[field]}`));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Origin-tag check — only when sources actually is an array.
|
||||||
|
if (Array.isArray(payload.sources)) {
|
||||||
|
payload.sources.forEach((s, i) => {
|
||||||
|
const origin = s && typeof s === 'object' ? s.origin : undefined;
|
||||||
|
if (!ORIGIN_VALUES.includes(origin)) {
|
||||||
|
errors.push(issue(
|
||||||
|
'DIGEST_SOURCE_BAD_ORIGIN',
|
||||||
|
`sources[${i}].origin must be one of ${ORIGIN_VALUES.join('|')}, got ${JSON.stringify(origin)}`,
|
||||||
|
'Tag every synthesised finding as codebase or research (Phase 7 rule 7).',
|
||||||
|
`sources[${i}]`,
|
||||||
|
));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return { valid: errors.length === 0, errors, warnings, parsed: payload };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Validate a synthesis-agent's raw output: extract the last json fence, parse it,
|
||||||
|
* then schema-validate. Parse-stage failures get stable codes so they flow
|
||||||
|
* through the same bounded re-ask path as schema failures.
|
||||||
|
* @param {string} rawText
|
||||||
|
* @returns {import('../util/result.mjs').Result}
|
||||||
|
*/
|
||||||
|
export function validateAgentOutput(rawText) {
|
||||||
|
const block = extractDigestBlock(rawText);
|
||||||
|
if (block === null) {
|
||||||
|
return fail(issue(
|
||||||
|
'DIGEST_NO_JSON_BLOCK',
|
||||||
|
'No trailing fenced ```json block found in synthesis-agent output',
|
||||||
|
'The synthesis-agent must end its output with a single ```json digest block.',
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let parsed;
|
||||||
|
try {
|
||||||
|
parsed = JSON.parse(block);
|
||||||
|
} catch (e) {
|
||||||
|
return fail(issue('DIGEST_PARSE_ERROR', `Digest JSON block did not parse: ${e.message}`));
|
||||||
|
}
|
||||||
|
return validateDigest(parsed);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- CLI shim ----------------------------------------------------------------
|
||||||
|
|
||||||
|
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
|
const args = process.argv.slice(2);
|
||||||
|
const filePath = args.find((a) => !a.startsWith('--'));
|
||||||
|
if (!filePath) {
|
||||||
|
process.stderr.write('Usage: synthesis-digest-schema.mjs [--json] <agent-output.txt|.md>\n');
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
if (!existsSync(filePath)) {
|
||||||
|
process.stderr.write(`synthesis-digest-schema: file not found: ${filePath}\n`);
|
||||||
|
process.exit(2);
|
||||||
|
}
|
||||||
|
const r = validateAgentOutput(readFileSync(filePath, 'utf-8'));
|
||||||
|
if (args.includes('--json')) {
|
||||||
|
process.stdout.write(JSON.stringify({ valid: r.valid, errors: r.errors, warnings: r.warnings }, null, 2) + '\n');
|
||||||
|
} else {
|
||||||
|
process.stdout.write(`synthesis-digest-schema: ${r.valid ? 'PASS' : 'FAIL'} ${filePath}\n`);
|
||||||
|
for (const e of r.errors) process.stderr.write(` ERROR [${e.code}] ${e.message}\n`);
|
||||||
|
for (const w of r.warnings) process.stderr.write(` WARN [${w.code}] ${w.message}\n`);
|
||||||
|
}
|
||||||
|
process.exit(r.valid ? 0 : 1);
|
||||||
|
}
|
||||||
297
scripts/synthesis-measure.mjs
Normal file
297
scripts/synthesis-measure.mjs
Normal file
|
|
@ -0,0 +1,297 @@
|
||||||
|
#!/usr/bin/env node
|
||||||
|
// scripts/synthesis-measure.mjs
|
||||||
|
// NW3 (S12) — deterministic Δ main-context measurement for the synthesis-agent.
|
||||||
|
//
|
||||||
|
// The CC-26 gate metric (T1 §2) is Δ main-context tokens for an equivalent
|
||||||
|
// digest. The live ≥3-run bake-off (T1 §5) is the empirically strongest
|
||||||
|
// instrument, but for THIS gate the binding answer is STRUCTURAL, not
|
||||||
|
// stochastic, so a deterministic token-accounting over real fixtures resolves it
|
||||||
|
// reproducibly and without blocked live infra (no API key; the installed plugin
|
||||||
|
// is a cache copy, so a new agent is invisible to `claude -p`). It models two
|
||||||
|
// framings:
|
||||||
|
//
|
||||||
|
// FAITHFUL (current flow): trekplan Phase 5 runs the swarm FOREGROUND, so its
|
||||||
|
// 6–10 outputs are already RESIDENT in main before Phase 7. Delegating only
|
||||||
|
// the synthesis read cannot evict them → main holds base+out+dig in BOTH
|
||||||
|
// arms → Δ ≈ 0. This is what NW3-as-scoped ("main still spawns the swarm;
|
||||||
|
// only the digest is delegated", T1 §6) would actually ship.
|
||||||
|
//
|
||||||
|
// DISK-POTENTIAL (upper bound): IF the swarm wrote outputs to disk and returned
|
||||||
|
// short (a separate Phase-5 change, OUT of NW3 scope), the delegated arm holds
|
||||||
|
// base+dig only → Δ = out/(base+out+dig). BASE-sensitive; swept, not asserted.
|
||||||
|
//
|
||||||
|
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5).
|
||||||
|
//
|
||||||
|
// Zero deps. Node stdlib only. Token figure is an explicit chars/4 estimate; the
|
||||||
|
// RATIO Δ% is what the gate turns on, and the chars/4 constant cancels in the
|
||||||
|
// `out` portion. BASE is environment-dependent (system prompt + plugin listings
|
||||||
|
// + CLAUDE.md) and NOT API-measured this session → swept across a documented band.
|
||||||
|
|
||||||
|
import { readFileSync, readdirSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||||
|
import { join, dirname, resolve } from 'node:path';
|
||||||
|
import { fileURLToPath } from 'node:url';
|
||||||
|
|
||||||
|
const HERE = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const ROOT = resolve(HERE, '..');
|
||||||
|
const DEFAULT_EXPLORATION_DIR = join(ROOT, 'tests/fixtures/synthesis/exploration');
|
||||||
|
const DEFAULT_DIGEST = join(ROOT, 'tests/fixtures/synthesis/digest.json');
|
||||||
|
const DEFAULT_OUT = join(ROOT, 'docs/T1-synthesis-poc-results.md');
|
||||||
|
|
||||||
|
// T1 §5 thresholds.
|
||||||
|
export const POSITIVE_THRESHOLD = 0.30;
|
||||||
|
export const NEGATIVE_THRESHOLD = 0.15;
|
||||||
|
|
||||||
|
// Documented BASE sweep: a Voyage main session's fixed resident baseline (CC
|
||||||
|
// system prompt + tool defs + plugin command/agent/skill listings + CLAUDE.md).
|
||||||
|
// Genuinely environment-dependent; not API-measured this session.
|
||||||
|
export const BASE_SWEEP = Object.freeze([30_000, 50_000, 80_000, 120_000]);
|
||||||
|
|
||||||
|
// Fixture-independent disk-potential sensitivity: sweep OUT at one illustrative
|
||||||
|
// typical baseline, so the verdict does not hinge on this run's fixture sizes.
|
||||||
|
export const REFERENCE_BASE = 60_000;
|
||||||
|
export const OUT_SENSITIVITY = Object.freeze([5_000, 10_000, 20_000, 30_000, 40_000]);
|
||||||
|
|
||||||
|
// ---- pure measurement core (unit-tested) ------------------------------------
|
||||||
|
|
||||||
|
/** Explicit chars/4 token estimate. @param {string} text @returns {number} */
|
||||||
|
export function estimateTokens(text) {
|
||||||
|
if (typeof text !== 'string' || text.length === 0) return 0;
|
||||||
|
return Math.ceil(text.length / 4);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Tokens resident in MAIN at synthesis-complete, per arm.
|
||||||
|
* @param {{base:number, out:number, dig:number, arm:string}} p
|
||||||
|
* @returns {number}
|
||||||
|
*/
|
||||||
|
export function mainContextTokens({ base, out, dig, arm }) {
|
||||||
|
switch (arm) {
|
||||||
|
case 'inline':
|
||||||
|
// main spawns swarm foreground (out resident) + synthesises inline (dig).
|
||||||
|
return base + out + dig;
|
||||||
|
case 'delegated_faithful':
|
||||||
|
// Phase 5 foreground already made `out` resident; the sub-agent's digest
|
||||||
|
// returns on TOP of it. Delegating Phase 7 evicts nothing.
|
||||||
|
return base + out + dig;
|
||||||
|
case 'delegated_disk':
|
||||||
|
// Hypothetical: outputs on disk, never resident in main; only the digest is.
|
||||||
|
return base + dig;
|
||||||
|
default:
|
||||||
|
throw new Error(`mainContextTokens: unknown arm "${arm}"`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fractional reduction (A−B)/A. Divide-by-zero guarded to 0. */
|
||||||
|
export function deltaPct(armA, armB) {
|
||||||
|
if (!armA) return 0;
|
||||||
|
return (armA - armB) / armA;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** T1 §5 verdict. Quality loss vetoes a token win. */
|
||||||
|
export function decideVerdict(delta, qualityOK) {
|
||||||
|
if (!qualityOK) return 'NEGATIVE';
|
||||||
|
if (delta >= POSITIVE_THRESHOLD) return 'POSITIVE';
|
||||||
|
if (delta < NEGATIVE_THRESHOLD) return 'NEGATIVE';
|
||||||
|
return 'INCONCLUSIVE';
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Both framings for one BASE.
|
||||||
|
* @param {{baseTokens:number, outTokens:number, digTokens:number, qualityOK:boolean}} p
|
||||||
|
*/
|
||||||
|
export function analyze({ baseTokens, outTokens, digTokens, qualityOK }) {
|
||||||
|
const armParams = { base: baseTokens, out: outTokens, dig: digTokens };
|
||||||
|
const inline = mainContextTokens({ ...armParams, arm: 'inline' });
|
||||||
|
const faithfulB = mainContextTokens({ ...armParams, arm: 'delegated_faithful' });
|
||||||
|
const diskB = mainContextTokens({ ...armParams, arm: 'delegated_disk' });
|
||||||
|
const fD = deltaPct(inline, faithfulB);
|
||||||
|
const dD = deltaPct(inline, diskB);
|
||||||
|
return {
|
||||||
|
faithful: { armA: inline, armB: faithfulB, deltaPct: fD, verdict: decideVerdict(fD, qualityOK) },
|
||||||
|
disk: { armA: inline, armB: diskB, deltaPct: dD, verdict: decideVerdict(dD, qualityOK) },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** BASE at which disk-potential Δ crosses exactly POSITIVE_THRESHOLD. */
|
||||||
|
export function breakEvenBase(outTokens, digTokens, threshold = POSITIVE_THRESHOLD) {
|
||||||
|
// out/(base+out+dig) = threshold → base = out/threshold - out - dig
|
||||||
|
return Math.round(outTokens / threshold - outTokens - digTokens);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- CLI shim ----------------------------------------------------------------
|
||||||
|
|
||||||
|
function pct(x) { return `${(x * 100).toFixed(1)}%`; }
|
||||||
|
|
||||||
|
function loadExploration(dir) {
|
||||||
|
const files = readdirSync(dir)
|
||||||
|
.filter((f) => f.endsWith('.md') || f.endsWith('.txt'))
|
||||||
|
.sort();
|
||||||
|
return files.map((f) => {
|
||||||
|
const text = readFileSync(join(dir, f), 'utf-8');
|
||||||
|
return { name: f, chars: text.length, tokens: estimateTokens(text) };
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
function buildResultsDoc({ explorationDir, digestPath, outputs, outTokens, digTokens, qualityOK }) {
|
||||||
|
const breakEven = breakEvenBase(outTokens, digTokens);
|
||||||
|
const L = [];
|
||||||
|
L.push('# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)');
|
||||||
|
L.push('');
|
||||||
|
L.push('**Status:** Measurement complete — verdict below. **Method:** deterministic token-');
|
||||||
|
L.push('accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the');
|
||||||
|
L.push('stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the');
|
||||||
|
L.push('installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;');
|
||||||
|
L.push('and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).');
|
||||||
|
L.push('**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).');
|
||||||
|
L.push('**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).');
|
||||||
|
L.push('');
|
||||||
|
L.push('> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a');
|
||||||
|
L.push('> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels');
|
||||||
|
L.push('> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and');
|
||||||
|
L.push('> was NOT API-measured this session → swept across a documented band, not asserted.');
|
||||||
|
L.push('');
|
||||||
|
L.push('## 1. The decisive structural finding (BASE-independent)');
|
||||||
|
L.push('');
|
||||||
|
L.push('trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode');
|
||||||
|
L.push('since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back');
|
||||||
|
L.push('into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already');
|
||||||
|
L.push('resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the');
|
||||||
|
L.push('"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore');
|
||||||
|
L.push('*reasons over already-resident context*. Delegating **only** the Phase-7 read to a');
|
||||||
|
L.push('synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —');
|
||||||
|
L.push('**cannot evict those outputs from main**; the digest simply returns on top of them.');
|
||||||
|
L.push('');
|
||||||
|
L.push('⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The');
|
||||||
|
L.push('≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-');
|
||||||
|
L.push('disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7');
|
||||||
|
L.push('explicitly declined and is OUT of NW3 scope.');
|
||||||
|
L.push('');
|
||||||
|
L.push('## 2. Fixtures (measured)');
|
||||||
|
L.push('');
|
||||||
|
L.push(`- Exploration dir: \`${explorationDir.replace(ROOT + '/', '')}\``);
|
||||||
|
L.push(`- Digest: \`${digestPath.replace(ROOT + '/', '')}\``);
|
||||||
|
L.push('');
|
||||||
|
L.push('| exploration output | chars | est. tokens |');
|
||||||
|
L.push('|--------------------|-------|-------------|');
|
||||||
|
for (const o of outputs) L.push(`| ${o.name} | ${o.chars} | ${o.tokens} |`);
|
||||||
|
L.push(`| **OUT (Σ resident in main)** | — | **${outTokens}** |`);
|
||||||
|
L.push(`| digest (DIG) | — | ${digTokens} |`);
|
||||||
|
L.push('');
|
||||||
|
L.push('## 3. Δ main-context — both framings, swept over BASE');
|
||||||
|
L.push('');
|
||||||
|
L.push('`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·');
|
||||||
|
L.push('`delegated (disk-potential)` = base+dig (out off-main).');
|
||||||
|
L.push('');
|
||||||
|
L.push('| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |');
|
||||||
|
L.push('|-------------|--------|------------|------------------|------------------|--------------|');
|
||||||
|
for (const base of BASE_SWEEP) {
|
||||||
|
const a = analyze({ baseTokens: base, outTokens, digTokens, qualityOK });
|
||||||
|
L.push(
|
||||||
|
`| ${base} | ${a.faithful.armA} | ${pct(a.faithful.deltaPct)} | ${a.faithful.verdict} ` +
|
||||||
|
`| ${pct(a.disk.deltaPct)} | ${a.disk.verdict} |`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
L.push('');
|
||||||
|
L.push(`Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: ` +
|
||||||
|
`**~${breakEven.toLocaleString('en-US')} tokens** (below this BASE the *hypothetical* disk path ` +
|
||||||
|
`would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE ` +
|
||||||
|
`(CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk ` +
|
||||||
|
`upper bound is itself fragile.`);
|
||||||
|
L.push('');
|
||||||
|
L.push('### Fixture-independent break-even (so the verdict does not hinge on fixture size)');
|
||||||
|
L.push('');
|
||||||
|
L.push('disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**');
|
||||||
|
L.push('`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the');
|
||||||
|
L.push('fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = ' +
|
||||||
|
`${REFERENCE_BASE.toLocaleString('en-US')}\` (independent of this run's fixtures):`);
|
||||||
|
L.push('');
|
||||||
|
L.push('| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |');
|
||||||
|
L.push('|----------------------------|----------------------------|-------------|');
|
||||||
|
for (const out of OUT_SENSITIVITY) {
|
||||||
|
const a = analyze({ baseTokens: REFERENCE_BASE, outTokens: out, digTokens: digTokens, qualityOK });
|
||||||
|
L.push(`| ${out.toLocaleString('en-US')} | ${pct(a.disk.deltaPct)} | ${a.disk.deltaPct >= POSITIVE_THRESHOLD ? 'yes' : 'no'} |`);
|
||||||
|
}
|
||||||
|
L.push('');
|
||||||
|
L.push(`This run's fixtures total **OUT = ${outTokens} tokens** across ${outputs.length} concise ` +
|
||||||
|
`representative outputs — one concrete point on the curve. Even a generously large real swarm ` +
|
||||||
|
`(OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and ` +
|
||||||
|
`*never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.`);
|
||||||
|
L.push('');
|
||||||
|
L.push('## 4. Quality');
|
||||||
|
L.push('');
|
||||||
|
L.push('The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7');
|
||||||
|
L.push('synthesis dimensions main produces inline (task, architecture_model, reusable_code,');
|
||||||
|
L.push('contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is');
|
||||||
|
L.push('structurally quality-equivalent to the inline one — but quality is moot here: the faithful');
|
||||||
|
L.push('Δ is ~0, so there is no token win for quality to defend.');
|
||||||
|
L.push('');
|
||||||
|
L.push('## 5. Verdict');
|
||||||
|
L.push('');
|
||||||
|
const faithfulVerdict = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK }).faithful;
|
||||||
|
L.push('**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,');
|
||||||
|
L.push('structural — the Phase-5 foreground swarm already makes the outputs resident; delegating');
|
||||||
|
L.push('Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-');
|
||||||
|
L.push('scope Phase-5 change and is itself BASE-fragile.');
|
||||||
|
L.push('');
|
||||||
|
L.push(`RESULT: NEGATIVE (Δ_faithful = ${pct(faithfulVerdict.deltaPct)} < ${pct(NEGATIVE_THRESHOLD)} adopt-floor)`);
|
||||||
|
L.push('');
|
||||||
|
L.push('## 6. Disposition');
|
||||||
|
L.push('');
|
||||||
|
L.push('- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);');
|
||||||
|
L.push(' `commands/trekplan.md` Phase 7 is **NOT** wired to it.');
|
||||||
|
L.push('- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-');
|
||||||
|
L.push(' writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).');
|
||||||
|
L.push('- The dormant agent + this harness make that future step cheap to re-measure: drop new');
|
||||||
|
L.push(' fixtures in and re-run.');
|
||||||
|
L.push('');
|
||||||
|
return L.join('\n') + '\n';
|
||||||
|
}
|
||||||
|
|
||||||
|
function parseArgs(argv) {
|
||||||
|
const o = { explorationDir: DEFAULT_EXPLORATION_DIR, digest: DEFAULT_DIGEST, out: DEFAULT_OUT, qualityOK: true, json: false };
|
||||||
|
for (let i = 0; i < argv.length; i++) {
|
||||||
|
const a = argv[i];
|
||||||
|
if (a === '--exploration') o.explorationDir = resolve(argv[++i]);
|
||||||
|
else if (a === '--digest') o.digest = resolve(argv[++i]);
|
||||||
|
else if (a === '--out') o.out = resolve(argv[++i]);
|
||||||
|
else if (a === '--quality-fail') o.qualityOK = false;
|
||||||
|
else if (a === '--json') o.json = true;
|
||||||
|
else if (a === '--help' || a === '-h') { o.help = true; }
|
||||||
|
else { process.stderr.write(`Unknown argument: ${a}\n`); process.exit(2); }
|
||||||
|
}
|
||||||
|
return o;
|
||||||
|
}
|
||||||
|
|
||||||
|
function mainCli() {
|
||||||
|
const o = parseArgs(process.argv.slice(2));
|
||||||
|
if (o.help) {
|
||||||
|
process.stdout.write('Usage: synthesis-measure.mjs [--exploration DIR] [--digest FILE] [--out FILE] [--quality-fail] [--json]\n');
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
if (!existsSync(o.explorationDir)) { process.stderr.write(`exploration dir not found: ${o.explorationDir}\n`); process.exit(2); }
|
||||||
|
if (!existsSync(o.digest)) { process.stderr.write(`digest not found: ${o.digest}\n`); process.exit(2); }
|
||||||
|
|
||||||
|
const outputs = loadExploration(o.explorationDir);
|
||||||
|
const outTokens = outputs.reduce((s, x) => s + x.tokens, 0);
|
||||||
|
const digTokens = estimateTokens(readFileSync(o.digest, 'utf-8'));
|
||||||
|
|
||||||
|
if (o.json) {
|
||||||
|
const rows = BASE_SWEEP.map((base) => ({ base, ...analyze({ baseTokens: base, outTokens, digTokens, qualityOK: o.qualityOK }) }));
|
||||||
|
process.stdout.write(JSON.stringify({ outTokens, digTokens, breakEvenBase: breakEvenBase(outTokens, digTokens), rows }, null, 2) + '\n');
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
const doc = buildResultsDoc({
|
||||||
|
explorationDir: o.explorationDir, digestPath: o.digest,
|
||||||
|
outputs, outTokens, digTokens, qualityOK: o.qualityOK,
|
||||||
|
});
|
||||||
|
if (!existsSync(dirname(o.out))) mkdirSync(dirname(o.out), { recursive: true });
|
||||||
|
writeFileSync(o.out, doc, 'utf-8');
|
||||||
|
const faithful = analyze({ baseTokens: BASE_SWEEP[1], outTokens, digTokens, qualityOK: o.qualityOK }).faithful;
|
||||||
|
process.stderr.write(`[synthesis-measure] OUT=${outTokens} DIG=${digTokens} tok · faithful Δ=${pct(faithful.deltaPct)} → ${faithful.verdict} · wrote ${o.out}\n`);
|
||||||
|
process.exit(0);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (import.meta.url === `file://${process.argv[1]}`) {
|
||||||
|
mainCli();
|
||||||
|
}
|
||||||
29
tests/fixtures/synthesis/digest.json
vendored
Normal file
29
tests/fixtures/synthesis/digest.json
vendored
Normal file
|
|
@ -0,0 +1,29 @@
|
||||||
|
{
|
||||||
|
"agent": "synthesis-agent",
|
||||||
|
"task": "Add a per-wave concurrency cap to trekexecute headless launches",
|
||||||
|
"architecture_model": "trekexecute's headless path (Phase 2.6) turns each independent wave from session-decomposer's dependency graph into a launch batch, and templates/headless-launch-template.md backgrounds every batch member as a parallel `claude -p` subprocess with no slot limit — degree of parallelism equals batch size. The fix straddles a JS/Bash boundary: a pure, testable max-parallel resolver in lib/ (reusing the arg-parser + profile-resolver lookup order) feeds one validated integer into the template's fan-out, which should consume it via `xargs -P` rather than a hand-rolled Bash semaphore.",
|
||||||
|
"reusable_code": [
|
||||||
|
{ "ref": "lib/parsers/arg-parser.mjs", "note": "parse --max-parallel <n>; do not hand-roll" },
|
||||||
|
{ "ref": "lib/profiles/profile-resolver.mjs", "note": "lookup order flag→signal→profile→default for the cap" },
|
||||||
|
{ "ref": "lib/util/result.mjs:33", "note": "issue() error shape for invalid cap values" },
|
||||||
|
{ "ref": "templates/headless-launch-template.md:10", "note": "the fan-out wire-in site" }
|
||||||
|
],
|
||||||
|
"contradictions": [
|
||||||
|
"architecture-mapper frames the cap as launch-time Bash-only; task-finder + convention-scanner argue the arithmetic must live in a pure lib resolver — reconciled: resolver computes, template consumes one integer"
|
||||||
|
],
|
||||||
|
"risks": [
|
||||||
|
{ "risk": "Too-low cap serializes independent waves, erasing the parallelism trekexecute exists for", "severity": "high" },
|
||||||
|
{ "risk": "Hand-rolled Bash semaphore can deadlock `wait` on subprocess crash — prefer xargs -P", "severity": "high" },
|
||||||
|
{ "risk": "Cap of 0/negative could stall the pipeline; clamp to >=1 with a stable error code", "severity": "medium" }
|
||||||
|
],
|
||||||
|
"gaps": [
|
||||||
|
"No measured per-wave token/process ceiling exists — the safe default (cap = batch size, i.e. no-op) is an assumption until profiled",
|
||||||
|
"Whether xargs -P is available/identical across the operator's macOS + any headless CI is unverified"
|
||||||
|
],
|
||||||
|
"sources": [
|
||||||
|
{ "finding": "launcher backgrounds all members then waits, no slot limit", "origin": "codebase" },
|
||||||
|
{ "finding": "project already serializes spawns in q3 harness to dodge spawn-burst limits", "origin": "codebase" },
|
||||||
|
{ "finding": "profile system (v4.1) is the canonical knob lookup order", "origin": "codebase" },
|
||||||
|
{ "finding": "xargs -P release-on-exit semantics vs hand-rolled semaphore", "origin": "research" }
|
||||||
|
]
|
||||||
|
}
|
||||||
96
tests/fixtures/synthesis/exploration/architecture-mapper.md
vendored
Normal file
96
tests/fixtures/synthesis/exploration/architecture-mapper.md
vendored
Normal file
|
|
@ -0,0 +1,96 @@
|
||||||
|
# Architecture Report — output of architecture-mapper
|
||||||
|
|
||||||
|
Task being planned: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Voyage is a contract-driven Claude Code plugin (Node ESM, zero runtime deps).
|
||||||
|
The pipeline is six commands (`commands/*.md`) backed by a `lib/` of pure,
|
||||||
|
unit-tested validators/parsers and a thin `scripts/` layer of measurement and
|
||||||
|
codegen harnesses. trekexecute is the disciplined plan/session-spec executor;
|
||||||
|
its headless path (Phase 2.6) fans out parallel "waves" of `claude -p`
|
||||||
|
subprocesses driven by a Bash launcher template.
|
||||||
|
|
||||||
|
## Tech stack
|
||||||
|
|
||||||
|
| Layer | Choice | Evidence |
|
||||||
|
|-------|--------|----------|
|
||||||
|
| Language | JavaScript (ESM, `.mjs`) | `lib/**/*.mjs`, `"type":"module"` in package.json |
|
||||||
|
| Tests | `node:test` + `node:assert/strict` | every `tests/**/*.test.mjs` |
|
||||||
|
| Validation | hand-rolled validators returning `{valid,errors,warnings}` | `lib/util/result.mjs` |
|
||||||
|
| Orchestration substrate | command prose + Bash + Agent/Task tool | `commands/trekexecute.md` |
|
||||||
|
| Headless launch | Bash here-doc template, backgrounded subprocesses | `templates/headless-launch-template.md` |
|
||||||
|
|
||||||
|
## Key patterns
|
||||||
|
|
||||||
|
- **3-layer module pattern** — Content validator → raw-text wrapper → CLI shim
|
||||||
|
(`if (import.meta.url === \`file://${process.argv[1]}\`)`), repeated across
|
||||||
|
`lib/validators`, `lib/parsers`, `lib/review`. Any new lib must follow it.
|
||||||
|
- **Structured Result type** — `issue(code,message,hint,location)` + `fail()` /
|
||||||
|
`ok()` from `lib/util/result.mjs`. Stable error codes are the contract.
|
||||||
|
- **Prose-as-orchestrator** — commands carry the control flow in markdown; the
|
||||||
|
harness executes it. Schema-drift defenses are *inlined* into command prose so
|
||||||
|
they survive even when agent docs are not implicitly loaded.
|
||||||
|
|
||||||
|
## Anti-patterns / debt near the task
|
||||||
|
|
||||||
|
- The headless launcher backgrounds **all** wave members at once with no upper
|
||||||
|
bound on concurrent `claude -p` processes; concurrency is implicit in how many
|
||||||
|
steps a wave contains. No central place caps it.
|
||||||
|
- Wave composition (which steps go in which wave) is computed by
|
||||||
|
session-decomposer, but the *launch* fan-out is template Bash, so a cap would
|
||||||
|
straddle a JS (decomposer) / Bash (launcher) boundary.
|
||||||
|
|
||||||
|
## Module map (task-relevant)
|
||||||
|
|
||||||
|
```
|
||||||
|
commands/trekexecute.md # Phase 2.6 parallel-wave orchestration prose
|
||||||
|
templates/headless-launch-template.md # the Bash fan-out site
|
||||||
|
lib/util/result.mjs # error shape any new guard returns
|
||||||
|
agents/session-decomposer.md # produces the wave/dependency graph
|
||||||
|
```
|
||||||
|
|
||||||
|
## Boundaries
|
||||||
|
|
||||||
|
The cap is a launch-time concern (Bash template + the Phase 2.6 prose that
|
||||||
|
generates it). It does not belong in the pure `lib/` validators unless we add a
|
||||||
|
small "max parallelism" resolver that the prose reads. Recommend a lib resolver
|
||||||
|
(testable) + a template wire-in (the actual `xargs -P` / job-slot mechanism).
|
||||||
|
|
||||||
|
## How Phase 2.6 assembles a wave (detail)
|
||||||
|
|
||||||
|
The executor reads the plan's `## Step N` blocks and the dependency edges
|
||||||
|
session-decomposer emitted (`depends_on:` frontmatter). Steps with no unmet
|
||||||
|
dependency at the current frontier form a wave. For each wave the prose:
|
||||||
|
|
||||||
|
1. builds a `SHARED_CONTEXT_FILE` (brief + plan + relevant exploration digest)
|
||||||
|
passed to every member via `--append-system-prompt-file` (cache-prefix
|
||||||
|
material — see q3 experiment);
|
||||||
|
2. emits one `claude -p … &` invocation per member from the here-doc, each with
|
||||||
|
`--max-turns`, `--max-budget-usd`, `GIT_OPTIONAL_LOCKS=0`, and the GH#36071
|
||||||
|
push-before-cleanup workaround;
|
||||||
|
3. collects the backgrounded PIDs and `wait`s for the batch to drain before
|
||||||
|
advancing the frontier.
|
||||||
|
|
||||||
|
The cap belongs strictly between (2) and (3): bound how many of the emitted
|
||||||
|
members run concurrently, leaving wave *composition* (1) untouched.
|
||||||
|
|
||||||
|
## Layering verdict
|
||||||
|
|
||||||
|
Three layers, in increasing blast radius: (a) a pure resolver in `lib/`
|
||||||
|
(arithmetic only — trivially testable, follows the 3-layer module pattern);
|
||||||
|
(b) Phase 2.6 prose passing the resolved integer + a `--max-parallel` flag into
|
||||||
|
the template; (c) the template's fan-out mechanism. Keep (a) the single source
|
||||||
|
of the number; (c) should only consume it. This matches every prior launcher
|
||||||
|
hardening, which added one bounded externality (budget, locks, turns) at a time
|
||||||
|
without reshaping wave composition.
|
||||||
|
|
||||||
|
## Cross-cutting observations
|
||||||
|
|
||||||
|
- The launcher is the most operationally sensitive file in the repo (it spends
|
||||||
|
money and mutates git). Every edit here is co-reviewed with its
|
||||||
|
doc-consistency needle list — treat the needle test as part of the contract,
|
||||||
|
not an afterthought.
|
||||||
|
- Nothing in `lib/` currently imports anything launcher-related; the resolver
|
||||||
|
will be a leaf module. Good — it can be tested and shipped independently of the
|
||||||
|
template wire-in, enabling a TDD-first slice.
|
||||||
42
tests/fixtures/synthesis/exploration/convention-scanner.md
vendored
Normal file
42
tests/fixtures/synthesis/exploration/convention-scanner.md
vendored
Normal file
|
|
@ -0,0 +1,42 @@
|
||||||
|
# Conventions Report — output of convention-scanner
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Summary
|
||||||
|
|
||||||
|
Mature, consistent conventions. New code should imitate the existing `lib/`
|
||||||
|
validators almost mechanically.
|
||||||
|
|
||||||
|
## Naming
|
||||||
|
|
||||||
|
| Element | Convention | Example |
|
||||||
|
|---------|-----------|---------|
|
||||||
|
| Files | kebab-case `.mjs` | `lib/profiles/profile-resolver.mjs` |
|
||||||
|
| Functions | camelCase, verb-first | `validateFindings`, `extractDigestBlock` |
|
||||||
|
| Error codes | UPPER_SNAKE, domain-prefixed | `FINDING_BAD_LINE`, `DIGEST_NOT_OBJECT` |
|
||||||
|
| Tests | `<module>.test.mjs` mirroring `lib/` tree | `tests/lib/profile-resolver.test.mjs` |
|
||||||
|
|
||||||
|
## Module shape (must follow)
|
||||||
|
|
||||||
|
Every `lib/` module is the **3-layer pattern**:
|
||||||
|
1. exported pure functions (validate/resolve/compute);
|
||||||
|
2. they return `{valid, errors, warnings, parsed}` via `lib/util/result.mjs`
|
||||||
|
`issue()`/`fail()`/`ok()`;
|
||||||
|
3. a CLI shim `if (import.meta.url === \`file://${process.argv[1]}\`)` for Bash.
|
||||||
|
|
||||||
|
A `max-parallel-resolver.mjs` must replicate this exactly (pure resolver +
|
||||||
|
`issue()` errors + CLI shim).
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- Never coerce bad input — return a `{valid:false}` Result with a stable code and
|
||||||
|
a `hint`. Throwing is reserved for genuine programmer error (see
|
||||||
|
`mainContextTokens` "unknown arm" throw).
|
||||||
|
- Unknown/extra fields are tolerated (forward-compat), load-bearing fields are
|
||||||
|
hard errors. Mirror this: clamp/validate the cap, tolerate extra profile keys.
|
||||||
|
|
||||||
|
## Imports / tests / commits
|
||||||
|
|
||||||
|
- Named ESM imports, relative paths, no path aliases, no barrels.
|
||||||
|
- Conventional Commits, `type(scope): description`, e.g. `feat(voyage): …`.
|
||||||
|
- New behavior is TDD'd: failing `node:test` first, then minimal code.
|
||||||
76
tests/fixtures/synthesis/exploration/dependency-tracer.md
vendored
Normal file
76
tests/fixtures/synthesis/exploration/dependency-tracer.md
vendored
Normal file
|
|
@ -0,0 +1,76 @@
|
||||||
|
# Dependency & Data-Flow Report — output of dependency-tracer
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Import / call chain relevant to the task
|
||||||
|
|
||||||
|
```
|
||||||
|
commands/trekexecute.md (Phase 2.6 prose)
|
||||||
|
└─ generates → templates/headless-launch-template.md (Bash here-doc)
|
||||||
|
├─ reads SHARED_CONTEXT_FILE (append-system-prompt-file)
|
||||||
|
├─ spawns claude -p ×N (one per wave member, backgrounded with &)
|
||||||
|
└─ waits via `wait` on collected PIDs
|
||||||
|
└─ consumes → .session-state.local.json (Handover 7; session graph)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Data flow
|
||||||
|
|
||||||
|
1. session-decomposer emits a plan with wave groupings + a dependency graph.
|
||||||
|
2. trekexecute Phase 2.6 turns each independent wave into a launch batch.
|
||||||
|
3. The template loops over batch members and backgrounds each `claude -p`,
|
||||||
|
collecting PIDs into a Bash array, then `wait`s for the whole batch.
|
||||||
|
4. There is **no slot-limiting** between "background member" and "wait" — the
|
||||||
|
degree of parallelism equals the batch size.
|
||||||
|
|
||||||
|
## Side effects
|
||||||
|
|
||||||
|
- Each subprocess does `git` work under `GIT_OPTIONAL_LOCKS=0`; high concurrency
|
||||||
|
raises the chance of index-lock contention (mitigated, not eliminated).
|
||||||
|
- `--max-budget-usd` is per-subprocess; total spend scales with batch size, so a
|
||||||
|
concurrency cap also indirectly bounds burst spend.
|
||||||
|
|
||||||
|
## What a cap touches
|
||||||
|
|
||||||
|
- **Pure-addable:** a `maxParallel` resolver in `lib/` (reads plan/profile/flag,
|
||||||
|
returns an integer ≥ 1). No existing module imports would change.
|
||||||
|
- **Wire-in:** the template's loop must consume slots (e.g. a counting semaphore
|
||||||
|
in Bash, or `xargs -P <n>`). This is the only behavioral edit.
|
||||||
|
|
||||||
|
## No hidden dependents
|
||||||
|
|
||||||
|
Grepped for other call sites of the launch template — only trekexecute Phase 2.6
|
||||||
|
and the headless-launch-template test reference it. A cap is local in blast
|
||||||
|
radius.
|
||||||
|
|
||||||
|
## Resolver input provenance (what the cap reads)
|
||||||
|
|
||||||
|
The resolved integer must be derived from, in lookup order:
|
||||||
|
|
||||||
|
1. **CLI flag** `--max-parallel <n>` — parsed by `lib/parsers/arg-parser.mjs`;
|
||||||
|
highest precedence (operator override).
|
||||||
|
2. **Brief signal** — `phase_signals` already carries per-phase orchestration
|
||||||
|
shape; a `max_parallel` hint here is honoured if no flag.
|
||||||
|
3. **Profile** — `lib/profiles/` resolves `--profile economy|balanced|premium`;
|
||||||
|
each profile can carry a `max_parallel` default. This is the same lookup
|
||||||
|
order `phase_models` uses, so the resolver should *reuse* profile-resolver,
|
||||||
|
not re-implement precedence.
|
||||||
|
4. **Hard default** — `batchSize` (i.e. no cap / current behavior), so the change
|
||||||
|
is a strict no-op until someone opts in.
|
||||||
|
|
||||||
|
## Downstream of the cap
|
||||||
|
|
||||||
|
- **Budget:** total burst spend = `min(cap, batchSize) × per-member --max-budget-usd`.
|
||||||
|
A cap therefore tightens the worst-case spend envelope — worth noting in the
|
||||||
|
plan's risk/observability section.
|
||||||
|
- **Git contention:** fewer concurrent `git`-touching subprocesses → fewer
|
||||||
|
`index.lock` races. `GIT_OPTIONAL_LOCKS=0` reduces lock acquisition but does
|
||||||
|
not serialize ref updates; the cap is the structural mitigation.
|
||||||
|
- **Classifier exposure:** a smaller concurrent fan-out under `auto`/`bypass`
|
||||||
|
lowers the surface the proliferation classifier (S7 F4) scrutinises.
|
||||||
|
|
||||||
|
## Data-flow invariant to preserve
|
||||||
|
|
||||||
|
`SHARED_CONTEXT_FILE` is built once per wave and read by every member; the cap
|
||||||
|
must not cause it to be rebuilt per-slot (would defeat the cache prefix). Slot
|
||||||
|
limiting happens at spawn time only; the context file is wave-scoped, not
|
||||||
|
slot-scoped.
|
||||||
31
tests/fixtures/synthesis/exploration/git-historian.md
vendored
Normal file
31
tests/fixtures/synthesis/exploration/git-historian.md
vendored
Normal file
|
|
@ -0,0 +1,31 @@
|
||||||
|
# Git History — output of git-historian
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Recent changes touching the launch path
|
||||||
|
|
||||||
|
| Commit (illustrative) | Area | Relevance |
|
||||||
|
|-----------------------|------|-----------|
|
||||||
|
| Phase 2.6 hardening series | trekexecute.md + headless-launch-template.md | Added `GIT_OPTIONAL_LOCKS`, `--max-budget-usd`, push-before-cleanup, GH#36071 workaround. The launcher is actively maintained and recently hardened — a cap is the next natural hardening. |
|
||||||
|
| session-decomposer wiring | agents/session-decomposer.md | Established the wave/dependency graph the launcher consumes. Stable; not the edit site. |
|
||||||
|
| profile system (v4.1) | lib/profiles/ | Introduced `phase_models` + `--profile`; the lookup order a `max_parallel` knob should reuse. |
|
||||||
|
|
||||||
|
## Ownership / hot files
|
||||||
|
|
||||||
|
- `templates/headless-launch-template.md` is a **hot file** — multiple recent
|
||||||
|
hardening commits. Expect a strict doc-consistency needle list; any edit must
|
||||||
|
keep all existing needles AND add the new one.
|
||||||
|
- `commands/trekexecute.md` Phase 2.6 is co-edited with the template in every
|
||||||
|
hardening commit ("template mirrors Phase 2.6"). Keep them in lockstep.
|
||||||
|
|
||||||
|
## Active branches / risk of conflict
|
||||||
|
|
||||||
|
Single active branch (`main`); polyrepo, frequent small commits. Low conflict
|
||||||
|
risk. The kjøremodus is one-task-per-session, so this change should be a single
|
||||||
|
focused commit touching resolver + template + prose + tests.
|
||||||
|
|
||||||
|
## Signal
|
||||||
|
|
||||||
|
The project's own history shows a consistent preference: bound risky externalities
|
||||||
|
explicitly (budget, locks, turns). An unbounded fan-out is the conspicuous gap in
|
||||||
|
that pattern — the change is in-character with how this launcher has evolved.
|
||||||
25
tests/fixtures/synthesis/exploration/risk-assessor.md
vendored
Normal file
25
tests/fixtures/synthesis/exploration/risk-assessor.md
vendored
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
# Risk & Failure-Mode Report — output of risk-assessor
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Risks (ranked)
|
||||||
|
|
||||||
|
| # | Risk | Severity | Mitigation |
|
||||||
|
|---|------|----------|------------|
|
||||||
|
| R1 | A cap that is too low **serializes** independent waves, erasing the parallelism trekexecute exists to provide. | high | Default the cap to the batch size (no-op) and only clamp when the operator/profile asks; never silently throttle. |
|
||||||
|
| R2 | Bash semaphore bugs **deadlock** the `wait` (slots never released on subprocess crash). | high | Prefer `xargs -P <n>` over a hand-rolled counting semaphore — release-on-exit is built in. Guard with a per-subprocess `--max-turns`/timeout already present. |
|
||||||
|
| R3 | The cap interacts with the **proliferation classifier** (S7 F4): a large parallel fan-out under `auto`/`bypass` is already scrutinised. A cap *reduces* this exposure, but mis-set to 0/negative could stall the pipeline. | medium | Clamp to `≥ 1`; reject `0`/negatives with a stable error code (`issue()`), do not coerce. |
|
||||||
|
| R4 | Index-lock contention under high concurrency (multiple `git` subprocesses) is *masked* today by luck; raising the cap re-exposes it. | medium | Document that `GIT_OPTIONAL_LOCKS=0` is necessary-not-sufficient; the cap is the real fix. |
|
||||||
|
| R5 | Drift between the JS resolver default and the Bash template's actual `-P` value. | low | Single source: prose passes the resolved integer into the template; a test asserts the template consumes `$MAX_PARALLEL`. |
|
||||||
|
|
||||||
|
## Edge cases
|
||||||
|
|
||||||
|
- Wave of size 1 → cap is irrelevant (no fan-out).
|
||||||
|
- Cap ≥ batch size → must be a pure no-op (R1).
|
||||||
|
- Non-integer / missing flag → fall through resolver to profile default, not crash.
|
||||||
|
|
||||||
|
## Complexity hotspots
|
||||||
|
|
||||||
|
The straddle between JS (resolver, testable) and Bash (launcher, hard to unit
|
||||||
|
test) is the main hazard. Keep ALL arithmetic in the resolver; the template
|
||||||
|
should only consume one already-validated integer.
|
||||||
36
tests/fixtures/synthesis/exploration/task-finder.md
vendored
Normal file
36
tests/fixtures/synthesis/exploration/task-finder.md
vendored
Normal file
|
|
@ -0,0 +1,36 @@
|
||||||
|
# Task-Relevant Code — output of task-finder
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Direct hits
|
||||||
|
|
||||||
|
| File | Lines | Why relevant |
|
||||||
|
|------|-------|--------------|
|
||||||
|
| `templates/headless-launch-template.md` | ~10–60 | The Bash fan-out: backgrounds each wave member, `wait`s on PIDs. Edit site for slot-limiting. |
|
||||||
|
| `commands/trekexecute.md` | Phase 2.6 | Prose that generates the launch batches; where a `--max-parallel` flag + profile lookup would be documented. |
|
||||||
|
| `lib/profiles/` | resolver dir | Profiles already carry per-phase knobs; a `max_parallel` default fits the existing `phase_signals`/profile lookup order. |
|
||||||
|
| `lib/util/result.mjs` | 33 | `issue()` — the error shape a `maxParallel` resolver returns on bad input. |
|
||||||
|
|
||||||
|
## Reuse candidates
|
||||||
|
|
||||||
|
- **arg-parser** (`lib/parsers/arg-parser.mjs`, per tests/lib/arg-parser.test.mjs)
|
||||||
|
already parses `--flag value` pairs for the commands. A `--max-parallel <n>`
|
||||||
|
flag plugs into the existing parser; do not hand-roll parsing.
|
||||||
|
- **profile-resolver** (`lib/profiles/`, profile-resolver.test.mjs) is the lookup
|
||||||
|
order CLI-flag → brief signal → profile → default. A concurrency default
|
||||||
|
belongs as a profile field consumed through this same resolver.
|
||||||
|
- **autonomy-gate** (`lib/` autonomy-gate.test.mjs) shows the established pattern
|
||||||
|
for "bounded integer with a safe default" — mirror its clamp/validate.
|
||||||
|
|
||||||
|
## Existing similar solutions
|
||||||
|
|
||||||
|
`scripts/q3-cache-prefix-experiment.mjs` spawns children **sequentially** to
|
||||||
|
avoid spawn-burst rate-limits — confirms the project already knows unbounded
|
||||||
|
fan-out is a risk and chose a manual bound there. A cap generalises that
|
||||||
|
instinct to the headless wave path.
|
||||||
|
|
||||||
|
## Models / config
|
||||||
|
|
||||||
|
No DB. Config lives in `settings.json` (`trekplan`/`trekresearch` scopes only —
|
||||||
|
doc-consistency pins this) and per-run profiles. A `max_parallel` knob should
|
||||||
|
ride the profile system, not add a new settings.json scope.
|
||||||
38
tests/fixtures/synthesis/exploration/test-strategist.md
vendored
Normal file
38
tests/fixtures/synthesis/exploration/test-strategist.md
vendored
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
# Test Strategy — output of test-strategist
|
||||||
|
|
||||||
|
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||||
|
|
||||||
|
## Existing patterns
|
||||||
|
|
||||||
|
- Framework: `node:test` + `node:assert/strict`. One file per module under
|
||||||
|
`tests/lib/` (mirrors `lib/`) and `tests/scripts/` for harness cores.
|
||||||
|
- Validators are tested as **pure functions**: feed a payload, assert
|
||||||
|
`{valid,errors}` + stable error codes (see findings-schema.test.mjs,
|
||||||
|
autonomy-gate.test.mjs).
|
||||||
|
- Template/prose invariants are pinned by **doc-consistency.test.mjs** (string
|
||||||
|
`assert.ok(text.includes(...))` against templates/commands).
|
||||||
|
|
||||||
|
## Coverage gaps for this task
|
||||||
|
|
||||||
|
1. No test asserts an upper bound on launch parallelism today (there is none).
|
||||||
|
2. headless-launch-template.md is pinned for a list of required needles
|
||||||
|
(`GIT_OPTIONAL_LOCKS`, `--max-turns`, `--max-budget-usd`, …) but NOT for any
|
||||||
|
concurrency mechanism.
|
||||||
|
|
||||||
|
## Recommended tests (TDD order)
|
||||||
|
|
||||||
|
1. **`lib/.../max-parallel-resolver.test.mjs`** (new) — pure resolver:
|
||||||
|
- flag `--max-parallel 3` wins over profile/default;
|
||||||
|
- missing flag → profile default → hard default;
|
||||||
|
- `0` / negative / non-integer → `{valid:false}` with a stable code;
|
||||||
|
- cap ≥ batchSize → returns batchSize (no-op clamp).
|
||||||
|
2. **doc-consistency extension** — assert headless-launch-template.md consumes
|
||||||
|
the resolved integer (e.g. includes `$MAX_PARALLEL` or `xargs -P`).
|
||||||
|
3. **arg-parser** — assert `--max-parallel` is recognised (extend
|
||||||
|
arg-parser.test.mjs / gates-flag-coverage pattern).
|
||||||
|
|
||||||
|
## Notes
|
||||||
|
|
||||||
|
The Bash fan-out itself is not unit-testable in `node:test`; rely on the
|
||||||
|
template-needle pin (#2) + keeping arithmetic in the JS resolver (#1). This
|
||||||
|
matches how the project already tests prose-driven behavior.
|
||||||
161
tests/lib/synthesis-digest-schema.test.mjs
Normal file
161
tests/lib/synthesis-digest-schema.test.mjs
Normal file
|
|
@ -0,0 +1,161 @@
|
||||||
|
// tests/lib/synthesis-digest-schema.test.mjs
|
||||||
|
// NW3 (S12) — digest-schema contract for the synthesis-agent.
|
||||||
|
//
|
||||||
|
// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7
|
||||||
|
// exploration outputs and emits a trailing fenced ```json block: the findings
|
||||||
|
// DIGEST main currently writes inline in Phase 7. This pins that digest's shape
|
||||||
|
// so a delegated path could VALIDATE it (not merely parse it) and so the
|
||||||
|
// measurement harness has a fixed quality contract to compare against.
|
||||||
|
//
|
||||||
|
// Load-bearing fields (what Phase 8 deep-planning consumes): task,
|
||||||
|
// architecture_model, and the five synthesis arrays (reusable_code,
|
||||||
|
// contradictions, risks, gaps, sources). Each source must be origin-tagged
|
||||||
|
// codebase|research (Phase 7 rule 7: "track whether it came from codebase
|
||||||
|
// analysis or external research"). Mirrors lib/review/findings-schema.mjs.
|
||||||
|
//
|
||||||
|
// When this test fails, fix the schema or the producer — do NOT relax the
|
||||||
|
// assertion to hide drift.
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import {
|
||||||
|
validateDigest,
|
||||||
|
validateAgentOutput,
|
||||||
|
extractDigestBlock,
|
||||||
|
ORIGIN_VALUES,
|
||||||
|
DIGEST_REQUIRED_FIELDS,
|
||||||
|
} from '../../lib/plan/synthesis-digest-schema.mjs';
|
||||||
|
|
||||||
|
function wellFormed() {
|
||||||
|
return {
|
||||||
|
agent: 'synthesis-agent',
|
||||||
|
task: 'Add a per-wave rate-limit guard to trekexecute headless launches',
|
||||||
|
architecture_model:
|
||||||
|
'trekexecute spawns parallel headless waves via the Bash launcher template; ' +
|
||||||
|
'concurrency is currently unbounded per wave.',
|
||||||
|
reusable_code: [
|
||||||
|
{ ref: 'lib/util/result.mjs:33', note: 'issue() for structured errors' },
|
||||||
|
{ ref: 'templates/headless-launch-template.md:10', note: 'wave dispatch site' },
|
||||||
|
],
|
||||||
|
contradictions: [
|
||||||
|
'architecture-mapper says waves are sequential; dependency-tracer shows a parallel fan-out',
|
||||||
|
],
|
||||||
|
risks: [
|
||||||
|
{ risk: 'A rate-limit that blocks too aggressively starves long waves', severity: 'medium' },
|
||||||
|
],
|
||||||
|
gaps: ['No measured per-wave token ceiling exists yet — becomes an assumption'],
|
||||||
|
sources: [
|
||||||
|
{ finding: 'wave launcher lives in the template', origin: 'codebase' },
|
||||||
|
{ finding: 'CC headless --max-budget-usd semantics', origin: 'research' },
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('digest schema: a well-formed digest validates', () => {
|
||||||
|
const r = validateDigest(wellFormed());
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
assert.deepEqual(r.errors, []);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: exposes the required-field + origin contracts', () => {
|
||||||
|
assert.ok(Array.isArray(DIGEST_REQUIRED_FIELDS) && DIGEST_REQUIRED_FIELDS.length > 0);
|
||||||
|
for (const f of ['task', 'architecture_model', 'reusable_code', 'contradictions', 'risks', 'gaps', 'sources']) {
|
||||||
|
assert.ok(DIGEST_REQUIRED_FIELDS.includes(f), `required fields must include ${f}`);
|
||||||
|
}
|
||||||
|
assert.deepEqual([...ORIGIN_VALUES].sort(), ['codebase', 'research']);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: rejects a non-object payload', () => {
|
||||||
|
for (const bad of [null, 42, 'x', ['a']]) {
|
||||||
|
const r = validateDigest(bad);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NOT_OBJECT'), `expected DIGEST_NOT_OBJECT for ${JSON.stringify(bad)}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: rejects missing/empty task', () => {
|
||||||
|
for (const t of [undefined, '', 123]) {
|
||||||
|
const d = wellFormed();
|
||||||
|
d.task = t;
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_TASK'));
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: rejects missing/empty architecture_model', () => {
|
||||||
|
const d = wellFormed();
|
||||||
|
delete d.architecture_model;
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_ARCHITECTURE'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: each synthesis array must be an array', () => {
|
||||||
|
const cases = [
|
||||||
|
['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'],
|
||||||
|
['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'],
|
||||||
|
['risks', 'DIGEST_RISKS_NOT_ARRAY'],
|
||||||
|
['gaps', 'DIGEST_GAPS_NOT_ARRAY'],
|
||||||
|
['sources', 'DIGEST_SOURCES_NOT_ARRAY'],
|
||||||
|
];
|
||||||
|
for (const [field, code] of cases) {
|
||||||
|
const d = wellFormed();
|
||||||
|
d[field] = { not: 'an array' };
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, false, `${field} as object should fail`);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === code), `expected ${code}`);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: empty synthesis arrays are valid (a clean digest can have no contradictions/gaps)', () => {
|
||||||
|
const d = wellFormed();
|
||||||
|
d.contradictions = [];
|
||||||
|
d.gaps = [];
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: a source with a non-enum origin is rejected', () => {
|
||||||
|
const d = wellFormed();
|
||||||
|
d.sources = [{ finding: 'x', origin: 'guess' }];
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_SOURCE_BAD_ORIGIN'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('digest schema: missing agent name is a warning, not an error', () => {
|
||||||
|
const d = wellFormed();
|
||||||
|
delete d.agent;
|
||||||
|
const r = validateDigest(d);
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
assert.ok(r.warnings.some((w) => w.code === 'DIGEST_MISSING_AGENT'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('extractDigestBlock: pulls the LAST fenced json block from agent prose', () => {
|
||||||
|
const text =
|
||||||
|
'Here is my synthesis.\n\n' +
|
||||||
|
'```json\n{"stale": true}\n```\n\n' +
|
||||||
|
'Actually, the final digest:\n\n' +
|
||||||
|
'```json\n' + JSON.stringify(wellFormed()) + '\n```\n';
|
||||||
|
const block = extractDigestBlock(text);
|
||||||
|
assert.ok(block && JSON.parse(block).task, 'should extract the last json block');
|
||||||
|
assert.equal(JSON.parse(block).agent, 'synthesis-agent');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateAgentOutput: no json fence → stable NO_JSON code', () => {
|
||||||
|
const r = validateAgentOutput('just prose, no fence');
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NO_JSON_BLOCK'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateAgentOutput: malformed json → stable PARSE code', () => {
|
||||||
|
const r = validateAgentOutput('```json\n{ not valid }\n```');
|
||||||
|
assert.equal(r.valid, false);
|
||||||
|
assert.ok(r.errors.some((e) => e.code === 'DIGEST_PARSE_ERROR'));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('validateAgentOutput: well-formed fenced digest validates end-to-end', () => {
|
||||||
|
const r = validateAgentOutput('Synthesis complete.\n```json\n' + JSON.stringify(wellFormed()) + '\n```');
|
||||||
|
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||||
|
});
|
||||||
101
tests/scripts/synthesis-measure.test.mjs
Normal file
101
tests/scripts/synthesis-measure.test.mjs
Normal file
|
|
@ -0,0 +1,101 @@
|
||||||
|
// tests/scripts/synthesis-measure.test.mjs
|
||||||
|
// NW3 (S12) — deterministic Δ main-context measurement core.
|
||||||
|
//
|
||||||
|
// The gate metric (T1 §2) is Δ main-context tokens for an equivalent-quality
|
||||||
|
// digest. This pins the pure accounting that turns fixture token counts into a
|
||||||
|
// verdict, under the two framings that decide NW3:
|
||||||
|
//
|
||||||
|
// - FAITHFUL (current flow): Phase 5 swarm runs FOREGROUND, so its outputs are
|
||||||
|
// already RESIDENT in main before Phase 7. Delegating only the synthesis read
|
||||||
|
// cannot evict them → main holds base+out+dig in BOTH arms → Δ ≈ 0.
|
||||||
|
// - DISK-POTENTIAL (upper bound): IF outputs were on disk (a separate Phase-5
|
||||||
|
// change, out of NW3 scope), the delegated arm holds base+dig only → Δ = out/(base+out+dig).
|
||||||
|
//
|
||||||
|
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5 thresholds).
|
||||||
|
|
||||||
|
import { test } from 'node:test';
|
||||||
|
import { strict as assert } from 'node:assert';
|
||||||
|
import {
|
||||||
|
estimateTokens,
|
||||||
|
mainContextTokens,
|
||||||
|
deltaPct,
|
||||||
|
decideVerdict,
|
||||||
|
analyze,
|
||||||
|
} from '../../scripts/synthesis-measure.mjs';
|
||||||
|
|
||||||
|
test('estimateTokens: ~chars/4 heuristic, monotonic, non-negative', () => {
|
||||||
|
assert.equal(estimateTokens(''), 0);
|
||||||
|
assert.equal(estimateTokens('abcd'), 1);
|
||||||
|
assert.equal(estimateTokens('abcde'), 2); // ceil(5/4)
|
||||||
|
assert.ok(estimateTokens('a'.repeat(400)) === 100);
|
||||||
|
assert.ok(estimateTokens('x'.repeat(1000)) > estimateTokens('x'.repeat(500)));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mainContextTokens: inline arm holds base + out + dig', () => {
|
||||||
|
assert.equal(mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'inline' }), 63500);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mainContextTokens: delegated_faithful still holds out (Phase-5 resident) + dig', () => {
|
||||||
|
// The decisive structural fact: foreground swarm delivery already made `out`
|
||||||
|
// resident; delegating Phase 7 does not evict it.
|
||||||
|
assert.equal(
|
||||||
|
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_faithful' }),
|
||||||
|
63500,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mainContextTokens: delegated_disk holds base + dig only (out lives in the sub-agent)', () => {
|
||||||
|
assert.equal(
|
||||||
|
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_disk' }),
|
||||||
|
51500,
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('mainContextTokens: unknown arm throws (no silent default)', () => {
|
||||||
|
assert.throws(() => mainContextTokens({ base: 1, out: 1, dig: 1, arm: 'nope' }));
|
||||||
|
});
|
||||||
|
|
||||||
|
test('deltaPct: (A-B)/A; equal arms → 0; B smaller → positive fraction', () => {
|
||||||
|
assert.equal(deltaPct(100, 100), 0);
|
||||||
|
assert.equal(deltaPct(100, 75), 0.25);
|
||||||
|
assert.equal(deltaPct(0, 0), 0); // guard divide-by-zero
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: ≥30% AND quality-ok → POSITIVE', () => {
|
||||||
|
assert.equal(decideVerdict(0.30, true), 'POSITIVE');
|
||||||
|
assert.equal(decideVerdict(0.45, true), 'POSITIVE');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: quality loss forces NEGATIVE even at a large Δ', () => {
|
||||||
|
assert.equal(decideVerdict(0.60, false), 'NEGATIVE');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('decideVerdict: <15% → NEGATIVE; the [15%,30%) band → INCONCLUSIVE', () => {
|
||||||
|
assert.equal(decideVerdict(0.149, true), 'NEGATIVE');
|
||||||
|
assert.equal(decideVerdict(0.00, true), 'NEGATIVE');
|
||||||
|
assert.equal(decideVerdict(0.15, true), 'INCONCLUSIVE');
|
||||||
|
assert.equal(decideVerdict(0.2999, true), 'INCONCLUSIVE');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('analyze: faithful arm is structurally Δ=0 → NEGATIVE regardless of sizes', () => {
|
||||||
|
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||||
|
assert.equal(a.faithful.deltaPct, 0);
|
||||||
|
assert.equal(a.faithful.verdict, 'NEGATIVE');
|
||||||
|
assert.equal(a.faithful.armA, a.faithful.armB);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('analyze: disk arm Δ = out/(base+out+dig)', () => {
|
||||||
|
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||||
|
const expected = 12000 / (50000 + 12000 + 1500);
|
||||||
|
assert.ok(Math.abs(a.disk.deltaPct - expected) < 1e-9);
|
||||||
|
assert.equal(a.disk.armB, 51500);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('analyze: disk verdict tracks BASE — large fixed baseline can sink the upper bound below 30%', () => {
|
||||||
|
// Small base → disk Δ clears 30%; large base → it does not. Demonstrates the
|
||||||
|
// upper bound is itself BASE-sensitive (sweep, do not assert one number).
|
||||||
|
const small = analyze({ baseTokens: 10000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||||
|
const large = analyze({ baseTokens: 200000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||||
|
assert.equal(small.disk.verdict, 'POSITIVE'); // 12000/23500 ≈ 0.51
|
||||||
|
assert.equal(large.disk.verdict, 'NEGATIVE'); // 12000/213500 ≈ 0.056
|
||||||
|
});
|
||||||
Loading…
Add table
Add a link
Reference in a new issue