feat(voyage): S12 — NW3 synthesis-agent built + measured → declined per measurement [skip-docs]

NW3 (CC-26 §6 PoC): delegate trekplan Phase 7 synthesis to a synthesis-agent,
adopt only if Δ main-context ≥30% with no quality loss. Operator chose the
deterministic-proof path (live ≥3-run bake-off was env-blocked: no API key;
installed plugin is a cache copy so a new agent is invisible to `claude -p`).

Decisive structural finding: trekplan Phase 5 runs the swarm FOREGROUND, so its
outputs are already resident in main before Phase 7. Delegating only Phase 7
evicts nothing → Δ_faithful = 0% (BASE-independent). The ≥30% saving needs an
out-of-scope Phase-5 redesign (swarm-writes-to-disk / nested orchestrator).
VERDICT: DECLINED per measurement.

- agents/synthesis-agent.md — dormant, schema-conformant deliverable (NOT wired)
- lib/plan/synthesis-digest-schema.mjs — digest output contract (+ tests)
- scripts/synthesis-measure.mjs — deterministic Δ-accounting core (+ tests)
- tests/fixtures/synthesis/ — 7 exploration outputs + representative digest
- docs/T1-synthesis-poc-results.md — measurement + verdict (reproducible)
- CLAUDE.md — agent table row (doc-consistency: 24 agents)

Tests 670 → 695 (693 pass / 2 skip / 0 fail). `claude plugin validate` clean
(only the pre-existing root-CLAUDE.md warning). commands/trekplan.md untouched.

[skip-docs] rationale: no user-facing feature ships (NW3 declined; agent dormant
and unwired). The substantive doc is docs/T1-synthesis-poc-results.md; the
README/CHANGELOG roll-up for NW1–NW3 is the S13 coordinated release per
docs/W1-narrow-wins-plan.md §S13.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 17:58:39 +02:00
commit 6b30483304
15 changed files with 1312 additions and 0 deletions

29
tests/fixtures/synthesis/digest.json vendored Normal file
View file

@ -0,0 +1,29 @@
{
"agent": "synthesis-agent",
"task": "Add a per-wave concurrency cap to trekexecute headless launches",
"architecture_model": "trekexecute's headless path (Phase 2.6) turns each independent wave from session-decomposer's dependency graph into a launch batch, and templates/headless-launch-template.md backgrounds every batch member as a parallel `claude -p` subprocess with no slot limit — degree of parallelism equals batch size. The fix straddles a JS/Bash boundary: a pure, testable max-parallel resolver in lib/ (reusing the arg-parser + profile-resolver lookup order) feeds one validated integer into the template's fan-out, which should consume it via `xargs -P` rather than a hand-rolled Bash semaphore.",
"reusable_code": [
{ "ref": "lib/parsers/arg-parser.mjs", "note": "parse --max-parallel <n>; do not hand-roll" },
{ "ref": "lib/profiles/profile-resolver.mjs", "note": "lookup order flag→signal→profile→default for the cap" },
{ "ref": "lib/util/result.mjs:33", "note": "issue() error shape for invalid cap values" },
{ "ref": "templates/headless-launch-template.md:10", "note": "the fan-out wire-in site" }
],
"contradictions": [
"architecture-mapper frames the cap as launch-time Bash-only; task-finder + convention-scanner argue the arithmetic must live in a pure lib resolver — reconciled: resolver computes, template consumes one integer"
],
"risks": [
{ "risk": "Too-low cap serializes independent waves, erasing the parallelism trekexecute exists for", "severity": "high" },
{ "risk": "Hand-rolled Bash semaphore can deadlock `wait` on subprocess crash — prefer xargs -P", "severity": "high" },
{ "risk": "Cap of 0/negative could stall the pipeline; clamp to >=1 with a stable error code", "severity": "medium" }
],
"gaps": [
"No measured per-wave token/process ceiling exists — the safe default (cap = batch size, i.e. no-op) is an assumption until profiled",
"Whether xargs -P is available/identical across the operator's macOS + any headless CI is unverified"
],
"sources": [
{ "finding": "launcher backgrounds all members then waits, no slot limit", "origin": "codebase" },
{ "finding": "project already serializes spawns in q3 harness to dodge spawn-burst limits", "origin": "codebase" },
{ "finding": "profile system (v4.1) is the canonical knob lookup order", "origin": "codebase" },
{ "finding": "xargs -P release-on-exit semantics vs hand-rolled semaphore", "origin": "research" }
]
}

View file

@ -0,0 +1,96 @@
# Architecture Report — output of architecture-mapper
Task being planned: Add a per-wave concurrency cap to trekexecute headless launches.
## Summary
Voyage is a contract-driven Claude Code plugin (Node ESM, zero runtime deps).
The pipeline is six commands (`commands/*.md`) backed by a `lib/` of pure,
unit-tested validators/parsers and a thin `scripts/` layer of measurement and
codegen harnesses. trekexecute is the disciplined plan/session-spec executor;
its headless path (Phase 2.6) fans out parallel "waves" of `claude -p`
subprocesses driven by a Bash launcher template.
## Tech stack
| Layer | Choice | Evidence |
|-------|--------|----------|
| Language | JavaScript (ESM, `.mjs`) | `lib/**/*.mjs`, `"type":"module"` in package.json |
| Tests | `node:test` + `node:assert/strict` | every `tests/**/*.test.mjs` |
| Validation | hand-rolled validators returning `{valid,errors,warnings}` | `lib/util/result.mjs` |
| Orchestration substrate | command prose + Bash + Agent/Task tool | `commands/trekexecute.md` |
| Headless launch | Bash here-doc template, backgrounded subprocesses | `templates/headless-launch-template.md` |
## Key patterns
- **3-layer module pattern** — Content validator → raw-text wrapper → CLI shim
(`if (import.meta.url === \`file://${process.argv[1]}\`)`), repeated across
`lib/validators`, `lib/parsers`, `lib/review`. Any new lib must follow it.
- **Structured Result type**`issue(code,message,hint,location)` + `fail()` /
`ok()` from `lib/util/result.mjs`. Stable error codes are the contract.
- **Prose-as-orchestrator** — commands carry the control flow in markdown; the
harness executes it. Schema-drift defenses are *inlined* into command prose so
they survive even when agent docs are not implicitly loaded.
## Anti-patterns / debt near the task
- The headless launcher backgrounds **all** wave members at once with no upper
bound on concurrent `claude -p` processes; concurrency is implicit in how many
steps a wave contains. No central place caps it.
- Wave composition (which steps go in which wave) is computed by
session-decomposer, but the *launch* fan-out is template Bash, so a cap would
straddle a JS (decomposer) / Bash (launcher) boundary.
## Module map (task-relevant)
```
commands/trekexecute.md # Phase 2.6 parallel-wave orchestration prose
templates/headless-launch-template.md # the Bash fan-out site
lib/util/result.mjs # error shape any new guard returns
agents/session-decomposer.md # produces the wave/dependency graph
```
## Boundaries
The cap is a launch-time concern (Bash template + the Phase 2.6 prose that
generates it). It does not belong in the pure `lib/` validators unless we add a
small "max parallelism" resolver that the prose reads. Recommend a lib resolver
(testable) + a template wire-in (the actual `xargs -P` / job-slot mechanism).
## How Phase 2.6 assembles a wave (detail)
The executor reads the plan's `## Step N` blocks and the dependency edges
session-decomposer emitted (`depends_on:` frontmatter). Steps with no unmet
dependency at the current frontier form a wave. For each wave the prose:
1. builds a `SHARED_CONTEXT_FILE` (brief + plan + relevant exploration digest)
passed to every member via `--append-system-prompt-file` (cache-prefix
material — see q3 experiment);
2. emits one `claude -p … &` invocation per member from the here-doc, each with
`--max-turns`, `--max-budget-usd`, `GIT_OPTIONAL_LOCKS=0`, and the GH#36071
push-before-cleanup workaround;
3. collects the backgrounded PIDs and `wait`s for the batch to drain before
advancing the frontier.
The cap belongs strictly between (2) and (3): bound how many of the emitted
members run concurrently, leaving wave *composition* (1) untouched.
## Layering verdict
Three layers, in increasing blast radius: (a) a pure resolver in `lib/`
(arithmetic only — trivially testable, follows the 3-layer module pattern);
(b) Phase 2.6 prose passing the resolved integer + a `--max-parallel` flag into
the template; (c) the template's fan-out mechanism. Keep (a) the single source
of the number; (c) should only consume it. This matches every prior launcher
hardening, which added one bounded externality (budget, locks, turns) at a time
without reshaping wave composition.
## Cross-cutting observations
- The launcher is the most operationally sensitive file in the repo (it spends
money and mutates git). Every edit here is co-reviewed with its
doc-consistency needle list — treat the needle test as part of the contract,
not an afterthought.
- Nothing in `lib/` currently imports anything launcher-related; the resolver
will be a leaf module. Good — it can be tested and shipped independently of the
template wire-in, enabling a TDD-first slice.

View file

@ -0,0 +1,42 @@
# Conventions Report — output of convention-scanner
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Summary
Mature, consistent conventions. New code should imitate the existing `lib/`
validators almost mechanically.
## Naming
| Element | Convention | Example |
|---------|-----------|---------|
| Files | kebab-case `.mjs` | `lib/profiles/profile-resolver.mjs` |
| Functions | camelCase, verb-first | `validateFindings`, `extractDigestBlock` |
| Error codes | UPPER_SNAKE, domain-prefixed | `FINDING_BAD_LINE`, `DIGEST_NOT_OBJECT` |
| Tests | `<module>.test.mjs` mirroring `lib/` tree | `tests/lib/profile-resolver.test.mjs` |
## Module shape (must follow)
Every `lib/` module is the **3-layer pattern**:
1. exported pure functions (validate/resolve/compute);
2. they return `{valid, errors, warnings, parsed}` via `lib/util/result.mjs`
`issue()`/`fail()`/`ok()`;
3. a CLI shim `if (import.meta.url === \`file://${process.argv[1]}\`)` for Bash.
A `max-parallel-resolver.mjs` must replicate this exactly (pure resolver +
`issue()` errors + CLI shim).
## Error handling
- Never coerce bad input — return a `{valid:false}` Result with a stable code and
a `hint`. Throwing is reserved for genuine programmer error (see
`mainContextTokens` "unknown arm" throw).
- Unknown/extra fields are tolerated (forward-compat), load-bearing fields are
hard errors. Mirror this: clamp/validate the cap, tolerate extra profile keys.
## Imports / tests / commits
- Named ESM imports, relative paths, no path aliases, no barrels.
- Conventional Commits, `type(scope): description`, e.g. `feat(voyage): …`.
- New behavior is TDD'd: failing `node:test` first, then minimal code.

View file

@ -0,0 +1,76 @@
# Dependency & Data-Flow Report — output of dependency-tracer
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Import / call chain relevant to the task
```
commands/trekexecute.md (Phase 2.6 prose)
└─ generates → templates/headless-launch-template.md (Bash here-doc)
├─ reads SHARED_CONTEXT_FILE (append-system-prompt-file)
├─ spawns claude -p ×N (one per wave member, backgrounded with &)
└─ waits via `wait` on collected PIDs
└─ consumes → .session-state.local.json (Handover 7; session graph)
```
## Data flow
1. session-decomposer emits a plan with wave groupings + a dependency graph.
2. trekexecute Phase 2.6 turns each independent wave into a launch batch.
3. The template loops over batch members and backgrounds each `claude -p`,
collecting PIDs into a Bash array, then `wait`s for the whole batch.
4. There is **no slot-limiting** between "background member" and "wait" — the
degree of parallelism equals the batch size.
## Side effects
- Each subprocess does `git` work under `GIT_OPTIONAL_LOCKS=0`; high concurrency
raises the chance of index-lock contention (mitigated, not eliminated).
- `--max-budget-usd` is per-subprocess; total spend scales with batch size, so a
concurrency cap also indirectly bounds burst spend.
## What a cap touches
- **Pure-addable:** a `maxParallel` resolver in `lib/` (reads plan/profile/flag,
returns an integer ≥ 1). No existing module imports would change.
- **Wire-in:** the template's loop must consume slots (e.g. a counting semaphore
in Bash, or `xargs -P <n>`). This is the only behavioral edit.
## No hidden dependents
Grepped for other call sites of the launch template — only trekexecute Phase 2.6
and the headless-launch-template test reference it. A cap is local in blast
radius.
## Resolver input provenance (what the cap reads)
The resolved integer must be derived from, in lookup order:
1. **CLI flag** `--max-parallel <n>` — parsed by `lib/parsers/arg-parser.mjs`;
highest precedence (operator override).
2. **Brief signal**`phase_signals` already carries per-phase orchestration
shape; a `max_parallel` hint here is honoured if no flag.
3. **Profile**`lib/profiles/` resolves `--profile economy|balanced|premium`;
each profile can carry a `max_parallel` default. This is the same lookup
order `phase_models` uses, so the resolver should *reuse* profile-resolver,
not re-implement precedence.
4. **Hard default**`batchSize` (i.e. no cap / current behavior), so the change
is a strict no-op until someone opts in.
## Downstream of the cap
- **Budget:** total burst spend = `min(cap, batchSize) × per-member --max-budget-usd`.
A cap therefore tightens the worst-case spend envelope — worth noting in the
plan's risk/observability section.
- **Git contention:** fewer concurrent `git`-touching subprocesses → fewer
`index.lock` races. `GIT_OPTIONAL_LOCKS=0` reduces lock acquisition but does
not serialize ref updates; the cap is the structural mitigation.
- **Classifier exposure:** a smaller concurrent fan-out under `auto`/`bypass`
lowers the surface the proliferation classifier (S7 F4) scrutinises.
## Data-flow invariant to preserve
`SHARED_CONTEXT_FILE` is built once per wave and read by every member; the cap
must not cause it to be rebuilt per-slot (would defeat the cache prefix). Slot
limiting happens at spawn time only; the context file is wave-scoped, not
slot-scoped.

View file

@ -0,0 +1,31 @@
# Git History — output of git-historian
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Recent changes touching the launch path
| Commit (illustrative) | Area | Relevance |
|-----------------------|------|-----------|
| Phase 2.6 hardening series | trekexecute.md + headless-launch-template.md | Added `GIT_OPTIONAL_LOCKS`, `--max-budget-usd`, push-before-cleanup, GH#36071 workaround. The launcher is actively maintained and recently hardened — a cap is the next natural hardening. |
| session-decomposer wiring | agents/session-decomposer.md | Established the wave/dependency graph the launcher consumes. Stable; not the edit site. |
| profile system (v4.1) | lib/profiles/ | Introduced `phase_models` + `--profile`; the lookup order a `max_parallel` knob should reuse. |
## Ownership / hot files
- `templates/headless-launch-template.md` is a **hot file** — multiple recent
hardening commits. Expect a strict doc-consistency needle list; any edit must
keep all existing needles AND add the new one.
- `commands/trekexecute.md` Phase 2.6 is co-edited with the template in every
hardening commit ("template mirrors Phase 2.6"). Keep them in lockstep.
## Active branches / risk of conflict
Single active branch (`main`); polyrepo, frequent small commits. Low conflict
risk. The kjøremodus is one-task-per-session, so this change should be a single
focused commit touching resolver + template + prose + tests.
## Signal
The project's own history shows a consistent preference: bound risky externalities
explicitly (budget, locks, turns). An unbounded fan-out is the conspicuous gap in
that pattern — the change is in-character with how this launcher has evolved.

View file

@ -0,0 +1,25 @@
# Risk & Failure-Mode Report — output of risk-assessor
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Risks (ranked)
| # | Risk | Severity | Mitigation |
|---|------|----------|------------|
| R1 | A cap that is too low **serializes** independent waves, erasing the parallelism trekexecute exists to provide. | high | Default the cap to the batch size (no-op) and only clamp when the operator/profile asks; never silently throttle. |
| R2 | Bash semaphore bugs **deadlock** the `wait` (slots never released on subprocess crash). | high | Prefer `xargs -P <n>` over a hand-rolled counting semaphore — release-on-exit is built in. Guard with a per-subprocess `--max-turns`/timeout already present. |
| R3 | The cap interacts with the **proliferation classifier** (S7 F4): a large parallel fan-out under `auto`/`bypass` is already scrutinised. A cap *reduces* this exposure, but mis-set to 0/negative could stall the pipeline. | medium | Clamp to `≥ 1`; reject `0`/negatives with a stable error code (`issue()`), do not coerce. |
| R4 | Index-lock contention under high concurrency (multiple `git` subprocesses) is *masked* today by luck; raising the cap re-exposes it. | medium | Document that `GIT_OPTIONAL_LOCKS=0` is necessary-not-sufficient; the cap is the real fix. |
| R5 | Drift between the JS resolver default and the Bash template's actual `-P` value. | low | Single source: prose passes the resolved integer into the template; a test asserts the template consumes `$MAX_PARALLEL`. |
## Edge cases
- Wave of size 1 → cap is irrelevant (no fan-out).
- Cap ≥ batch size → must be a pure no-op (R1).
- Non-integer / missing flag → fall through resolver to profile default, not crash.
## Complexity hotspots
The straddle between JS (resolver, testable) and Bash (launcher, hard to unit
test) is the main hazard. Keep ALL arithmetic in the resolver; the template
should only consume one already-validated integer.

View file

@ -0,0 +1,36 @@
# Task-Relevant Code — output of task-finder
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Direct hits
| File | Lines | Why relevant |
|------|-------|--------------|
| `templates/headless-launch-template.md` | ~1060 | The Bash fan-out: backgrounds each wave member, `wait`s on PIDs. Edit site for slot-limiting. |
| `commands/trekexecute.md` | Phase 2.6 | Prose that generates the launch batches; where a `--max-parallel` flag + profile lookup would be documented. |
| `lib/profiles/` | resolver dir | Profiles already carry per-phase knobs; a `max_parallel` default fits the existing `phase_signals`/profile lookup order. |
| `lib/util/result.mjs` | 33 | `issue()` — the error shape a `maxParallel` resolver returns on bad input. |
## Reuse candidates
- **arg-parser** (`lib/parsers/arg-parser.mjs`, per tests/lib/arg-parser.test.mjs)
already parses `--flag value` pairs for the commands. A `--max-parallel <n>`
flag plugs into the existing parser; do not hand-roll parsing.
- **profile-resolver** (`lib/profiles/`, profile-resolver.test.mjs) is the lookup
order CLI-flag → brief signal → profile → default. A concurrency default
belongs as a profile field consumed through this same resolver.
- **autonomy-gate** (`lib/` autonomy-gate.test.mjs) shows the established pattern
for "bounded integer with a safe default" — mirror its clamp/validate.
## Existing similar solutions
`scripts/q3-cache-prefix-experiment.mjs` spawns children **sequentially** to
avoid spawn-burst rate-limits — confirms the project already knows unbounded
fan-out is a risk and chose a manual bound there. A cap generalises that
instinct to the headless wave path.
## Models / config
No DB. Config lives in `settings.json` (`trekplan`/`trekresearch` scopes only —
doc-consistency pins this) and per-run profiles. A `max_parallel` knob should
ride the profile system, not add a new settings.json scope.

View file

@ -0,0 +1,38 @@
# Test Strategy — output of test-strategist
Task: Add a per-wave concurrency cap to trekexecute headless launches.
## Existing patterns
- Framework: `node:test` + `node:assert/strict`. One file per module under
`tests/lib/` (mirrors `lib/`) and `tests/scripts/` for harness cores.
- Validators are tested as **pure functions**: feed a payload, assert
`{valid,errors}` + stable error codes (see findings-schema.test.mjs,
autonomy-gate.test.mjs).
- Template/prose invariants are pinned by **doc-consistency.test.mjs** (string
`assert.ok(text.includes(...))` against templates/commands).
## Coverage gaps for this task
1. No test asserts an upper bound on launch parallelism today (there is none).
2. headless-launch-template.md is pinned for a list of required needles
(`GIT_OPTIONAL_LOCKS`, `--max-turns`, `--max-budget-usd`, …) but NOT for any
concurrency mechanism.
## Recommended tests (TDD order)
1. **`lib/.../max-parallel-resolver.test.mjs`** (new) — pure resolver:
- flag `--max-parallel 3` wins over profile/default;
- missing flag → profile default → hard default;
- `0` / negative / non-integer → `{valid:false}` with a stable code;
- cap ≥ batchSize → returns batchSize (no-op clamp).
2. **doc-consistency extension** — assert headless-launch-template.md consumes
the resolved integer (e.g. includes `$MAX_PARALLEL` or `xargs -P`).
3. **arg-parser** — assert `--max-parallel` is recognised (extend
arg-parser.test.mjs / gates-flag-coverage pattern).
## Notes
The Bash fan-out itself is not unit-testable in `node:test`; rely on the
template-needle pin (#2) + keeping arithmetic in the JS resolver (#1). This
matches how the project already tests prose-driven behavior.

View file

@ -0,0 +1,161 @@
// tests/lib/synthesis-digest-schema.test.mjs
// NW3 (S12) — digest-schema contract for the synthesis-agent.
//
// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7
// exploration outputs and emits a trailing fenced ```json block: the findings
// DIGEST main currently writes inline in Phase 7. This pins that digest's shape
// so a delegated path could VALIDATE it (not merely parse it) and so the
// measurement harness has a fixed quality contract to compare against.
//
// Load-bearing fields (what Phase 8 deep-planning consumes): task,
// architecture_model, and the five synthesis arrays (reusable_code,
// contradictions, risks, gaps, sources). Each source must be origin-tagged
// codebase|research (Phase 7 rule 7: "track whether it came from codebase
// analysis or external research"). Mirrors lib/review/findings-schema.mjs.
//
// When this test fails, fix the schema or the producer — do NOT relax the
// assertion to hide drift.
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import {
validateDigest,
validateAgentOutput,
extractDigestBlock,
ORIGIN_VALUES,
DIGEST_REQUIRED_FIELDS,
} from '../../lib/plan/synthesis-digest-schema.mjs';
function wellFormed() {
return {
agent: 'synthesis-agent',
task: 'Add a per-wave rate-limit guard to trekexecute headless launches',
architecture_model:
'trekexecute spawns parallel headless waves via the Bash launcher template; ' +
'concurrency is currently unbounded per wave.',
reusable_code: [
{ ref: 'lib/util/result.mjs:33', note: 'issue() for structured errors' },
{ ref: 'templates/headless-launch-template.md:10', note: 'wave dispatch site' },
],
contradictions: [
'architecture-mapper says waves are sequential; dependency-tracer shows a parallel fan-out',
],
risks: [
{ risk: 'A rate-limit that blocks too aggressively starves long waves', severity: 'medium' },
],
gaps: ['No measured per-wave token ceiling exists yet — becomes an assumption'],
sources: [
{ finding: 'wave launcher lives in the template', origin: 'codebase' },
{ finding: 'CC headless --max-budget-usd semantics', origin: 'research' },
],
};
}
test('digest schema: a well-formed digest validates', () => {
const r = validateDigest(wellFormed());
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.deepEqual(r.errors, []);
});
test('digest schema: exposes the required-field + origin contracts', () => {
assert.ok(Array.isArray(DIGEST_REQUIRED_FIELDS) && DIGEST_REQUIRED_FIELDS.length > 0);
for (const f of ['task', 'architecture_model', 'reusable_code', 'contradictions', 'risks', 'gaps', 'sources']) {
assert.ok(DIGEST_REQUIRED_FIELDS.includes(f), `required fields must include ${f}`);
}
assert.deepEqual([...ORIGIN_VALUES].sort(), ['codebase', 'research']);
});
test('digest schema: rejects a non-object payload', () => {
for (const bad of [null, 42, 'x', ['a']]) {
const r = validateDigest(bad);
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NOT_OBJECT'), `expected DIGEST_NOT_OBJECT for ${JSON.stringify(bad)}`);
}
});
test('digest schema: rejects missing/empty task', () => {
for (const t of [undefined, '', 123]) {
const d = wellFormed();
d.task = t;
const r = validateDigest(d);
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_TASK'));
}
});
test('digest schema: rejects missing/empty architecture_model', () => {
const d = wellFormed();
delete d.architecture_model;
const r = validateDigest(d);
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_ARCHITECTURE'));
});
test('digest schema: each synthesis array must be an array', () => {
const cases = [
['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'],
['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'],
['risks', 'DIGEST_RISKS_NOT_ARRAY'],
['gaps', 'DIGEST_GAPS_NOT_ARRAY'],
['sources', 'DIGEST_SOURCES_NOT_ARRAY'],
];
for (const [field, code] of cases) {
const d = wellFormed();
d[field] = { not: 'an array' };
const r = validateDigest(d);
assert.equal(r.valid, false, `${field} as object should fail`);
assert.ok(r.errors.some((e) => e.code === code), `expected ${code}`);
}
});
test('digest schema: empty synthesis arrays are valid (a clean digest can have no contradictions/gaps)', () => {
const d = wellFormed();
d.contradictions = [];
d.gaps = [];
const r = validateDigest(d);
assert.equal(r.valid, true, JSON.stringify(r.errors));
});
test('digest schema: a source with a non-enum origin is rejected', () => {
const d = wellFormed();
d.sources = [{ finding: 'x', origin: 'guess' }];
const r = validateDigest(d);
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_SOURCE_BAD_ORIGIN'));
});
test('digest schema: missing agent name is a warning, not an error', () => {
const d = wellFormed();
delete d.agent;
const r = validateDigest(d);
assert.equal(r.valid, true, JSON.stringify(r.errors));
assert.ok(r.warnings.some((w) => w.code === 'DIGEST_MISSING_AGENT'));
});
test('extractDigestBlock: pulls the LAST fenced json block from agent prose', () => {
const text =
'Here is my synthesis.\n\n' +
'```json\n{"stale": true}\n```\n\n' +
'Actually, the final digest:\n\n' +
'```json\n' + JSON.stringify(wellFormed()) + '\n```\n';
const block = extractDigestBlock(text);
assert.ok(block && JSON.parse(block).task, 'should extract the last json block');
assert.equal(JSON.parse(block).agent, 'synthesis-agent');
});
test('validateAgentOutput: no json fence → stable NO_JSON code', () => {
const r = validateAgentOutput('just prose, no fence');
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NO_JSON_BLOCK'));
});
test('validateAgentOutput: malformed json → stable PARSE code', () => {
const r = validateAgentOutput('```json\n{ not valid }\n```');
assert.equal(r.valid, false);
assert.ok(r.errors.some((e) => e.code === 'DIGEST_PARSE_ERROR'));
});
test('validateAgentOutput: well-formed fenced digest validates end-to-end', () => {
const r = validateAgentOutput('Synthesis complete.\n```json\n' + JSON.stringify(wellFormed()) + '\n```');
assert.equal(r.valid, true, JSON.stringify(r.errors));
});

View file

@ -0,0 +1,101 @@
// tests/scripts/synthesis-measure.test.mjs
// NW3 (S12) — deterministic Δ main-context measurement core.
//
// The gate metric (T1 §2) is Δ main-context tokens for an equivalent-quality
// digest. This pins the pure accounting that turns fixture token counts into a
// verdict, under the two framings that decide NW3:
//
// - FAITHFUL (current flow): Phase 5 swarm runs FOREGROUND, so its outputs are
// already RESIDENT in main before Phase 7. Delegating only the synthesis read
// cannot evict them → main holds base+out+dig in BOTH arms → Δ ≈ 0.
// - DISK-POTENTIAL (upper bound): IF outputs were on disk (a separate Phase-5
// change, out of NW3 scope), the delegated arm holds base+dig only → Δ = out/(base+out+dig).
//
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5 thresholds).
import { test } from 'node:test';
import { strict as assert } from 'node:assert';
import {
estimateTokens,
mainContextTokens,
deltaPct,
decideVerdict,
analyze,
} from '../../scripts/synthesis-measure.mjs';
test('estimateTokens: ~chars/4 heuristic, monotonic, non-negative', () => {
assert.equal(estimateTokens(''), 0);
assert.equal(estimateTokens('abcd'), 1);
assert.equal(estimateTokens('abcde'), 2); // ceil(5/4)
assert.ok(estimateTokens('a'.repeat(400)) === 100);
assert.ok(estimateTokens('x'.repeat(1000)) > estimateTokens('x'.repeat(500)));
});
test('mainContextTokens: inline arm holds base + out + dig', () => {
assert.equal(mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'inline' }), 63500);
});
test('mainContextTokens: delegated_faithful still holds out (Phase-5 resident) + dig', () => {
// The decisive structural fact: foreground swarm delivery already made `out`
// resident; delegating Phase 7 does not evict it.
assert.equal(
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_faithful' }),
63500,
);
});
test('mainContextTokens: delegated_disk holds base + dig only (out lives in the sub-agent)', () => {
assert.equal(
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_disk' }),
51500,
);
});
test('mainContextTokens: unknown arm throws (no silent default)', () => {
assert.throws(() => mainContextTokens({ base: 1, out: 1, dig: 1, arm: 'nope' }));
});
test('deltaPct: (A-B)/A; equal arms → 0; B smaller → positive fraction', () => {
assert.equal(deltaPct(100, 100), 0);
assert.equal(deltaPct(100, 75), 0.25);
assert.equal(deltaPct(0, 0), 0); // guard divide-by-zero
});
test('decideVerdict: ≥30% AND quality-ok → POSITIVE', () => {
assert.equal(decideVerdict(0.30, true), 'POSITIVE');
assert.equal(decideVerdict(0.45, true), 'POSITIVE');
});
test('decideVerdict: quality loss forces NEGATIVE even at a large Δ', () => {
assert.equal(decideVerdict(0.60, false), 'NEGATIVE');
});
test('decideVerdict: <15% → NEGATIVE; the [15%,30%) band → INCONCLUSIVE', () => {
assert.equal(decideVerdict(0.149, true), 'NEGATIVE');
assert.equal(decideVerdict(0.00, true), 'NEGATIVE');
assert.equal(decideVerdict(0.15, true), 'INCONCLUSIVE');
assert.equal(decideVerdict(0.2999, true), 'INCONCLUSIVE');
});
test('analyze: faithful arm is structurally Δ=0 → NEGATIVE regardless of sizes', () => {
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
assert.equal(a.faithful.deltaPct, 0);
assert.equal(a.faithful.verdict, 'NEGATIVE');
assert.equal(a.faithful.armA, a.faithful.armB);
});
test('analyze: disk arm Δ = out/(base+out+dig)', () => {
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
const expected = 12000 / (50000 + 12000 + 1500);
assert.ok(Math.abs(a.disk.deltaPct - expected) < 1e-9);
assert.equal(a.disk.armB, 51500);
});
test('analyze: disk verdict tracks BASE — large fixed baseline can sink the upper bound below 30%', () => {
// Small base → disk Δ clears 30%; large base → it does not. Demonstrates the
// upper bound is itself BASE-sensitive (sweep, do not assert one number).
const small = analyze({ baseTokens: 10000, outTokens: 12000, digTokens: 1500, qualityOK: true });
const large = analyze({ baseTokens: 200000, outTokens: 12000, digTokens: 1500, qualityOK: true });
assert.equal(small.disk.verdict, 'POSITIVE'); // 12000/23500 ≈ 0.51
assert.equal(large.disk.verdict, 'NEGATIVE'); // 12000/213500 ≈ 0.056
});