open/voyage
1
0
Fork 0
Code Issues Releases 1 Activity

feat(voyage): S12 — NW3 synthesis-agent built + measured → declined per measurement [skip-docs]

Browse source 
NW3 (CC-26 §6 PoC): delegate trekplan Phase 7 synthesis to a synthesis-agent,
adopt only if Δ main-context ≥30% with no quality loss. Operator chose the
deterministic-proof path (live ≥3-run bake-off was env-blocked: no API key;
installed plugin is a cache copy so a new agent is invisible to `claude -p`).

Decisive structural finding: trekplan Phase 5 runs the swarm FOREGROUND, so its
outputs are already resident in main before Phase 7. Delegating only Phase 7
evicts nothing → Δ_faithful = 0% (BASE-independent). The ≥30% saving needs an
out-of-scope Phase-5 redesign (swarm-writes-to-disk / nested orchestrator).
VERDICT: DECLINED per measurement.

- agents/synthesis-agent.md — dormant, schema-conformant deliverable (NOT wired)
- lib/plan/synthesis-digest-schema.mjs — digest output contract (+ tests)
- scripts/synthesis-measure.mjs — deterministic Δ-accounting core (+ tests)
- tests/fixtures/synthesis/ — 7 exploration outputs + representative digest
- docs/T1-synthesis-poc-results.md — measurement + verdict (reproducible)
- CLAUDE.md — agent table row (doc-consistency: 24 agents)

Tests 670 → 695 (693 pass / 2 skip / 0 fail). `claude plugin validate` clean
(only the pre-existing root-CLAUDE.md warning). commands/trekplan.md untouched.

[skip-docs] rationale: no user-facing feature ships (NW3 declined; agent dormant
and unwired). The substantive doc is docs/T1-synthesis-poc-results.md; the
README/CHANGELOG roll-up for NW1–NW3 is the S13 coordinated release per
docs/W1-narrow-wins-plan.md §S13.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 17:58:39 +02:00
commit 6b30483304
15 changed files with 1312 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

104
docs/T1-synthesis-poc-results.md Normal file
View file

@ -0,0 +1,104 @@
# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)
**Status:** Measurement complete — verdict below. **Method:** deterministic token-
accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the
stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the
installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;
and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).
**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).
**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).
> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a
> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels
> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and
> was NOT API-measured this session → swept across a documented band, not asserted.
## 1. The decisive structural finding (BASE-independent)
trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode
since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back
into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already
resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the
"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore
*reasons over already-resident context*. Delegating **only** the Phase-7 read to a
synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —
**cannot evict those outputs from main**; the digest simply returns on top of them.
⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The
≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-
disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7
explicitly declined and is OUT of NW3 scope.
## 2. Fixtures (measured)
- Exploration dir: `tests/fixtures/synthesis/exploration`
- Digest: `tests/fixtures/synthesis/digest.json`
| exploration output | chars | est. tokens |
|--------------------|-------|-------------|
| architecture-mapper.md | 4774 | 1194 |
| convention-scanner.md | 1704 | 426 |
| dependency-tracer.md | 3470 | 868 |
| git-historian.md | 1703 | 426 |
| risk-assessor.md | 1950 | 488 |
| task-finder.md | 1927 | 482 |
| test-strategist.md | 1720 | 430 |
| **OUT (Σ resident in main)** | — | **4314** |
| digest (DIG) | — | 624 |
## 3. Δ main-context — both framings, swept over BASE
`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·
`delegated (disk-potential)` = base+dig (out off-main).
| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |
|-------------|--------|------------|------------------|------------------|--------------|
| 30000 | 34938 | 0.0% | NEGATIVE | 12.3% | NEGATIVE |
| 50000 | 54938 | 0.0% | NEGATIVE | 7.9% | NEGATIVE |
| 80000 | 84938 | 0.0% | NEGATIVE | 5.1% | NEGATIVE |
| 120000 | 124938 | 0.0% | NEGATIVE | 3.5% | NEGATIVE |
Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: **~9,442 tokens** (below this BASE the *hypothetical* disk path would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE (CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk upper bound is itself fragile.
### Fixture-independent break-even (so the verdict does not hinge on fixture size)
disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**
`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the
fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = 60,000` (independent of this run's fixtures):
| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |
|----------------------------|----------------------------|-------------|
| 5,000 | 7.6% | no |
| 10,000 | 14.2% | no |
| 20,000 | 24.8% | no |
| 30,000 | 33.1% | yes |
| 40,000 | 39.8% | yes |
This run's fixtures total **OUT = 4314 tokens** across 7 concise representative outputs — one concrete point on the curve. Even a generously large real swarm (OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and *never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.
## 4. Quality
The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7
synthesis dimensions main produces inline (task, architecture_model, reusable_code,
contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is
structurally quality-equivalent to the inline one — but quality is moot here: the faithful
Δ is ~0, so there is no token win for quality to defend.
## 5. Verdict
**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,
structural — the Phase-5 foreground swarm already makes the outputs resident; delegating
Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-
scope Phase-5 change and is itself BASE-fragile.
RESULT: NEGATIVE (Δ_faithful = 0.0% < 15.0% adopt-floor)
## 6. Disposition
- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);
`commands/trekplan.md` Phase 7 is **NOT** wired to it.
- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-
writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).
- The dormant agent + this harness make that future step cheap to re-measure: drop new
fixtures in and re-run.