Kjell Tore Guttormsen
6b30483304
NW3 (CC-26 §6 PoC): delegate trekplan Phase 7 synthesis to a synthesis-agent, adopt only if Δ main-context ≥30% with no quality loss. Operator chose the deterministic-proof path (live ≥3-run bake-off was env-blocked: no API key; installed plugin is a cache copy so a new agent is invisible to `claude -p`). Decisive structural finding: trekplan Phase 5 runs the swarm FOREGROUND, so its outputs are already resident in main before Phase 7. Delegating only Phase 7 evicts nothing → Δ_faithful = 0% (BASE-independent). The ≥30% saving needs an out-of-scope Phase-5 redesign (swarm-writes-to-disk / nested orchestrator). VERDICT: DECLINED per measurement. - agents/synthesis-agent.md — dormant, schema-conformant deliverable (NOT wired) - lib/plan/synthesis-digest-schema.mjs — digest output contract (+ tests) - scripts/synthesis-measure.mjs — deterministic Δ-accounting core (+ tests) - tests/fixtures/synthesis/ — 7 exploration outputs + representative digest - docs/T1-synthesis-poc-results.md — measurement + verdict (reproducible) - CLAUDE.md — agent table row (doc-consistency: 24 agents) Tests 670 → 695 (693 pass / 2 skip / 0 fail). `claude plugin validate` clean (only the pre-existing root-CLAUDE.md warning). commands/trekplan.md untouched. [skip-docs] rationale: no user-facing feature ships (NW3 declined; agent dormant and unwired). The substantive doc is docs/T1-synthesis-poc-results.md; the README/CHANGELOG roll-up for NW1–NW3 is the S13 coordinated release per docs/W1-narrow-wins-plan.md §S13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
104 lines
5.8 KiB
Markdown
104 lines
5.8 KiB
Markdown
# T1 — Synthesis-agent PoC: Δ main-context measurement (NW3 / S12)
|
||
|
||
**Status:** Measurement complete — verdict below. **Method:** deterministic token-
|
||
accounting over real exploration fixtures (the live ≥3-run bake-off of T1 §5 is the
|
||
stronger instrument but is (a) environment-blocked here — no `ANTHROPIC_API_KEY`, and the
|
||
installed plugin is a cache copy so a fresh `synthesis-agent` is invisible to `claude -p`;
|
||
and (b) unnecessary, because the binding answer is STRUCTURAL, not stochastic).
|
||
**Resolves:** decision-matrix §W1 / CC-26 narrow PoC (`docs/T1-cc26-delegated-orchestration.md` §6).
|
||
**Reproduce:** `node scripts/synthesis-measure.mjs` (regenerates this file).
|
||
|
||
> Verifiseringsplikt: token figures are an explicit **chars/4 estimate** (labelled), not a
|
||
> tokenizer count. The gate turns on the RATIO Δ%, in which the per-token constant cancels
|
||
> for the `out` term. BASE (the fixed main-session baseline) is environment-dependent and
|
||
> was NOT API-measured this session → swept across a documented band, not asserted.
|
||
|
||
## 1. The decisive structural finding (BASE-independent)
|
||
|
||
trekplan runs the Phase 5 exploration swarm **foreground** (foreground is the only mode
|
||
since v2.4.0; `commands/trekplan.md`). Foreground Agent/Task results are delivered back
|
||
into the main transcript, so after Phase 5 the 6–10 exploration outputs are **already
|
||
resident in main**. Raw outputs are never written to disk (`trekplan.md:569` reserves the
|
||
"do NOT write to disk" rule for the synthesis text only). Phase 7 synthesis therefore
|
||
*reasons over already-resident context*. Delegating **only** the Phase-7 read to a
|
||
synthesis-agent — "main still spawns the swarm; only the digest is delegated" (T1 §6) —
|
||
**cannot evict those outputs from main**; the digest simply returns on top of them.
|
||
|
||
⇒ **Δ main-context (faithful flow) ≈ 0** — independent of every token count below. The
|
||
≥30% saving is only realizable by ALSO moving Phase-5 delivery off-main (swarm-writes-to-
|
||
disk, or a nested orchestrator owning the swarm), which is the wholesale change T1 §7
|
||
explicitly declined and is OUT of NW3 scope.
|
||
|
||
## 2. Fixtures (measured)
|
||
|
||
- Exploration dir: `tests/fixtures/synthesis/exploration`
|
||
- Digest: `tests/fixtures/synthesis/digest.json`
|
||
|
||
| exploration output | chars | est. tokens |
|
||
|--------------------|-------|-------------|
|
||
| architecture-mapper.md | 4774 | 1194 |
|
||
| convention-scanner.md | 1704 | 426 |
|
||
| dependency-tracer.md | 3470 | 868 |
|
||
| git-historian.md | 1703 | 426 |
|
||
| risk-assessor.md | 1950 | 488 |
|
||
| task-finder.md | 1927 | 482 |
|
||
| test-strategist.md | 1720 | 430 |
|
||
| **OUT (Σ resident in main)** | — | **4314** |
|
||
| digest (DIG) | — | 624 |
|
||
|
||
## 3. Δ main-context — both framings, swept over BASE
|
||
|
||
`inline` = base+out+dig · `delegated (faithful)` = base+out+dig (out already resident) ·
|
||
`delegated (disk-potential)` = base+dig (out off-main).
|
||
|
||
| BASE (est.) | inline | faithful Δ | faithful verdict | disk-potential Δ | disk verdict |
|
||
|-------------|--------|------------|------------------|------------------|--------------|
|
||
| 30000 | 34938 | 0.0% | NEGATIVE | 12.3% | NEGATIVE |
|
||
| 50000 | 54938 | 0.0% | NEGATIVE | 7.9% | NEGATIVE |
|
||
| 80000 | 84938 | 0.0% | NEGATIVE | 5.1% | NEGATIVE |
|
||
| 120000 | 124938 | 0.0% | NEGATIVE | 3.5% | NEGATIVE |
|
||
|
||
Break-even BASE for the disk-potential upper bound to reach the 30% adopt bar: **~9,442 tokens** (below this BASE the *hypothetical* disk path would clear 30%; at/above it, even the upper bound fails). A real Voyage main session's BASE (CC system prompt + plugin command/agent/skill listings + CLAUDE.md) is large, so the disk upper bound is itself fragile.
|
||
|
||
### Fixture-independent break-even (so the verdict does not hinge on fixture size)
|
||
|
||
disk-potential Δ = out/(base+out+dig), so it clears the 30% adopt bar **iff**
|
||
`out / base > 0.30/0.70 ≈ 0.43` — the combined exploration output must exceed ~43% of the
|
||
fixed main baseline. The table below sweeps OUT at an illustrative typical `BASE = 60,000` (independent of this run's fixtures):
|
||
|
||
| OUT (Σ exploration tokens) | disk-potential Δ @ ref BASE | clears 30%? |
|
||
|----------------------------|----------------------------|-------------|
|
||
| 5,000 | 7.6% | no |
|
||
| 10,000 | 14.2% | no |
|
||
| 20,000 | 24.8% | no |
|
||
| 30,000 | 33.1% | yes |
|
||
| 40,000 | 39.8% | yes |
|
||
|
||
This run's fixtures total **OUT = 4314 tokens** across 7 concise representative outputs — one concrete point on the curve. Even a generously large real swarm (OUT in the tens of thousands) only clears 30% when the main baseline is unusually small, and *never* in the faithful flow (Δ=0). The verdict is therefore robust to fixture size.
|
||
|
||
## 4. Quality
|
||
|
||
The digest-output contract (`lib/plan/synthesis-digest-schema.mjs`) pins the same Phase-7
|
||
synthesis dimensions main produces inline (task, architecture_model, reusable_code,
|
||
contradictions, risks, gaps, source-tagged findings). A delegated digest that validates is
|
||
structurally quality-equivalent to the inline one — but quality is moot here: the faithful
|
||
Δ is ~0, so there is no token win for quality to defend.
|
||
|
||
## 5. Verdict
|
||
|
||
**DECLINED per measurement.** NW3-as-scoped yields **Δ main-context ≈ 0%** (faithful flow,
|
||
structural — the Phase-5 foreground swarm already makes the outputs resident; delegating
|
||
Phase 7 evicts nothing). The disk-potential upper bound is reachable only via an out-of-
|
||
scope Phase-5 change and is itself BASE-fragile.
|
||
|
||
RESULT: NEGATIVE (Δ_faithful = 0.0% < 15.0% adopt-floor)
|
||
|
||
## 6. Disposition
|
||
|
||
- `agents/synthesis-agent.md` ships **dormant** (a documented, schema-conformant deliverable);
|
||
`commands/trekplan.md` Phase 7 is **NOT** wired to it.
|
||
- If main-context relief is later wanted, the prerequisite is a Phase-5 redesign (swarm-
|
||
writes-to-disk / nested orchestrator) — a separate, larger decision (re-open CC-26 §7).
|
||
- The dormant agent + this harness make that future step cheap to re-measure: drop new
|
||
fixtures in and re-run.
|
||
|