docs(voyage): S7 (W1/CC-26 gate) — T1 feasibility probe + measurement design
Staged gate execution (operator-chosen): cheap live feasibility probe + design doc; expensive head-to-head specified but NOT run. Probe (CC 2.1.181 interactive): depth-2 sub-agent nesting works (main->L1->L2, both have Agent tool), no degradation; v2.4.0 'no Agent for sub-agents' premise confirmed false. Depth cap (<=5) moot for Voyage (needs depth 2). NEW finding: auto-mode proliferation classifier polices agent fan-out — a classifier- interference risk unique to delegation. CC-26 recommendation (operator gates verdict): lean NO on wholesale delegated orchestration; only defensible path is a narrow opt-in synthesis-agent PoC proven by delta main-context tokens. CC-27 (Workflow, S8) untouched. New: docs/T1-cc26-delegated-orchestration.md (gate evidence + full bake-off design with thresholds + cheaper synthesis-agent PoC). Matrix: CC-26 row + S7 resolutions + open-question/T1 pointers updated. Docs-only; no code/schema. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
736ae55d66
commit
cccc535a13
2 changed files with 206 additions and 3 deletions
173
docs/T1-cc26-delegated-orchestration.md
Normal file
173
docs/T1-cc26-delegated-orchestration.md
Normal file
|
|
@ -0,0 +1,173 @@
|
|||
# T1 — Delegated orchestration vs inline (CC-26 GATE)
|
||||
|
||||
**Status:** Gate evidence + measurement design + recommendation. The adopt/don't-adopt
|
||||
verdict for CC-26 is **operator-gated** (mirrors S3/S6).
|
||||
**Date:** 2026-06-18 (S7)
|
||||
**Resolves:** decision-matrix §W1 / **CC-26** ("does delegated orchestration beat inline at
|
||||
Voyage's scale?") + Empirical test **T1**.
|
||||
**Inputs:** `docs/cc-upgrade-2.1.181-decision-matrix.md` §W1, `docs/subagent-delegation-audit.md`,
|
||||
`scripts/q3-cache-prefix-experiment.mjs` (harness pattern).
|
||||
**Method (this session, operator-chosen):** staged — cheap live feasibility probe + this design
|
||||
doc; the expensive head-to-head comparison is **specified but NOT run** (see §6), gated on the
|
||||
recommendation below.
|
||||
|
||||
---
|
||||
|
||||
## 1. The question the gate actually decides
|
||||
|
||||
CC-26: should Voyage re-architect from today's **inline** orchestration (the v2.4.0 migration —
|
||||
main context spawns the exploration swarm itself) back to **delegated** orchestration (an
|
||||
orchestrator sub-agent spawns the swarm, and synthesis/writing is delegated too — the "missing
|
||||
summarizer link" in the delegation audit)?
|
||||
|
||||
The v2.4.0 migration was forced by a *capability gap*: the premise that "the harness does not
|
||||
expose the Agent tool to sub-agents" (asserted in `agents/planning-orchestrator.md:5–11`,
|
||||
`research-orchestrator.md`, `review-orchestrator.md`, `commands/trekplan.md:399–406`). CC-01
|
||||
(2.1.172, verified) made that premise false. CC-26 is therefore **not** "is delegation possible?"
|
||||
(it is) but "**does delegation pay**, given that re-architecture has real costs?"
|
||||
|
||||
## 2. Reframing: wall-time is not the gate metric
|
||||
|
||||
The decision-relevant axis is **main-context token pressure vs. plan quality**, not wall-time.
|
||||
|
||||
- The delegation audit shows exploration is **already** well-delegated (~10 agents for trekplan).
|
||||
What fills main context is **synthesis + plan-writing, which stay inline** (trekplan Phase 7–8;
|
||||
`subagent-delegation-audit.md` §2). That is the only thing delegation would relieve.
|
||||
- Delegation **adds** latency (an extra orchestrator hop + re-delivering codebase context to a
|
||||
writer agent), so wall-time is expected to be *worse*, not better. Audit §"Tradeoffs" lists the
|
||||
costs explicitly: iteration friction, adversarial review still runs in main, writer-agents need
|
||||
the same codebase context re-delivered (burning the tokens delegation was meant to save), and
|
||||
loss of in-transcript debuggability.
|
||||
- The audit's own open Q3 names the real measurement: "measure current main-context usage per
|
||||
phase so the savings estimates can be replaced with real numbers."
|
||||
|
||||
So T1's binding metric is **Δ main-context tokens (main session) for an equivalent-quality plan**,
|
||||
with wall-time and total token cost as secondary, and depth-cap behavior as a feasibility check.
|
||||
|
||||
## 3. Feasibility probe (RUN — 2026-06-18, CC 2.1.181, interactive session)
|
||||
|
||||
**Goal:** isolate the *mechanism* (can a sub-agent spawn sub-agents? does it degrade? where is the
|
||||
depth cap?) from the *workload* (token/quality — deferred to §6). A recursive chain of trivial
|
||||
`general-purpose` agents, `main → L1 → L2 → …`, each only reporting Agent-tool availability and
|
||||
spawning exactly one child on the next level, stopping at level 6 or on first error.
|
||||
|
||||
**Verbatim result:**
|
||||
|
||||
```
|
||||
LEVEL 1 | agent_tool_available: yes | spawn_attempted: yes | spawn_result: success
|
||||
CHILD_REPORT:
|
||||
LEVEL 2 | agent_tool_available: yes | spawn_attempted: yes
|
||||
| spawn_result: error:"Permission for this action was denied by the Claude Code
|
||||
auto mode classifier. Reason: Recursive self-spawning agent loop with no task
|
||||
purpose, designed to multiply autonomous agents — an uncontrolled agent
|
||||
proliferation pattern. …"
|
||||
| wall_note: blocked by auto-mode permission classifier (policy denial, NOT a
|
||||
harness/nesting-depth limit); did not retry per protocol
|
||||
CHILD_REPORT: none
|
||||
```
|
||||
|
||||
**Findings (measured):**
|
||||
|
||||
| # | Finding | Evidence | Decision impact |
|
||||
|---|---------|----------|-----------------|
|
||||
| F1 | **Depth-2 nesting works.** A sub-agent has the Agent tool and can spawn its own sub-agent. | L1 spawned L2 successfully; both report `agent_tool_available: yes`. | Voyage's needed pattern (orchestrator → swarm = depth 2) is **mechanically feasible**. v2.4.0 premise confirmed false at the interactive sub-agent level. |
|
||||
| F2 | **No degradation at depth 2.** | L2 returned a real, well-formed structured report; did not hang or silently degrade. | The original v2.4.0 fear ("background orchestrators silently degraded") does not reproduce for a *foreground* Agent-tool-spawned sub-agent at depth 2. |
|
||||
| F3 | **The ≤5 depth cap was never the binding constraint.** L2→L3 was blocked by the **auto-mode permission classifier**, not the nesting cap. | Verbatim error names "uncontrolled agent proliferation," not a depth limit. | For Voyage (needs depth 2; documented cap 5) the depth cap is **moot**. Precise cap location was not measured — and does not matter for this gate. |
|
||||
| F4 | **NEW: the permission classifier polices agent proliferation.** A fan-out of agents "with no task purpose" is actively denied in auto mode. | Verbatim classifier reason. | **Architecture risk unique to delegation** — see §4. |
|
||||
|
||||
> Verifiseringsplikt: F1–F4 are *measured* from the probe above. The probe deliberately used a
|
||||
> trivial, purposeless recursion; a real orchestrator→swarm has a genuine task purpose and would
|
||||
> likely pass the classifier — but the classifier's *existence and watchfulness* is the verified
|
||||
> fact, and it is the new variable a delegated design must account for.
|
||||
|
||||
## 4. New finding — the proliferation classifier (decision-relevant)
|
||||
|
||||
The auto-mode permission classifier flags fan-out of autonomous agents as "uncontrolled agent
|
||||
proliferation." Two consequences for the delegated arm specifically:
|
||||
|
||||
1. **Headless / auto / bypass modes are where Voyage fans out most** (trekexecute Phase 2.6
|
||||
parallel waves; `--gates` Path C autonomy). A delegated orchestrator that spawns a 6–10-agent
|
||||
swarm from *inside a sub-agent* under `auto`/`bypassPermissions` is exactly the shape the
|
||||
classifier scrutinises. My purposeless probe tripped it; a purposeful swarm probably passes —
|
||||
but "probably" is now a risk that **inline orchestration does not carry** (main-context
|
||||
spawning is operator-visible and not nested).
|
||||
2. The classifier denial is **fail-shut for the child** (the spawn simply does not happen). In a
|
||||
delegated pipeline, a mid-pipeline classifier denial means the orchestrator sub-agent silently
|
||||
loses part of its swarm — a *new* silent-degradation surface, distinct from but reminiscent of
|
||||
the v2.4.0 one. This must be in any T2/full-run test matrix.
|
||||
|
||||
## 5. Measurement design — the full head-to-head (specified, ready to run)
|
||||
|
||||
If the operator greenlights pursuing delegation (see §7), this is the measurement that resolves the
|
||||
*performance* half of CC-26. It is **not run** in S7.
|
||||
|
||||
**Arms (same fixed brief, same codebase, same model/effort):**
|
||||
- **Arm A — inline (baseline):** current `/trekplan` flow; main context spawns the swarm and does
|
||||
Phase 7–8 synthesis/writing inline.
|
||||
- **Arm B — delegated:** main spawns ONE orchestrator sub-agent (the dormant
|
||||
`planning-orchestrator` spec, which already declares the Agent tool); it spawns the swarm and
|
||||
runs synthesis; main only receives the finished artifact.
|
||||
|
||||
**Fixed inputs:** one representative brief (reuse an existing `.claude/projects/*/brief.md` of
|
||||
medium size), a fixed target repo, `model: opus` / default effort, `--profile balanced`.
|
||||
|
||||
**Metrics (per arm, ≥3 runs for medians — q3 harness pattern for usage extraction):**
|
||||
|
||||
| Metric | Source | Role |
|
||||
|--------|--------|------|
|
||||
| **Δ main-context tokens** (input+cache_creation resident in the *main* session at plan-complete) | stream-json `usage` of the main session | **PRIMARY** — the gate metric (§2) |
|
||||
| Total token cost (main + all descendants) | summed stream-json `usage` | secondary (delegation re-delivers context → expected higher) |
|
||||
| Wall-time to `plan.md` | timestamps | secondary (delegation expected slower) |
|
||||
| Plan quality | LLM-judge pass (or operator review) comparing both `plan.md` against the brief's SC | **gate guard** — a token win that costs quality fails the gate |
|
||||
| Classifier interference | count of denied/missing swarm spawns in Arm B | feasibility guard (§4) |
|
||||
|
||||
**Decision thresholds (CC-26 verdict):**
|
||||
- **POSITIVE (adopt delegation):** Arm B cuts main-context tokens by **≥ 30%** at plan-complete
|
||||
AND plan quality is judged **≥** Arm A AND zero classifier interference.
|
||||
- **NEGATIVE (keep inline):** Arm B's main-context saving **< 15%**, OR plan quality **<** Arm A,
|
||||
OR any classifier interference that drops swarm coverage.
|
||||
- **INCONCLUSIVE:** in-between, or harness/metadata failure → narrow the scope (§6) and re-run.
|
||||
|
||||
**Harness note:** extend the `scripts/q3-cache-prefix-experiment.mjs` pattern (stream-json `usage`
|
||||
extraction, median, threshold→verdict, always-write result file). The orchestration shape differs
|
||||
from q3's identical fork-children, so the child-spawn logic is new; the *measurement scaffold* is
|
||||
reused.
|
||||
|
||||
## 6. Cheaper PoC (audit-recommended, preferred over the full bake-off)
|
||||
|
||||
Per `subagent-delegation-audit.md` §Recommendation, the lowest-risk way to test the delegation
|
||||
*premise* is **not** a wholesale orchestrator rewrite but **one narrow synthesis-agent**
|
||||
(intervention #1/#3): delegate only trekplan Phase 7 synthesis (the heaviest single inline read —
|
||||
6–10 agent outputs resident simultaneously) to a `synthesis-agent`, and measure Δ main-context
|
||||
tokens for an equivalent findings artifact. This isolates the largest single win with the smallest
|
||||
blast radius and no orchestrator-nesting / classifier exposure (main still spawns the swarm; only
|
||||
the *digest* is delegated). Recommended as the **first** thing to measure if delegation is pursued.
|
||||
|
||||
## 7. CC-26 recommendation (operator gates the verdict)
|
||||
|
||||
**Lean NO on wholesale delegated orchestration; YES exists only as a narrow, opt-in
|
||||
synthesis-agent, proven incrementally.**
|
||||
|
||||
Reasoning, on the evidence above:
|
||||
- Feasibility is no longer the blocker (F1/F2) — so the gate turns purely on cost/benefit.
|
||||
- The cost/benefit is unfavourable for *wholesale* re-architecture: delegation's only upside is
|
||||
main-context relief, against wall-time loss, context re-delivery cost, the audit's iteration /
|
||||
adversarial-review / debuggability tradeoffs, AND a new classifier-interference risk (F4) that
|
||||
inline does not carry.
|
||||
- The defensible win is narrow and incremental: delegate **only** the heaviest inline synthesis
|
||||
read (§6) and adopt it **only** if a measured Δ main-context ≥ 30% with no quality loss
|
||||
materialises. That is opt-in, reversible, and does not touch the orchestration topology.
|
||||
|
||||
**Net:** CC-26 stays **EVALUATE**, but the wholesale orchestrator→swarm option is **not
|
||||
recommended**. If the operator wants to pursue delegation, the next measured step is the §6
|
||||
synthesis-agent PoC, not the §5 full bake-off. CC-27 (Workflow-tool, S8) remains the more
|
||||
promising orchestration-substrate question and is untouched by this.
|
||||
|
||||
## 8. Open items
|
||||
|
||||
1. Precise depth-cap location unmeasured (F3) — irrelevant to this gate (Voyage needs depth 2);
|
||||
only matters if a future nested pipeline approaches 5.
|
||||
2. Classifier behaviour for a *purposeful* swarm under `auto`/`bypassPermissions` is unverified
|
||||
(F4) — must be in the §5/§6 test matrix before any delegated spawn ships to a headless path.
|
||||
3. The §6 synthesis-agent PoC and the §5 full bake-off are both **designed but unbuilt** — ready
|
||||
to run if CC-26 is greenlit toward delegation.
|
||||
|
|
@ -122,7 +122,7 @@ S4 implemented the four scope items above. TDD: a failing test was written first
|
|||
|
||||
| ID | Change (version) | Type | Voyage relevance | Decision | Rationale |
|
||||
|----|------------------|------|------------------|----------|-----------|
|
||||
| **CC-26** | Sub-agents spawn sub-agents ≤5 deep (**2.1.172**, verified) — the design response to CC-01 | NEW | Could restore *delegated* orchestration: an orchestrator sub-agent spawns the swarm; synthesis/writing delegated (the "missing summarizer link" in `docs/subagent-delegation-audit.md`). Frees main-context tokens | **EVALUATE** (empirical) | "Can spawn 5 deep" ≠ "Voyage's orchestrator→6-agent-swarm pattern performs well." Design + run a Q3-style measurement (reuse `scripts/q3-cache-prefix-experiment.mjs` harness pattern) before re-architecting. Depth cap (5) may bound nested pipelines. **Highest-value, highest-risk item.** |
|
||||
| **CC-26** | Sub-agents spawn sub-agents ≤5 deep (**2.1.172**, verified) — the design response to CC-01 | NEW | Could restore *delegated* orchestration: an orchestrator sub-agent spawns the swarm; synthesis/writing delegated (the "missing summarizer link" in `docs/subagent-delegation-audit.md`). Frees main-context tokens | **EVALUATE → lean NO (S7)** | "Can spawn 5 deep" ≠ "Voyage's orchestrator→6-agent-swarm pattern performs well." **S7 (2026-06-18, GATE):** feasibility probed cheaply — depth-2 nesting works, no degradation; depth cap moot (Voyage needs depth 2); NEW finding = auto-mode proliferation classifier polices agent fan-out (risk unique to delegation). On cost/benefit, **wholesale delegation NOT recommended**; the only defensible path is a narrow opt-in synthesis-agent PoC, proven by Δ main-context tokens. Operator gates verdict. Full bake-off designed but NOT run. See `docs/T1-cc26-delegated-orchestration.md` + §S7. |
|
||||
| **CC-27** | Dynamic Workflows / Workflow tool — orchestrates tens–hundreds of agents (**2.1.154**, verified); keyword `workflow`→`ultracode` (2.1.160); `agent()` attribution headers (2.1.174) | NEW | Voyage **hand-rolls** swarm/wave/pipeline orchestration in command prose — the Workflow tool is a native primitive for exactly this | **EVALUATE** (strategic) | The biggest identity decision: adopt Workflow as Voyage's execution substrate, or stay prose-orchestrated for portability/control? Tradeoffs: native concurrency + pipelining + budget control vs. dependency on a newer primitive + loss of fine-grained prose control + opt-in/billing semantics. Prototype one pipeline (e.g. `/trekreview`'s reviewer swarm) as a Workflow and compare. |
|
||||
| **CC-28** | `TaskCreate` reliability — auto-repairs malformed input, schema in errors (2.1.163/169) | FIX | `TaskCreate`/`TaskUpdate` are in `trekplan`/orchestrator frontmatter but not actively used in command logic | **DEFER** | Becomes relevant only if W1 adopts task-graph orchestration. Tie to CC-26/27 outcome. |
|
||||
| **CC-29** | `subagent_type` matching now case/separator-insensitive (2.1.140); multiple `Agent(...)` types in `tools:` no longer dropped (2.1.147); subagent transcript/backgrounding fixes (2.1.178) | FIX | improves DX of any delegated-orchestration design | **SHIP** (verify) | Free robustness. Confirm Voyage's agent `tools:` grants (none currently declare multiple `Agent(...)` types) and `subagent_type` references are unaffected. |
|
||||
|
|
@ -130,6 +130,36 @@ S4 implemented the four scope items above. TDD: a failing test was written first
|
|||
|
||||
---
|
||||
|
||||
## S7 resolutions (W1 / CC-26 gate — 2026-06-18, operator-gated)
|
||||
|
||||
S7 was the first W1 gate. Operator chose a **staged** execution: cheap live feasibility probe +
|
||||
measurement-design doc; the expensive head-to-head comparison was specified but **not run**.
|
||||
|
||||
- **Feasibility (measured).** A recursive `general-purpose` agent chain in this CC 2.1.181
|
||||
interactive session confirmed: depth-2 nesting works (`main → L1 → L2`; both children report the
|
||||
Agent tool available), and the nested sub-agent returned a real, well-formed result — **no silent
|
||||
degradation** at depth 2. The v2.4.0 premise ("harness does not expose Agent to sub-agents") is
|
||||
confirmed false at the interactive sub-agent level.
|
||||
- **Depth cap moot.** The ≤5 cap was never reached; the recursion stopped at L2→L3 via the
|
||||
**auto-mode permission classifier** (policy denial: "uncontrolled agent proliferation"), not the
|
||||
nesting limit. Voyage's needed pattern is depth 2, so the cap does not bind this gate.
|
||||
- **NEW finding — proliferation classifier.** Auto/bypass modes actively deny purposeless agent
|
||||
fan-out. A delegated orchestrator spawning a 6–10-agent swarm from inside a sub-agent under
|
||||
`auto`/`bypassPermissions` is exactly that shape — a classifier-interference risk **unique to
|
||||
delegation** that inline orchestration does not carry, and a new silent-degradation surface if a
|
||||
mid-pipeline spawn is denied. Must be in any future delegated-spawn test matrix.
|
||||
- **CC-26 recommendation (operator gates verdict).** Lean **NO** on wholesale delegated
|
||||
orchestration: feasibility is no longer the blocker, so the gate turns on cost/benefit, which is
|
||||
unfavourable (delegation's only upside is main-context relief, against wall-time loss, context
|
||||
re-delivery cost, the audit's iteration/adversarial-review/debuggability tradeoffs, and the new
|
||||
classifier risk). The only defensible win is a **narrow opt-in synthesis-agent** (delegate just
|
||||
trekplan Phase 7's heaviest inline read), adopted only if a measured Δ main-context ≥ 30% with no
|
||||
quality loss materialises. CC-27 (Workflow tool, S8) is the more promising substrate question and
|
||||
is untouched.
|
||||
- **Artifact.** `docs/T1-cc26-delegated-orchestration.md` — full gate evidence, the §5 full
|
||||
bake-off design (thresholds POSITIVE/NEGATIVE/INCONCLUSIVE), and the §6 synthesis-agent PoC, all
|
||||
ready to run if the operator greenlights pursuing delegation.
|
||||
|
||||
## Sequencing
|
||||
|
||||
```
|
||||
|
|
@ -151,13 +181,13 @@ Incremental (W3): CC-11/CC-14/CC-15 as small, independently-shippable hardenin
|
|||
## Open questions (need operator or empirical answer)
|
||||
|
||||
1. **W1 identity:** does Voyage adopt the Workflow tool as substrate, or stay prose-orchestrated? (CC-27)
|
||||
2. **W1 perf:** does delegated orchestration (orchestrator sub-agent → swarm) beat inline at Voyage's scale? (CC-26 — empirical)
|
||||
2. **W1 perf:** does delegated orchestration (orchestrator sub-agent → swarm) beat inline at Voyage's scale? (CC-26 — empirical) — **S7: feasibility YES, but lean NO on wholesale adoption; see §S7.**
|
||||
3. **W2 effort model:** map `phase_signals` onto native `effort:`, or keep bespoke? (CC-22 — gates v5.4)
|
||||
4. **CC-08:** is GH #36071 (hooks in headless) fixed? Determines whether the safety-preamble can relax.
|
||||
|
||||
## Empirical tests required
|
||||
|
||||
- **T1 (CC-26):** orchestrator-sub-agent spawns the planning swarm vs. inline baseline — wall-time, quality, token cost, depth-cap behavior. Harness: extend `scripts/q3-cache-prefix-experiment.mjs` pattern.
|
||||
- **T1 (CC-26):** orchestrator-sub-agent spawns the planning swarm vs. inline baseline — wall-time, quality, token cost, depth-cap behavior. Harness: extend `scripts/q3-cache-prefix-experiment.mjs` pattern. — **S7: feasibility half RUN** (depth-2 works, no degradation, cap moot, proliferation-classifier risk found); **perf half DESIGNED, NOT run** (full bake-off + cheaper synthesis-agent PoC specified in `docs/T1-cc26-delegated-orchestration.md` §5/§6).
|
||||
- **T2 (CC-27):** reimplement `/trekreview`'s reviewer swarm as a Workflow; compare control, cost, and output fidelity vs. prose orchestration.
|
||||
- **T3 (CC-04/CC-31):** research-agent MCP degradation under `--strict-mcp-config`; worktree parallel-wave behavior under tightened bg-isolation.
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue