feat(voyage): S12 — NW3 synthesis-agent built + measured → declined per measurement [skip-docs]
NW3 (CC-26 §6 PoC): delegate trekplan Phase 7 synthesis to a synthesis-agent, adopt only if Δ main-context ≥30% with no quality loss. Operator chose the deterministic-proof path (live ≥3-run bake-off was env-blocked: no API key; installed plugin is a cache copy so a new agent is invisible to `claude -p`). Decisive structural finding: trekplan Phase 5 runs the swarm FOREGROUND, so its outputs are already resident in main before Phase 7. Delegating only Phase 7 evicts nothing → Δ_faithful = 0% (BASE-independent). The ≥30% saving needs an out-of-scope Phase-5 redesign (swarm-writes-to-disk / nested orchestrator). VERDICT: DECLINED per measurement. - agents/synthesis-agent.md — dormant, schema-conformant deliverable (NOT wired) - lib/plan/synthesis-digest-schema.mjs — digest output contract (+ tests) - scripts/synthesis-measure.mjs — deterministic Δ-accounting core (+ tests) - tests/fixtures/synthesis/ — 7 exploration outputs + representative digest - docs/T1-synthesis-poc-results.md — measurement + verdict (reproducible) - CLAUDE.md — agent table row (doc-consistency: 24 agents) Tests 670 → 695 (693 pass / 2 skip / 0 fail). `claude plugin validate` clean (only the pre-existing root-CLAUDE.md warning). commands/trekplan.md untouched. [skip-docs] rationale: no user-facing feature ships (NW3 declined; agent dormant and unwired). The substantive doc is docs/T1-synthesis-poc-results.md; the README/CHANGELOG roll-up for NW1–NW3 is the S13 coordinated release per docs/W1-narrow-wins-plan.md §S13. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
9fb536e2d8
commit
6b30483304
15 changed files with 1312 additions and 0 deletions
29
tests/fixtures/synthesis/digest.json
vendored
Normal file
29
tests/fixtures/synthesis/digest.json
vendored
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
{
|
||||
"agent": "synthesis-agent",
|
||||
"task": "Add a per-wave concurrency cap to trekexecute headless launches",
|
||||
"architecture_model": "trekexecute's headless path (Phase 2.6) turns each independent wave from session-decomposer's dependency graph into a launch batch, and templates/headless-launch-template.md backgrounds every batch member as a parallel `claude -p` subprocess with no slot limit — degree of parallelism equals batch size. The fix straddles a JS/Bash boundary: a pure, testable max-parallel resolver in lib/ (reusing the arg-parser + profile-resolver lookup order) feeds one validated integer into the template's fan-out, which should consume it via `xargs -P` rather than a hand-rolled Bash semaphore.",
|
||||
"reusable_code": [
|
||||
{ "ref": "lib/parsers/arg-parser.mjs", "note": "parse --max-parallel <n>; do not hand-roll" },
|
||||
{ "ref": "lib/profiles/profile-resolver.mjs", "note": "lookup order flag→signal→profile→default for the cap" },
|
||||
{ "ref": "lib/util/result.mjs:33", "note": "issue() error shape for invalid cap values" },
|
||||
{ "ref": "templates/headless-launch-template.md:10", "note": "the fan-out wire-in site" }
|
||||
],
|
||||
"contradictions": [
|
||||
"architecture-mapper frames the cap as launch-time Bash-only; task-finder + convention-scanner argue the arithmetic must live in a pure lib resolver — reconciled: resolver computes, template consumes one integer"
|
||||
],
|
||||
"risks": [
|
||||
{ "risk": "Too-low cap serializes independent waves, erasing the parallelism trekexecute exists for", "severity": "high" },
|
||||
{ "risk": "Hand-rolled Bash semaphore can deadlock `wait` on subprocess crash — prefer xargs -P", "severity": "high" },
|
||||
{ "risk": "Cap of 0/negative could stall the pipeline; clamp to >=1 with a stable error code", "severity": "medium" }
|
||||
],
|
||||
"gaps": [
|
||||
"No measured per-wave token/process ceiling exists — the safe default (cap = batch size, i.e. no-op) is an assumption until profiled",
|
||||
"Whether xargs -P is available/identical across the operator's macOS + any headless CI is unverified"
|
||||
],
|
||||
"sources": [
|
||||
{ "finding": "launcher backgrounds all members then waits, no slot limit", "origin": "codebase" },
|
||||
{ "finding": "project already serializes spawns in q3 harness to dodge spawn-burst limits", "origin": "codebase" },
|
||||
{ "finding": "profile system (v4.1) is the canonical knob lookup order", "origin": "codebase" },
|
||||
{ "finding": "xargs -P release-on-exit semantics vs hand-rolled semaphore", "origin": "research" }
|
||||
]
|
||||
}
|
||||
96
tests/fixtures/synthesis/exploration/architecture-mapper.md
vendored
Normal file
96
tests/fixtures/synthesis/exploration/architecture-mapper.md
vendored
Normal file
|
|
@ -0,0 +1,96 @@
|
|||
# Architecture Report — output of architecture-mapper
|
||||
|
||||
Task being planned: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Summary
|
||||
|
||||
Voyage is a contract-driven Claude Code plugin (Node ESM, zero runtime deps).
|
||||
The pipeline is six commands (`commands/*.md`) backed by a `lib/` of pure,
|
||||
unit-tested validators/parsers and a thin `scripts/` layer of measurement and
|
||||
codegen harnesses. trekexecute is the disciplined plan/session-spec executor;
|
||||
its headless path (Phase 2.6) fans out parallel "waves" of `claude -p`
|
||||
subprocesses driven by a Bash launcher template.
|
||||
|
||||
## Tech stack
|
||||
|
||||
| Layer | Choice | Evidence |
|
||||
|-------|--------|----------|
|
||||
| Language | JavaScript (ESM, `.mjs`) | `lib/**/*.mjs`, `"type":"module"` in package.json |
|
||||
| Tests | `node:test` + `node:assert/strict` | every `tests/**/*.test.mjs` |
|
||||
| Validation | hand-rolled validators returning `{valid,errors,warnings}` | `lib/util/result.mjs` |
|
||||
| Orchestration substrate | command prose + Bash + Agent/Task tool | `commands/trekexecute.md` |
|
||||
| Headless launch | Bash here-doc template, backgrounded subprocesses | `templates/headless-launch-template.md` |
|
||||
|
||||
## Key patterns
|
||||
|
||||
- **3-layer module pattern** — Content validator → raw-text wrapper → CLI shim
|
||||
(`if (import.meta.url === \`file://${process.argv[1]}\`)`), repeated across
|
||||
`lib/validators`, `lib/parsers`, `lib/review`. Any new lib must follow it.
|
||||
- **Structured Result type** — `issue(code,message,hint,location)` + `fail()` /
|
||||
`ok()` from `lib/util/result.mjs`. Stable error codes are the contract.
|
||||
- **Prose-as-orchestrator** — commands carry the control flow in markdown; the
|
||||
harness executes it. Schema-drift defenses are *inlined* into command prose so
|
||||
they survive even when agent docs are not implicitly loaded.
|
||||
|
||||
## Anti-patterns / debt near the task
|
||||
|
||||
- The headless launcher backgrounds **all** wave members at once with no upper
|
||||
bound on concurrent `claude -p` processes; concurrency is implicit in how many
|
||||
steps a wave contains. No central place caps it.
|
||||
- Wave composition (which steps go in which wave) is computed by
|
||||
session-decomposer, but the *launch* fan-out is template Bash, so a cap would
|
||||
straddle a JS (decomposer) / Bash (launcher) boundary.
|
||||
|
||||
## Module map (task-relevant)
|
||||
|
||||
```
|
||||
commands/trekexecute.md # Phase 2.6 parallel-wave orchestration prose
|
||||
templates/headless-launch-template.md # the Bash fan-out site
|
||||
lib/util/result.mjs # error shape any new guard returns
|
||||
agents/session-decomposer.md # produces the wave/dependency graph
|
||||
```
|
||||
|
||||
## Boundaries
|
||||
|
||||
The cap is a launch-time concern (Bash template + the Phase 2.6 prose that
|
||||
generates it). It does not belong in the pure `lib/` validators unless we add a
|
||||
small "max parallelism" resolver that the prose reads. Recommend a lib resolver
|
||||
(testable) + a template wire-in (the actual `xargs -P` / job-slot mechanism).
|
||||
|
||||
## How Phase 2.6 assembles a wave (detail)
|
||||
|
||||
The executor reads the plan's `## Step N` blocks and the dependency edges
|
||||
session-decomposer emitted (`depends_on:` frontmatter). Steps with no unmet
|
||||
dependency at the current frontier form a wave. For each wave the prose:
|
||||
|
||||
1. builds a `SHARED_CONTEXT_FILE` (brief + plan + relevant exploration digest)
|
||||
passed to every member via `--append-system-prompt-file` (cache-prefix
|
||||
material — see q3 experiment);
|
||||
2. emits one `claude -p … &` invocation per member from the here-doc, each with
|
||||
`--max-turns`, `--max-budget-usd`, `GIT_OPTIONAL_LOCKS=0`, and the GH#36071
|
||||
push-before-cleanup workaround;
|
||||
3. collects the backgrounded PIDs and `wait`s for the batch to drain before
|
||||
advancing the frontier.
|
||||
|
||||
The cap belongs strictly between (2) and (3): bound how many of the emitted
|
||||
members run concurrently, leaving wave *composition* (1) untouched.
|
||||
|
||||
## Layering verdict
|
||||
|
||||
Three layers, in increasing blast radius: (a) a pure resolver in `lib/`
|
||||
(arithmetic only — trivially testable, follows the 3-layer module pattern);
|
||||
(b) Phase 2.6 prose passing the resolved integer + a `--max-parallel` flag into
|
||||
the template; (c) the template's fan-out mechanism. Keep (a) the single source
|
||||
of the number; (c) should only consume it. This matches every prior launcher
|
||||
hardening, which added one bounded externality (budget, locks, turns) at a time
|
||||
without reshaping wave composition.
|
||||
|
||||
## Cross-cutting observations
|
||||
|
||||
- The launcher is the most operationally sensitive file in the repo (it spends
|
||||
money and mutates git). Every edit here is co-reviewed with its
|
||||
doc-consistency needle list — treat the needle test as part of the contract,
|
||||
not an afterthought.
|
||||
- Nothing in `lib/` currently imports anything launcher-related; the resolver
|
||||
will be a leaf module. Good — it can be tested and shipped independently of the
|
||||
template wire-in, enabling a TDD-first slice.
|
||||
42
tests/fixtures/synthesis/exploration/convention-scanner.md
vendored
Normal file
42
tests/fixtures/synthesis/exploration/convention-scanner.md
vendored
Normal file
|
|
@ -0,0 +1,42 @@
|
|||
# Conventions Report — output of convention-scanner
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Summary
|
||||
|
||||
Mature, consistent conventions. New code should imitate the existing `lib/`
|
||||
validators almost mechanically.
|
||||
|
||||
## Naming
|
||||
|
||||
| Element | Convention | Example |
|
||||
|---------|-----------|---------|
|
||||
| Files | kebab-case `.mjs` | `lib/profiles/profile-resolver.mjs` |
|
||||
| Functions | camelCase, verb-first | `validateFindings`, `extractDigestBlock` |
|
||||
| Error codes | UPPER_SNAKE, domain-prefixed | `FINDING_BAD_LINE`, `DIGEST_NOT_OBJECT` |
|
||||
| Tests | `<module>.test.mjs` mirroring `lib/` tree | `tests/lib/profile-resolver.test.mjs` |
|
||||
|
||||
## Module shape (must follow)
|
||||
|
||||
Every `lib/` module is the **3-layer pattern**:
|
||||
1. exported pure functions (validate/resolve/compute);
|
||||
2. they return `{valid, errors, warnings, parsed}` via `lib/util/result.mjs`
|
||||
`issue()`/`fail()`/`ok()`;
|
||||
3. a CLI shim `if (import.meta.url === \`file://${process.argv[1]}\`)` for Bash.
|
||||
|
||||
A `max-parallel-resolver.mjs` must replicate this exactly (pure resolver +
|
||||
`issue()` errors + CLI shim).
|
||||
|
||||
## Error handling
|
||||
|
||||
- Never coerce bad input — return a `{valid:false}` Result with a stable code and
|
||||
a `hint`. Throwing is reserved for genuine programmer error (see
|
||||
`mainContextTokens` "unknown arm" throw).
|
||||
- Unknown/extra fields are tolerated (forward-compat), load-bearing fields are
|
||||
hard errors. Mirror this: clamp/validate the cap, tolerate extra profile keys.
|
||||
|
||||
## Imports / tests / commits
|
||||
|
||||
- Named ESM imports, relative paths, no path aliases, no barrels.
|
||||
- Conventional Commits, `type(scope): description`, e.g. `feat(voyage): …`.
|
||||
- New behavior is TDD'd: failing `node:test` first, then minimal code.
|
||||
76
tests/fixtures/synthesis/exploration/dependency-tracer.md
vendored
Normal file
76
tests/fixtures/synthesis/exploration/dependency-tracer.md
vendored
Normal file
|
|
@ -0,0 +1,76 @@
|
|||
# Dependency & Data-Flow Report — output of dependency-tracer
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Import / call chain relevant to the task
|
||||
|
||||
```
|
||||
commands/trekexecute.md (Phase 2.6 prose)
|
||||
└─ generates → templates/headless-launch-template.md (Bash here-doc)
|
||||
├─ reads SHARED_CONTEXT_FILE (append-system-prompt-file)
|
||||
├─ spawns claude -p ×N (one per wave member, backgrounded with &)
|
||||
└─ waits via `wait` on collected PIDs
|
||||
└─ consumes → .session-state.local.json (Handover 7; session graph)
|
||||
```
|
||||
|
||||
## Data flow
|
||||
|
||||
1. session-decomposer emits a plan with wave groupings + a dependency graph.
|
||||
2. trekexecute Phase 2.6 turns each independent wave into a launch batch.
|
||||
3. The template loops over batch members and backgrounds each `claude -p`,
|
||||
collecting PIDs into a Bash array, then `wait`s for the whole batch.
|
||||
4. There is **no slot-limiting** between "background member" and "wait" — the
|
||||
degree of parallelism equals the batch size.
|
||||
|
||||
## Side effects
|
||||
|
||||
- Each subprocess does `git` work under `GIT_OPTIONAL_LOCKS=0`; high concurrency
|
||||
raises the chance of index-lock contention (mitigated, not eliminated).
|
||||
- `--max-budget-usd` is per-subprocess; total spend scales with batch size, so a
|
||||
concurrency cap also indirectly bounds burst spend.
|
||||
|
||||
## What a cap touches
|
||||
|
||||
- **Pure-addable:** a `maxParallel` resolver in `lib/` (reads plan/profile/flag,
|
||||
returns an integer ≥ 1). No existing module imports would change.
|
||||
- **Wire-in:** the template's loop must consume slots (e.g. a counting semaphore
|
||||
in Bash, or `xargs -P <n>`). This is the only behavioral edit.
|
||||
|
||||
## No hidden dependents
|
||||
|
||||
Grepped for other call sites of the launch template — only trekexecute Phase 2.6
|
||||
and the headless-launch-template test reference it. A cap is local in blast
|
||||
radius.
|
||||
|
||||
## Resolver input provenance (what the cap reads)
|
||||
|
||||
The resolved integer must be derived from, in lookup order:
|
||||
|
||||
1. **CLI flag** `--max-parallel <n>` — parsed by `lib/parsers/arg-parser.mjs`;
|
||||
highest precedence (operator override).
|
||||
2. **Brief signal** — `phase_signals` already carries per-phase orchestration
|
||||
shape; a `max_parallel` hint here is honoured if no flag.
|
||||
3. **Profile** — `lib/profiles/` resolves `--profile economy|balanced|premium`;
|
||||
each profile can carry a `max_parallel` default. This is the same lookup
|
||||
order `phase_models` uses, so the resolver should *reuse* profile-resolver,
|
||||
not re-implement precedence.
|
||||
4. **Hard default** — `batchSize` (i.e. no cap / current behavior), so the change
|
||||
is a strict no-op until someone opts in.
|
||||
|
||||
## Downstream of the cap
|
||||
|
||||
- **Budget:** total burst spend = `min(cap, batchSize) × per-member --max-budget-usd`.
|
||||
A cap therefore tightens the worst-case spend envelope — worth noting in the
|
||||
plan's risk/observability section.
|
||||
- **Git contention:** fewer concurrent `git`-touching subprocesses → fewer
|
||||
`index.lock` races. `GIT_OPTIONAL_LOCKS=0` reduces lock acquisition but does
|
||||
not serialize ref updates; the cap is the structural mitigation.
|
||||
- **Classifier exposure:** a smaller concurrent fan-out under `auto`/`bypass`
|
||||
lowers the surface the proliferation classifier (S7 F4) scrutinises.
|
||||
|
||||
## Data-flow invariant to preserve
|
||||
|
||||
`SHARED_CONTEXT_FILE` is built once per wave and read by every member; the cap
|
||||
must not cause it to be rebuilt per-slot (would defeat the cache prefix). Slot
|
||||
limiting happens at spawn time only; the context file is wave-scoped, not
|
||||
slot-scoped.
|
||||
31
tests/fixtures/synthesis/exploration/git-historian.md
vendored
Normal file
31
tests/fixtures/synthesis/exploration/git-historian.md
vendored
Normal file
|
|
@ -0,0 +1,31 @@
|
|||
# Git History — output of git-historian
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Recent changes touching the launch path
|
||||
|
||||
| Commit (illustrative) | Area | Relevance |
|
||||
|-----------------------|------|-----------|
|
||||
| Phase 2.6 hardening series | trekexecute.md + headless-launch-template.md | Added `GIT_OPTIONAL_LOCKS`, `--max-budget-usd`, push-before-cleanup, GH#36071 workaround. The launcher is actively maintained and recently hardened — a cap is the next natural hardening. |
|
||||
| session-decomposer wiring | agents/session-decomposer.md | Established the wave/dependency graph the launcher consumes. Stable; not the edit site. |
|
||||
| profile system (v4.1) | lib/profiles/ | Introduced `phase_models` + `--profile`; the lookup order a `max_parallel` knob should reuse. |
|
||||
|
||||
## Ownership / hot files
|
||||
|
||||
- `templates/headless-launch-template.md` is a **hot file** — multiple recent
|
||||
hardening commits. Expect a strict doc-consistency needle list; any edit must
|
||||
keep all existing needles AND add the new one.
|
||||
- `commands/trekexecute.md` Phase 2.6 is co-edited with the template in every
|
||||
hardening commit ("template mirrors Phase 2.6"). Keep them in lockstep.
|
||||
|
||||
## Active branches / risk of conflict
|
||||
|
||||
Single active branch (`main`); polyrepo, frequent small commits. Low conflict
|
||||
risk. The kjøremodus is one-task-per-session, so this change should be a single
|
||||
focused commit touching resolver + template + prose + tests.
|
||||
|
||||
## Signal
|
||||
|
||||
The project's own history shows a consistent preference: bound risky externalities
|
||||
explicitly (budget, locks, turns). An unbounded fan-out is the conspicuous gap in
|
||||
that pattern — the change is in-character with how this launcher has evolved.
|
||||
25
tests/fixtures/synthesis/exploration/risk-assessor.md
vendored
Normal file
25
tests/fixtures/synthesis/exploration/risk-assessor.md
vendored
Normal file
|
|
@ -0,0 +1,25 @@
|
|||
# Risk & Failure-Mode Report — output of risk-assessor
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Risks (ranked)
|
||||
|
||||
| # | Risk | Severity | Mitigation |
|
||||
|---|------|----------|------------|
|
||||
| R1 | A cap that is too low **serializes** independent waves, erasing the parallelism trekexecute exists to provide. | high | Default the cap to the batch size (no-op) and only clamp when the operator/profile asks; never silently throttle. |
|
||||
| R2 | Bash semaphore bugs **deadlock** the `wait` (slots never released on subprocess crash). | high | Prefer `xargs -P <n>` over a hand-rolled counting semaphore — release-on-exit is built in. Guard with a per-subprocess `--max-turns`/timeout already present. |
|
||||
| R3 | The cap interacts with the **proliferation classifier** (S7 F4): a large parallel fan-out under `auto`/`bypass` is already scrutinised. A cap *reduces* this exposure, but mis-set to 0/negative could stall the pipeline. | medium | Clamp to `≥ 1`; reject `0`/negatives with a stable error code (`issue()`), do not coerce. |
|
||||
| R4 | Index-lock contention under high concurrency (multiple `git` subprocesses) is *masked* today by luck; raising the cap re-exposes it. | medium | Document that `GIT_OPTIONAL_LOCKS=0` is necessary-not-sufficient; the cap is the real fix. |
|
||||
| R5 | Drift between the JS resolver default and the Bash template's actual `-P` value. | low | Single source: prose passes the resolved integer into the template; a test asserts the template consumes `$MAX_PARALLEL`. |
|
||||
|
||||
## Edge cases
|
||||
|
||||
- Wave of size 1 → cap is irrelevant (no fan-out).
|
||||
- Cap ≥ batch size → must be a pure no-op (R1).
|
||||
- Non-integer / missing flag → fall through resolver to profile default, not crash.
|
||||
|
||||
## Complexity hotspots
|
||||
|
||||
The straddle between JS (resolver, testable) and Bash (launcher, hard to unit
|
||||
test) is the main hazard. Keep ALL arithmetic in the resolver; the template
|
||||
should only consume one already-validated integer.
|
||||
36
tests/fixtures/synthesis/exploration/task-finder.md
vendored
Normal file
36
tests/fixtures/synthesis/exploration/task-finder.md
vendored
Normal file
|
|
@ -0,0 +1,36 @@
|
|||
# Task-Relevant Code — output of task-finder
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Direct hits
|
||||
|
||||
| File | Lines | Why relevant |
|
||||
|------|-------|--------------|
|
||||
| `templates/headless-launch-template.md` | ~10–60 | The Bash fan-out: backgrounds each wave member, `wait`s on PIDs. Edit site for slot-limiting. |
|
||||
| `commands/trekexecute.md` | Phase 2.6 | Prose that generates the launch batches; where a `--max-parallel` flag + profile lookup would be documented. |
|
||||
| `lib/profiles/` | resolver dir | Profiles already carry per-phase knobs; a `max_parallel` default fits the existing `phase_signals`/profile lookup order. |
|
||||
| `lib/util/result.mjs` | 33 | `issue()` — the error shape a `maxParallel` resolver returns on bad input. |
|
||||
|
||||
## Reuse candidates
|
||||
|
||||
- **arg-parser** (`lib/parsers/arg-parser.mjs`, per tests/lib/arg-parser.test.mjs)
|
||||
already parses `--flag value` pairs for the commands. A `--max-parallel <n>`
|
||||
flag plugs into the existing parser; do not hand-roll parsing.
|
||||
- **profile-resolver** (`lib/profiles/`, profile-resolver.test.mjs) is the lookup
|
||||
order CLI-flag → brief signal → profile → default. A concurrency default
|
||||
belongs as a profile field consumed through this same resolver.
|
||||
- **autonomy-gate** (`lib/` autonomy-gate.test.mjs) shows the established pattern
|
||||
for "bounded integer with a safe default" — mirror its clamp/validate.
|
||||
|
||||
## Existing similar solutions
|
||||
|
||||
`scripts/q3-cache-prefix-experiment.mjs` spawns children **sequentially** to
|
||||
avoid spawn-burst rate-limits — confirms the project already knows unbounded
|
||||
fan-out is a risk and chose a manual bound there. A cap generalises that
|
||||
instinct to the headless wave path.
|
||||
|
||||
## Models / config
|
||||
|
||||
No DB. Config lives in `settings.json` (`trekplan`/`trekresearch` scopes only —
|
||||
doc-consistency pins this) and per-run profiles. A `max_parallel` knob should
|
||||
ride the profile system, not add a new settings.json scope.
|
||||
38
tests/fixtures/synthesis/exploration/test-strategist.md
vendored
Normal file
38
tests/fixtures/synthesis/exploration/test-strategist.md
vendored
Normal file
|
|
@ -0,0 +1,38 @@
|
|||
# Test Strategy — output of test-strategist
|
||||
|
||||
Task: Add a per-wave concurrency cap to trekexecute headless launches.
|
||||
|
||||
## Existing patterns
|
||||
|
||||
- Framework: `node:test` + `node:assert/strict`. One file per module under
|
||||
`tests/lib/` (mirrors `lib/`) and `tests/scripts/` for harness cores.
|
||||
- Validators are tested as **pure functions**: feed a payload, assert
|
||||
`{valid,errors}` + stable error codes (see findings-schema.test.mjs,
|
||||
autonomy-gate.test.mjs).
|
||||
- Template/prose invariants are pinned by **doc-consistency.test.mjs** (string
|
||||
`assert.ok(text.includes(...))` against templates/commands).
|
||||
|
||||
## Coverage gaps for this task
|
||||
|
||||
1. No test asserts an upper bound on launch parallelism today (there is none).
|
||||
2. headless-launch-template.md is pinned for a list of required needles
|
||||
(`GIT_OPTIONAL_LOCKS`, `--max-turns`, `--max-budget-usd`, …) but NOT for any
|
||||
concurrency mechanism.
|
||||
|
||||
## Recommended tests (TDD order)
|
||||
|
||||
1. **`lib/.../max-parallel-resolver.test.mjs`** (new) — pure resolver:
|
||||
- flag `--max-parallel 3` wins over profile/default;
|
||||
- missing flag → profile default → hard default;
|
||||
- `0` / negative / non-integer → `{valid:false}` with a stable code;
|
||||
- cap ≥ batchSize → returns batchSize (no-op clamp).
|
||||
2. **doc-consistency extension** — assert headless-launch-template.md consumes
|
||||
the resolved integer (e.g. includes `$MAX_PARALLEL` or `xargs -P`).
|
||||
3. **arg-parser** — assert `--max-parallel` is recognised (extend
|
||||
arg-parser.test.mjs / gates-flag-coverage pattern).
|
||||
|
||||
## Notes
|
||||
|
||||
The Bash fan-out itself is not unit-testable in `node:test`; rely on the
|
||||
template-needle pin (#2) + keeping arithmetic in the JS resolver (#1). This
|
||||
matches how the project already tests prose-driven behavior.
|
||||
161
tests/lib/synthesis-digest-schema.test.mjs
Normal file
161
tests/lib/synthesis-digest-schema.test.mjs
Normal file
|
|
@ -0,0 +1,161 @@
|
|||
// tests/lib/synthesis-digest-schema.test.mjs
|
||||
// NW3 (S12) — digest-schema contract for the synthesis-agent.
|
||||
//
|
||||
// The synthesis-agent (agents/synthesis-agent.md) ingests the trekplan Phase-5/7
|
||||
// exploration outputs and emits a trailing fenced ```json block: the findings
|
||||
// DIGEST main currently writes inline in Phase 7. This pins that digest's shape
|
||||
// so a delegated path could VALIDATE it (not merely parse it) and so the
|
||||
// measurement harness has a fixed quality contract to compare against.
|
||||
//
|
||||
// Load-bearing fields (what Phase 8 deep-planning consumes): task,
|
||||
// architecture_model, and the five synthesis arrays (reusable_code,
|
||||
// contradictions, risks, gaps, sources). Each source must be origin-tagged
|
||||
// codebase|research (Phase 7 rule 7: "track whether it came from codebase
|
||||
// analysis or external research"). Mirrors lib/review/findings-schema.mjs.
|
||||
//
|
||||
// When this test fails, fix the schema or the producer — do NOT relax the
|
||||
// assertion to hide drift.
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import {
|
||||
validateDigest,
|
||||
validateAgentOutput,
|
||||
extractDigestBlock,
|
||||
ORIGIN_VALUES,
|
||||
DIGEST_REQUIRED_FIELDS,
|
||||
} from '../../lib/plan/synthesis-digest-schema.mjs';
|
||||
|
||||
function wellFormed() {
|
||||
return {
|
||||
agent: 'synthesis-agent',
|
||||
task: 'Add a per-wave rate-limit guard to trekexecute headless launches',
|
||||
architecture_model:
|
||||
'trekexecute spawns parallel headless waves via the Bash launcher template; ' +
|
||||
'concurrency is currently unbounded per wave.',
|
||||
reusable_code: [
|
||||
{ ref: 'lib/util/result.mjs:33', note: 'issue() for structured errors' },
|
||||
{ ref: 'templates/headless-launch-template.md:10', note: 'wave dispatch site' },
|
||||
],
|
||||
contradictions: [
|
||||
'architecture-mapper says waves are sequential; dependency-tracer shows a parallel fan-out',
|
||||
],
|
||||
risks: [
|
||||
{ risk: 'A rate-limit that blocks too aggressively starves long waves', severity: 'medium' },
|
||||
],
|
||||
gaps: ['No measured per-wave token ceiling exists yet — becomes an assumption'],
|
||||
sources: [
|
||||
{ finding: 'wave launcher lives in the template', origin: 'codebase' },
|
||||
{ finding: 'CC headless --max-budget-usd semantics', origin: 'research' },
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
test('digest schema: a well-formed digest validates', () => {
|
||||
const r = validateDigest(wellFormed());
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
assert.deepEqual(r.errors, []);
|
||||
});
|
||||
|
||||
test('digest schema: exposes the required-field + origin contracts', () => {
|
||||
assert.ok(Array.isArray(DIGEST_REQUIRED_FIELDS) && DIGEST_REQUIRED_FIELDS.length > 0);
|
||||
for (const f of ['task', 'architecture_model', 'reusable_code', 'contradictions', 'risks', 'gaps', 'sources']) {
|
||||
assert.ok(DIGEST_REQUIRED_FIELDS.includes(f), `required fields must include ${f}`);
|
||||
}
|
||||
assert.deepEqual([...ORIGIN_VALUES].sort(), ['codebase', 'research']);
|
||||
});
|
||||
|
||||
test('digest schema: rejects a non-object payload', () => {
|
||||
for (const bad of [null, 42, 'x', ['a']]) {
|
||||
const r = validateDigest(bad);
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NOT_OBJECT'), `expected DIGEST_NOT_OBJECT for ${JSON.stringify(bad)}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('digest schema: rejects missing/empty task', () => {
|
||||
for (const t of [undefined, '', 123]) {
|
||||
const d = wellFormed();
|
||||
d.task = t;
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_TASK'));
|
||||
}
|
||||
});
|
||||
|
||||
test('digest schema: rejects missing/empty architecture_model', () => {
|
||||
const d = wellFormed();
|
||||
delete d.architecture_model;
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_MISSING_ARCHITECTURE'));
|
||||
});
|
||||
|
||||
test('digest schema: each synthesis array must be an array', () => {
|
||||
const cases = [
|
||||
['reusable_code', 'DIGEST_REUSABLE_NOT_ARRAY'],
|
||||
['contradictions', 'DIGEST_CONTRADICTIONS_NOT_ARRAY'],
|
||||
['risks', 'DIGEST_RISKS_NOT_ARRAY'],
|
||||
['gaps', 'DIGEST_GAPS_NOT_ARRAY'],
|
||||
['sources', 'DIGEST_SOURCES_NOT_ARRAY'],
|
||||
];
|
||||
for (const [field, code] of cases) {
|
||||
const d = wellFormed();
|
||||
d[field] = { not: 'an array' };
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, false, `${field} as object should fail`);
|
||||
assert.ok(r.errors.some((e) => e.code === code), `expected ${code}`);
|
||||
}
|
||||
});
|
||||
|
||||
test('digest schema: empty synthesis arrays are valid (a clean digest can have no contradictions/gaps)', () => {
|
||||
const d = wellFormed();
|
||||
d.contradictions = [];
|
||||
d.gaps = [];
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
});
|
||||
|
||||
test('digest schema: a source with a non-enum origin is rejected', () => {
|
||||
const d = wellFormed();
|
||||
d.sources = [{ finding: 'x', origin: 'guess' }];
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_SOURCE_BAD_ORIGIN'));
|
||||
});
|
||||
|
||||
test('digest schema: missing agent name is a warning, not an error', () => {
|
||||
const d = wellFormed();
|
||||
delete d.agent;
|
||||
const r = validateDigest(d);
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
assert.ok(r.warnings.some((w) => w.code === 'DIGEST_MISSING_AGENT'));
|
||||
});
|
||||
|
||||
test('extractDigestBlock: pulls the LAST fenced json block from agent prose', () => {
|
||||
const text =
|
||||
'Here is my synthesis.\n\n' +
|
||||
'```json\n{"stale": true}\n```\n\n' +
|
||||
'Actually, the final digest:\n\n' +
|
||||
'```json\n' + JSON.stringify(wellFormed()) + '\n```\n';
|
||||
const block = extractDigestBlock(text);
|
||||
assert.ok(block && JSON.parse(block).task, 'should extract the last json block');
|
||||
assert.equal(JSON.parse(block).agent, 'synthesis-agent');
|
||||
});
|
||||
|
||||
test('validateAgentOutput: no json fence → stable NO_JSON code', () => {
|
||||
const r = validateAgentOutput('just prose, no fence');
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_NO_JSON_BLOCK'));
|
||||
});
|
||||
|
||||
test('validateAgentOutput: malformed json → stable PARSE code', () => {
|
||||
const r = validateAgentOutput('```json\n{ not valid }\n```');
|
||||
assert.equal(r.valid, false);
|
||||
assert.ok(r.errors.some((e) => e.code === 'DIGEST_PARSE_ERROR'));
|
||||
});
|
||||
|
||||
test('validateAgentOutput: well-formed fenced digest validates end-to-end', () => {
|
||||
const r = validateAgentOutput('Synthesis complete.\n```json\n' + JSON.stringify(wellFormed()) + '\n```');
|
||||
assert.equal(r.valid, true, JSON.stringify(r.errors));
|
||||
});
|
||||
101
tests/scripts/synthesis-measure.test.mjs
Normal file
101
tests/scripts/synthesis-measure.test.mjs
Normal file
|
|
@ -0,0 +1,101 @@
|
|||
// tests/scripts/synthesis-measure.test.mjs
|
||||
// NW3 (S12) — deterministic Δ main-context measurement core.
|
||||
//
|
||||
// The gate metric (T1 §2) is Δ main-context tokens for an equivalent-quality
|
||||
// digest. This pins the pure accounting that turns fixture token counts into a
|
||||
// verdict, under the two framings that decide NW3:
|
||||
//
|
||||
// - FAITHFUL (current flow): Phase 5 swarm runs FOREGROUND, so its outputs are
|
||||
// already RESIDENT in main before Phase 7. Delegating only the synthesis read
|
||||
// cannot evict them → main holds base+out+dig in BOTH arms → Δ ≈ 0.
|
||||
// - DISK-POTENTIAL (upper bound): IF outputs were on disk (a separate Phase-5
|
||||
// change, out of NW3 scope), the delegated arm holds base+dig only → Δ = out/(base+out+dig).
|
||||
//
|
||||
// POSITIVE adopt requires Δ ≥ 30% AND quality ≥ inline (T1 §5 thresholds).
|
||||
|
||||
import { test } from 'node:test';
|
||||
import { strict as assert } from 'node:assert';
|
||||
import {
|
||||
estimateTokens,
|
||||
mainContextTokens,
|
||||
deltaPct,
|
||||
decideVerdict,
|
||||
analyze,
|
||||
} from '../../scripts/synthesis-measure.mjs';
|
||||
|
||||
test('estimateTokens: ~chars/4 heuristic, monotonic, non-negative', () => {
|
||||
assert.equal(estimateTokens(''), 0);
|
||||
assert.equal(estimateTokens('abcd'), 1);
|
||||
assert.equal(estimateTokens('abcde'), 2); // ceil(5/4)
|
||||
assert.ok(estimateTokens('a'.repeat(400)) === 100);
|
||||
assert.ok(estimateTokens('x'.repeat(1000)) > estimateTokens('x'.repeat(500)));
|
||||
});
|
||||
|
||||
test('mainContextTokens: inline arm holds base + out + dig', () => {
|
||||
assert.equal(mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'inline' }), 63500);
|
||||
});
|
||||
|
||||
test('mainContextTokens: delegated_faithful still holds out (Phase-5 resident) + dig', () => {
|
||||
// The decisive structural fact: foreground swarm delivery already made `out`
|
||||
// resident; delegating Phase 7 does not evict it.
|
||||
assert.equal(
|
||||
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_faithful' }),
|
||||
63500,
|
||||
);
|
||||
});
|
||||
|
||||
test('mainContextTokens: delegated_disk holds base + dig only (out lives in the sub-agent)', () => {
|
||||
assert.equal(
|
||||
mainContextTokens({ base: 50000, out: 12000, dig: 1500, arm: 'delegated_disk' }),
|
||||
51500,
|
||||
);
|
||||
});
|
||||
|
||||
test('mainContextTokens: unknown arm throws (no silent default)', () => {
|
||||
assert.throws(() => mainContextTokens({ base: 1, out: 1, dig: 1, arm: 'nope' }));
|
||||
});
|
||||
|
||||
test('deltaPct: (A-B)/A; equal arms → 0; B smaller → positive fraction', () => {
|
||||
assert.equal(deltaPct(100, 100), 0);
|
||||
assert.equal(deltaPct(100, 75), 0.25);
|
||||
assert.equal(deltaPct(0, 0), 0); // guard divide-by-zero
|
||||
});
|
||||
|
||||
test('decideVerdict: ≥30% AND quality-ok → POSITIVE', () => {
|
||||
assert.equal(decideVerdict(0.30, true), 'POSITIVE');
|
||||
assert.equal(decideVerdict(0.45, true), 'POSITIVE');
|
||||
});
|
||||
|
||||
test('decideVerdict: quality loss forces NEGATIVE even at a large Δ', () => {
|
||||
assert.equal(decideVerdict(0.60, false), 'NEGATIVE');
|
||||
});
|
||||
|
||||
test('decideVerdict: <15% → NEGATIVE; the [15%,30%) band → INCONCLUSIVE', () => {
|
||||
assert.equal(decideVerdict(0.149, true), 'NEGATIVE');
|
||||
assert.equal(decideVerdict(0.00, true), 'NEGATIVE');
|
||||
assert.equal(decideVerdict(0.15, true), 'INCONCLUSIVE');
|
||||
assert.equal(decideVerdict(0.2999, true), 'INCONCLUSIVE');
|
||||
});
|
||||
|
||||
test('analyze: faithful arm is structurally Δ=0 → NEGATIVE regardless of sizes', () => {
|
||||
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||
assert.equal(a.faithful.deltaPct, 0);
|
||||
assert.equal(a.faithful.verdict, 'NEGATIVE');
|
||||
assert.equal(a.faithful.armA, a.faithful.armB);
|
||||
});
|
||||
|
||||
test('analyze: disk arm Δ = out/(base+out+dig)', () => {
|
||||
const a = analyze({ baseTokens: 50000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||
const expected = 12000 / (50000 + 12000 + 1500);
|
||||
assert.ok(Math.abs(a.disk.deltaPct - expected) < 1e-9);
|
||||
assert.equal(a.disk.armB, 51500);
|
||||
});
|
||||
|
||||
test('analyze: disk verdict tracks BASE — large fixed baseline can sink the upper bound below 30%', () => {
|
||||
// Small base → disk Δ clears 30%; large base → it does not. Demonstrates the
|
||||
// upper bound is itself BASE-sensitive (sweep, do not assert one number).
|
||||
const small = analyze({ baseTokens: 10000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||
const large = analyze({ baseTokens: 200000, outTokens: 12000, digTokens: 1500, qualityOK: true });
|
||||
assert.equal(small.disk.verdict, 'POSITIVE'); // 12000/23500 ≈ 0.51
|
||||
assert.equal(large.disk.verdict, 'NEGATIVE'); // 12000/213500 ≈ 0.056
|
||||
});
|
||||
Loading…
Add table
Add a link
Reference in a new issue