Kjell Tore Guttormsen
cae698a204
commands/trekplan.md:885 documents the effort=='low' route into the --quick-equivalent code-path, not the --quick flag itself. Quote :122 as the primary source and describe :885 as the second way in, so the anchor says what the line actually says.
295 lines
17 KiB
Markdown
295 lines
17 KiB
Markdown
# BRIEF — external assessment (VURDERING-V2 §5.12), verified against this repo
|
||
|
||
**Status:** assessment complete. **No implementation in this session** (scope-guard).
|
||
**Source under review:** `claude-playlist-corpus/docs/VURDERING-V2.md` §5.12 (+ §3 G1–G8, §4),
|
||
transferred via that repo's `docs/OVERFORING-V2.md` §5. It is an external recommendation
|
||
built on 442 video analyses; its repo facts come from a subagent survey (2026-07-17) that
|
||
is **not** re-verified and in which at least one error was already demonstrated.
|
||
|
||
**Method.** Every asserted GAP is marked **BEKREFTET** (confirmed) / **AVKREFTET**
|
||
(refuted) / **ENDRET** (true in altered form) only after checking it against code in this
|
||
repo. Every Claude Code *feature* claim is checked against the OKF bundle at
|
||
`claude-code-llm-wiki/bundle/`, index-first — **where V2 and the bundle disagree, the
|
||
bundle wins**. Absences are positive-controlled before being reported (a grep that finds
|
||
nothing is a measurement, not a fact).
|
||
|
||
---
|
||
|
||
## TL;DR
|
||
|
||
Of six recommended tiltak, **one is a real defect** (fail-open verdict computation),
|
||
**one is unmeasured rather than unresolved** (the Workflow port), **three are already
|
||
decided or already partly built**, and **one is a cheap confirmed gap**.
|
||
|
||
The single most consequential correction: V2 presents the Workflow port as the answer to
|
||
this repo's "main-context relief is unmeasured" caveat. The bundle confirms the
|
||
*mechanism*, but **this repo has still never measured main-context relief on a large
|
||
fan-out** — the one bake-off that exists measured *subagent* tokens and says so. The port
|
||
is therefore neither refuted nor justified. The next step is a measurement, not a port,
|
||
and the measurement design already exists in `docs/T1-cc26-delegated-orchestration.md`.
|
||
|
||
---
|
||
|
||
## Tiltak 1 — Port orchestration to dynamic Workflows
|
||
|
||
**Verdict: split per command. Do not issue one global verdict.**
|
||
|
||
### `/trekreview` — **ENDRET** (already ported, and measured)
|
||
|
||
The port exists: opt-in `--workflow`, `scripts/trekreview-armB.workflow.mjs`, documented at
|
||
`commands/trekreview.md:270` (*§ Phase 5–6 via the Workflow substrate*). The S10 bake-off
|
||
(`docs/T2-bakeoff-results.md`, ≥3 runs/arm) found it **fidelity-equivalent**: verdict-match
|
||
1.0, issue-coverage 100%, `(file,rule_key)` jaccard ≥ within-arm. It stays opt-in because
|
||
the Workflow tool raises the consumer floor to Claude Code 2.1.154+, and the prose path
|
||
keeps the lower floor.
|
||
|
||
### `/trekplan`, `/trekresearch` — **BEKREFTET as unported, UNMEASURED as beneficial**
|
||
|
||
Both still orchestrate their swarms from main context (`commands/trekplan.md:406`,
|
||
`commands/trekresearch.md:388`). That part of V2 is correct.
|
||
|
||
What is **not** established is V2's stated effect — that Workflow answers the
|
||
main-context-relief caveat. Two measurements exist in this repo and **neither measures it**:
|
||
|
||
| Measurement | What it actually measured | Bearing on the relief claim |
|
||
|---|---|---|
|
||
| `docs/T2-bakeoff-results.md:154` | **Subagent** tokens: Arm A median 92.8k vs Arm B 96.9k ⇒ **+4.4%** | None. The same line states Arm A's main-context hand-orchestration cost is **uncounted**. |
|
||
| `docs/T1-synthesis-poc-results.md` | Δ **main-context** ≈ **0%** — for delegating *Phase 7 synthesis alone* | Negative, but out of scope: that doc itself names a **Phase-5 redesign** as the prerequisite for any relief. |
|
||
|
||
So the Phase-5 swarm — the case V2 is actually recommending — has never been measured in
|
||
either arm, in either command.
|
||
|
||
**The measurement already has a design.** `docs/T1-cc26-delegated-orchestration.md:118`
|
||
defines the gate metric (Δ main-context tokens resident in the main session at
|
||
plan-complete, from stream-json `usage`) with pre-registered bars: adopt at ≥30%, reject
|
||
below 15%. Running Arm A vs Arm B once against the trekplan swarm settles this.
|
||
|
||
**Cost framing must be corrected.** The bundle states plainly that a workflow run "can use
|
||
meaningfully more tokens than working through the same task in conversation"
|
||
(`bundle/concepts/docs-en-workflows.md:364`) — consistent with the measured +4.4%. The port
|
||
cannot be justified as a cost saving. **Main-context headroom is the only defensible
|
||
motive, and it is exactly the unmeasured quantity.**
|
||
|
||
**Recommendation:** measure before porting. Treat V2's tiltak 1 as a *hypothesis with a
|
||
ready-made experiment*, not as an approved change.
|
||
|
||
---
|
||
|
||
## Tiltak 2 — Codify the holdout rules
|
||
|
||
**Verdict: (a) and (b) satisfied in practice but uncodified · (c) BEKREFTET, a real defect ·
|
||
(d) ENDRET.** This is the tiltak with genuine substance.
|
||
|
||
### (a) Reviewers must never be forks — **AVKREFTET as a live defect, BEKREFTET as uncodified**
|
||
|
||
Reviewers are spawned as plain subagents (`voyage:brief-conformance-reviewer`,
|
||
`voyage:code-correctness-reviewer`). `subagent_type: "fork"` appears nowhere in `commands/`
|
||
or `agents/`; the only `subagent_type` in the tree is `"Explore"` at
|
||
`commands/trekplan.md:501`, an exploration spawn, not a review spawn. The bundle confirms
|
||
the rule is real — a forked subagent "inherits your full conversation context instead of
|
||
starting fresh" (`bundle/concepts/docs-en-agents.md:37`). Voyage satisfies it by habit;
|
||
nothing written forbids a future edit from breaking it.
|
||
|
||
### (b) Explicit "you shall not see" list — **AVKREFTET as a live defect, BEKREFTET as uncodified**
|
||
|
||
Phase 5 feeds reviewers exactly four things: the Phase-3 unified diff, the triage map, the
|
||
brief path, and the rule catalogue (`commands/trekreview.md`, Phase 5 input list).
|
||
Implementation plans and commit messages are **not** fed; `git log` appears only to compute
|
||
the diff range and to print a suggested narrowing command (`commands/trekreview.md:132`,
|
||
`:181`). Again: correct today, unprotected tomorrow.
|
||
|
||
### (c) Fail-closed verdicts — **BEKREFTET. This is the one real defect.**
|
||
|
||
`agents/review-coordinator.md` computes the verdict mechanically from the findings that
|
||
*survive* filtering:
|
||
|
||
- **Pass 2** (HubSpot judge) and **Pass 3** (Cloudflare reasonableness) **drop** findings
|
||
that fail their tests (`agents/review-coordinator.md:87`, `:101`).
|
||
- **Pass 4** counts the survivors: `BLOCKER ≥ 1 → BLOCK`; else `MAJOR ≥ 1 → WARN`; else
|
||
`ALLOW` (`:132`). "Verdict is mechanical — never override."
|
||
|
||
There is no third state. Three consequences follow:
|
||
|
||
1. **A finding that cannot be substantiated is dropped, and a dropped finding contributes
|
||
zero — which moves the verdict toward ALLOW.** Pass 3's non-existent-file test ("`file`
|
||
does not exist in the working tree AND the diff does not show it as `(new file)`") has
|
||
no inconclusive branch: unresolvable and refuted are treated identically. This is
|
||
structurally the failure V2 attributes to its source case, where "not E2E testable"
|
||
passed as approved.
|
||
2. **A reviewer that returns nothing produces the same result.** Phase 5 has no
|
||
empty-return handling — grep for `empty|no findings|did not return|abort` over
|
||
`commands/trekreview.md` returns 0 (positive control: `Phase 5` = 7 hits, `reviewer` =
|
||
24 hits, so the file and the query are both live). Zero findings from a silent reviewer
|
||
is indistinguishable from zero findings from a clean diff. This is the same surface as
|
||
the idle-agent hole recorded in STATE's open decisions.
|
||
3. **One fail-closed branch already exists, and it is the pattern to copy.** When reviewer
|
||
output fails schema validation, Phase 5 allows 2 bounded re-asks and then *stops*: "do
|
||
not feed unvalidated findings to the coordinator." That is the correct shape, applied to
|
||
one failure mode only.
|
||
|
||
**The defect is codified, not merely prose — which makes it testable.** The four passes
|
||
exist deterministically in `lib/review/coordinator-contract.mjs`, and
|
||
`computeVerdict(findings)` (`:184`–`:193`) counts *only* the findings handed to it.
|
||
`runContract` hands it `reasoned.kept` (`:206`); `suppressed` and `skipped` are returned in
|
||
the result object but **carry no weight in the verdict**. So a dropped finding is
|
||
arithmetically identical to a finding that never existed.
|
||
|
||
**Minimal fix shape (not implemented here):** an `unverified` bucket that is neither kept
|
||
nor dropped, plus one rule — *a non-empty `unverified` bucket forbids ALLOW* — and a
|
||
Phase-5 check that a reviewer actually returned. Both are additive; neither touches the
|
||
severity catalogue or the existing thresholds. Because the logic is deterministic ("No LLM,
|
||
no network, no time, no randomness", `:23`) and already has
|
||
`tests/lib/coordinator-contract.test.mjs`, this can be driven test-first under the Iron
|
||
Law — a failing test asserting that an unverifiable BLOCKER-severity finding cannot yield
|
||
ALLOW is writable before any production change.
|
||
|
||
### (d) Presume-failure framing — **ENDRET**
|
||
|
||
Both reviewers are already adversarially framed: "Adversarial reviewer" in each
|
||
`description`, "You never praise", "You never say 'looks good'"
|
||
(`agents/code-correctness-reviewer.md:36`, `agents/brief-conformance-reviewer.md:35`). What
|
||
is absent is specifically the presume-failure formulation V2 names ("this agent was lazy —
|
||
find out why"). This is a wording change with no measurement behind it in the corpus; treat
|
||
it as optional polish, not a gap.
|
||
|
||
---
|
||
|
||
## Tiltak 3 — Lightweight lane — **ENDRET** (narrower gap than stated, already tracked)
|
||
|
||
A lightweight path exists: `--quick` on both `/trekbrief` and `/trekplan`, and
|
||
`/trekplan --quick` already skips the Phase-5 exploration swarm — "Skip agent swarm; use
|
||
lightweight Glob/Grep scan and go directly to planning + adversarial review"
|
||
(`commands/trekplan.md:122`). The same code-path is reachable a second way: a brief
|
||
carrying `effort == 'low'` activates it without the flag (`:885`). What it does **not** do
|
||
is bypass the 2.2 ceremony —
|
||
`commands/trekbrief.md:126` states the framing question "is asked even in `--quick` mode",
|
||
and a 2.2 brief still requires `framing` and a `## TL;DR` (`:472`, `:474`).
|
||
|
||
So the accurate gap is not "no lightweight path" but "the lightweight path still pays the
|
||
2.2 toll". Already carried as open operator decision 1 in STATE — this is not a discovery.
|
||
|
||
---
|
||
|
||
## Tiltak 4 — Definition-of-done as an object — **BEKREFTET**
|
||
|
||
`commands/trekbrief.md` contains no notion of evidence, proof, verifier, residual risk, or
|
||
next-step owner (grep for `evidence|proof|screenshot|verifier|residual risk|owner` → 0;
|
||
positive control: `Success Criteri` = 5 hits). The gap is real.
|
||
|
||
**Cost note V2 does not carry:** the brief schema is **Handover 1**, a public contract
|
||
(`docs/HANDOVER-CONTRACTS.md`). Adding required DoD fields is a breaking change for
|
||
downstream consumers, i.e. a `brief_version` bump with a gate, not an edit. That moves it
|
||
out of "cheap win" and into planned work.
|
||
|
||
---
|
||
|
||
## Tiltak 5 — Prune, don't build — **mostly ALREADY DECIDED**
|
||
|
||
- **`synthesis-agent`**: already dormant and labelled as such (`agents/synthesis-agent.md:3`),
|
||
with the measurement that justified it (`docs/T1-synthesis-poc-results.md`, Δ≈0,
|
||
DECLINED). Retiring versus keeping was decided in favour of keeping it as a
|
||
re-measurable building block. V2 recommends a decision that has been made and recorded.
|
||
- **24 all-opus agents**: 24 of 24 agent files carry `model: opus` (verified by count).
|
||
This is an **operator pin**, not drift — commit `40d8742` "pin all sub-agents to Opus
|
||
permanently (operator request)" — and the reconsideration V2 asks for already happened
|
||
and is written down in `docs/voyage-vs-cc-balance-analysis.md` §10. Re-opening it is an
|
||
operator decision, not an analysis task.
|
||
- **`gemini-bridge`**: **BEKREFTET**, and the cheapest real item in this tiltak. It is
|
||
already flagged `THIN_WRAP` / `DROP→NATIVE` in the same analysis (§V09, lines 81, 121,
|
||
149) *and* it is broken at the engine (the `gemini-mcp` server fails deterministically on
|
||
an SDK/API mismatch). Fix-or-drop is decidable today.
|
||
|
||
---
|
||
|
||
## Tiltak 6 — LSP as a harness component — **BEKREFTET**
|
||
|
||
`LSP` returns 0 hits across `commands/`, `agents/`, `lib/`, `docs/`. Voyage navigates with
|
||
Glob/Grep, as V2 says. One qualifier: LSP is a harness-level capability available to
|
||
agents, not a voyage feature — adoption is mostly permitting and prompting agents to use
|
||
it, not code in this repo. Value is highest in large target repos, which is where voyage's
|
||
exploration swarm actually runs.
|
||
|
||
---
|
||
|
||
## Bundle-gap
|
||
|
||
Checked against `bundle/concepts/docs-en-workflows.md`
|
||
(`source_sha 363819ed9ec325275ca22023f6bb6b98fbbf6fcc12db0667478c5207e117751b`,
|
||
timestamp 2026-08-15), plus `docs-en-agents.md` and `docs-en-sub-agents.md`.
|
||
|
||
**Confirmed by the bundle:**
|
||
- The **2.1.154 floor** for dynamic workflows, on paid plans.
|
||
- The **main-context relief mechanism**: "A workflow script holds the loop, the branching,
|
||
and the intermediate results itself, so Claude's context holds only the final answer"
|
||
(`:38`); "Intermediate results stay in script variables instead of landing in Claude's
|
||
context" (`:316`). V2's mechanism claim is sound — it is the *magnitude* that is unmeasured.
|
||
- **Per-stage model routing** (`:375`–`:378`).
|
||
- **Cost caps**: agent caps, size guideline, and a `Large workflow` warning above 25 agents
|
||
or 1.5M projected tokens (`:368`).
|
||
|
||
**Where V2 overstates, and the bundle wins:**
|
||
- **"tokenbudsjetter" (token budgets).** The page documents *caps and size guidelines*, not
|
||
a token budget — `budget` returns 0 hits on the page. V2's warning to "set an explicit
|
||
token budget" has no documented primitive to point at in the official docs.
|
||
- **"automatisk retry".** 0 hits. Not documented on the page.
|
||
|
||
**Limitation V2 omits, material for this repo:** resumability is **session-scoped** —
|
||
"Resume works within the same Claude Code session. If you exit Claude Code while a workflow
|
||
is running, the next session starts the workflow fresh" (`:360`). Further, stopping mid
|
||
fan-out re-runs every agent that started after the stopped one (`:354`). Voyage is
|
||
explicitly a *multi-session* tool, so resumability is a weaker argument here than V2 implies.
|
||
|
||
**Gap proper (feedback toward wiki v1.0):**
|
||
1. The mirrored docs describe no programmatic **token-budget** primitive for workflows,
|
||
while the Workflow tool's own runtime surface does expose a budget derived from an
|
||
operator token directive. If that surface is real and stable, it is missing from the
|
||
mirror. Flagged as a discrepancy, not asserted as a doc error.
|
||
2. Nothing in the mirrored pages quantifies what a workflow's **final return** costs the
|
||
orchestrating session. The relief mechanism is described qualitatively only, so a
|
||
consumer cannot size the benefit from the docs alone — which is precisely why the
|
||
measurement below cannot be replaced by reading.
|
||
|
||
---
|
||
|
||
## Recommended order (operator decides; nothing started)
|
||
|
||
1. **Fail-closed coordinator (tiltak 2c)** — smallest surface, highest value, no
|
||
dependencies, and it closes a defect rather than adding a feature.
|
||
2. **Measure Δ main-context on the `/trekplan` Phase-5 swarm (tiltak 1)** — Arm A vs Arm B
|
||
against the pre-registered bars in `docs/T1-cc26-delegated-orchestration.md`. The port
|
||
decision follows the number; it does not precede it.
|
||
3. **`gemini-bridge` fix-or-drop (tiltak 5)** — already analysed, currently broken, cheap.
|
||
|
||
**Deferred with reasons:** tiltak 3 (already open operator decision 1) · tiltak 4 (requires
|
||
a `brief_version` bump against a public contract) · tiltak 5's opus/synthesis items
|
||
(decided; operator-pinned) · tiltak 6 (harness-level, not repo code).
|
||
|
||
---
|
||
|
||
## Verification log
|
||
|
||
| Claim | How verified |
|
||
|---|---|
|
||
| `/trekreview` Workflow port exists and was measured | `commands/trekreview.md:270`; `docs/T2-bakeoff-results.md` §Full run |
|
||
| +4.4% is **subagent** tokens, main context uncounted | `docs/T2-bakeoff-results.md:154` (verbatim) |
|
||
| Δ main-context ≈ 0 applies to Phase 7 only | `docs/T1-synthesis-poc-results.md:1`, `:100` |
|
||
| Measurement design already exists | `docs/T1-cc26-delegated-orchestration.md:118`, `:125` |
|
||
| Reviewers are not forks | `grep subagent_type\|fork commands/ agents/` → only `Explore` at `commands/trekplan.md:501` |
|
||
| Plans/commit messages not fed to reviewers | Phase 5 input list; `commands/trekreview.md:132`, `:181` |
|
||
| Verdict is computed from survivors only | `agents/review-coordinator.md:87`, `:101`, `:132` |
|
||
| Same rule codified deterministically in `lib/` | `lib/review/coordinator-contract.mjs:184`–`:193`, `:206`; existing test `tests/lib/coordinator-contract.test.mjs` |
|
||
| No empty-return handling in Phase 5 | grep → 0, positive control `Phase 5`=7, `reviewer`=24 |
|
||
| Existing fail-closed branch on schema failure | `commands/trekreview.md`, Phase 5 bounded-retry paragraph |
|
||
| `--quick` does not bypass 2.2 | `commands/trekbrief.md:126`, `:472`, `:474` |
|
||
| No DoD fields in trekbrief | grep → 0, positive control `Success Criteri`=5 |
|
||
| 24/24 agents pinned opus, by operator | file count 24/24; `git log 40d8742` |
|
||
| gemini-bridge already flagged for drop | `docs/voyage-vs-cc-balance-analysis.md:81`, `:121`, `:149` |
|
||
| No LSP usage | `grep -rn LSP commands/ agents/ lib/ docs/` → 0 |
|
||
| Workflow floor, relief mechanism, model routing, caps | `bundle/concepts/docs-en-workflows.md:19`, `:38`, `:316`, `:375`, `:368` |
|
||
| No documented token budget or retry in the bundle | `grep -i budget\|retry` on that page → 0 relevant |
|
||
| Resume is session-scoped | `bundle/concepts/docs-en-workflows.md:354`, `:360` |
|
||
| Fork inherits full conversation context | `bundle/concepts/docs-en-agents.md:37` |
|
||
|
||
**Not verified / stated as unverified:** V2's underlying video-corpus claims (the StrongDM
|
||
bad-merge case, the 5.5-hour bake-off, the "$200 plan in 30 minutes" workflow) were not
|
||
independently checked — they are cited here as V2's evidence, not as this repo's findings.
|