docs(brief): assess VURDERING-V2 §5.12 against ground truth
External assessment from claude-playlist-corpus, verified per tiltak against
code in this repo and against the claude-code-llm-wiki OKF bundle (bundle wins
on feature claims). No implementation — scope-guard.
Verdicts across the six tiltak:
1. Workflow port — split, not global. /trekreview is already ported (opt-in
--workflow) and measured fidelity-equivalent; /trekplan and /trekresearch are
unported, but the claimed benefit is UNMEASURED, not refuted. T2's +4.4% is
subagent tokens and states main context is uncounted; T1's Δ≈0 covers Phase 7
synthesis only. The measurement design already exists in T1-cc26. Measure
before porting.
2. Holdout — (a) and (b) hold today but are uncodified; (d) is already partly
there; (c) fail-closed is a REAL defect. computeVerdict counts only kept
findings, so a dropped or unreturned finding is arithmetically identical to
one that never existed. Codified in lib/review/coordinator-contract.mjs, so
it is testable test-first.
3. Lightweight lane — narrower than stated: --quick exists but does not bypass
the 2.2 framing gate. Already an open operator decision.
4. Definition-of-done as object — confirmed gap, but it is a Handover 1
breaking change (brief_version bump), not a cheap edit.
5. Prune — mostly already decided: synthesis-agent dormant by measurement, the
24 opus pins are an operator pin with a decision record (40d8742), and
gemini-bridge is already flagged DROP→NATIVE.
6. LSP — confirmed absent.
Bundle-gap: V2 overstates two Workflow primitives (token budgets, automatic
retry — neither documented on the mirrored page) and omits that resumability is
session-scoped, which matters for a multi-session tool.
This commit is contained in:
parent
4f90aa8af0
commit
9ee120dd7e
1 changed files with 292 additions and 0 deletions
292
docs/BRIEF-vurdering-v2.md
Normal file
292
docs/BRIEF-vurdering-v2.md
Normal file
|
|
@ -0,0 +1,292 @@
|
|||
# BRIEF — external assessment (VURDERING-V2 §5.12), verified against this repo
|
||||
|
||||
**Status:** assessment complete. **No implementation in this session** (scope-guard).
|
||||
**Source under review:** `claude-playlist-corpus/docs/VURDERING-V2.md` §5.12 (+ §3 G1–G8, §4),
|
||||
transferred via that repo's `docs/OVERFORING-V2.md` §5. It is an external recommendation
|
||||
built on 442 video analyses; its repo facts come from a subagent survey (2026-07-17) that
|
||||
is **not** re-verified and in which at least one error was already demonstrated.
|
||||
|
||||
**Method.** Every asserted GAP is marked **BEKREFTET** (confirmed) / **AVKREFTET**
|
||||
(refuted) / **ENDRET** (true in altered form) only after checking it against code in this
|
||||
repo. Every Claude Code *feature* claim is checked against the OKF bundle at
|
||||
`claude-code-llm-wiki/bundle/`, index-first — **where V2 and the bundle disagree, the
|
||||
bundle wins**. Absences are positive-controlled before being reported (a grep that finds
|
||||
nothing is a measurement, not a fact).
|
||||
|
||||
---
|
||||
|
||||
## TL;DR
|
||||
|
||||
Of six recommended tiltak, **one is a real defect** (fail-open verdict computation),
|
||||
**one is unmeasured rather than unresolved** (the Workflow port), **three are already
|
||||
decided or already partly built**, and **one is a cheap confirmed gap**.
|
||||
|
||||
The single most consequential correction: V2 presents the Workflow port as the answer to
|
||||
this repo's "main-context relief is unmeasured" caveat. The bundle confirms the
|
||||
*mechanism*, but **this repo has still never measured main-context relief on a large
|
||||
fan-out** — the one bake-off that exists measured *subagent* tokens and says so. The port
|
||||
is therefore neither refuted nor justified. The next step is a measurement, not a port,
|
||||
and the measurement design already exists in `docs/T1-cc26-delegated-orchestration.md`.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 1 — Port orchestration to dynamic Workflows
|
||||
|
||||
**Verdict: split per command. Do not issue one global verdict.**
|
||||
|
||||
### `/trekreview` — **ENDRET** (already ported, and measured)
|
||||
|
||||
The port exists: opt-in `--workflow`, `scripts/trekreview-armB.workflow.mjs`, documented at
|
||||
`commands/trekreview.md:270` (*§ Phase 5–6 via the Workflow substrate*). The S10 bake-off
|
||||
(`docs/T2-bakeoff-results.md`, ≥3 runs/arm) found it **fidelity-equivalent**: verdict-match
|
||||
1.0, issue-coverage 100%, `(file,rule_key)` jaccard ≥ within-arm. It stays opt-in because
|
||||
the Workflow tool raises the consumer floor to Claude Code 2.1.154+, and the prose path
|
||||
keeps the lower floor.
|
||||
|
||||
### `/trekplan`, `/trekresearch` — **BEKREFTET as unported, UNMEASURED as beneficial**
|
||||
|
||||
Both still orchestrate their swarms from main context (`commands/trekplan.md:406`,
|
||||
`commands/trekresearch.md:388`). That part of V2 is correct.
|
||||
|
||||
What is **not** established is V2's stated effect — that Workflow answers the
|
||||
main-context-relief caveat. Two measurements exist in this repo and **neither measures it**:
|
||||
|
||||
| Measurement | What it actually measured | Bearing on the relief claim |
|
||||
|---|---|---|
|
||||
| `docs/T2-bakeoff-results.md:154` | **Subagent** tokens: Arm A median 92.8k vs Arm B 96.9k ⇒ **+4.4%** | None. The same line states Arm A's main-context hand-orchestration cost is **uncounted**. |
|
||||
| `docs/T1-synthesis-poc-results.md` | Δ **main-context** ≈ **0%** — for delegating *Phase 7 synthesis alone* | Negative, but out of scope: that doc itself names a **Phase-5 redesign** as the prerequisite for any relief. |
|
||||
|
||||
So the Phase-5 swarm — the case V2 is actually recommending — has never been measured in
|
||||
either arm, in either command.
|
||||
|
||||
**The measurement already has a design.** `docs/T1-cc26-delegated-orchestration.md:118`
|
||||
defines the gate metric (Δ main-context tokens resident in the main session at
|
||||
plan-complete, from stream-json `usage`) with pre-registered bars: adopt at ≥30%, reject
|
||||
below 15%. Running Arm A vs Arm B once against the trekplan swarm settles this.
|
||||
|
||||
**Cost framing must be corrected.** The bundle states plainly that a workflow run "can use
|
||||
meaningfully more tokens than working through the same task in conversation"
|
||||
(`bundle/concepts/docs-en-workflows.md:364`) — consistent with the measured +4.4%. The port
|
||||
cannot be justified as a cost saving. **Main-context headroom is the only defensible
|
||||
motive, and it is exactly the unmeasured quantity.**
|
||||
|
||||
**Recommendation:** measure before porting. Treat V2's tiltak 1 as a *hypothesis with a
|
||||
ready-made experiment*, not as an approved change.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 2 — Codify the holdout rules
|
||||
|
||||
**Verdict: (a) and (b) satisfied in practice but uncodified · (c) BEKREFTET, a real defect ·
|
||||
(d) ENDRET.** This is the tiltak with genuine substance.
|
||||
|
||||
### (a) Reviewers must never be forks — **AVKREFTET as a live defect, BEKREFTET as uncodified**
|
||||
|
||||
Reviewers are spawned as plain subagents (`voyage:brief-conformance-reviewer`,
|
||||
`voyage:code-correctness-reviewer`). `subagent_type: "fork"` appears nowhere in `commands/`
|
||||
or `agents/`; the only `subagent_type` in the tree is `"Explore"` at
|
||||
`commands/trekplan.md:501`, an exploration spawn, not a review spawn. The bundle confirms
|
||||
the rule is real — a forked subagent "inherits your full conversation context instead of
|
||||
starting fresh" (`bundle/concepts/docs-en-agents.md:37`). Voyage satisfies it by habit;
|
||||
nothing written forbids a future edit from breaking it.
|
||||
|
||||
### (b) Explicit "you shall not see" list — **AVKREFTET as a live defect, BEKREFTET as uncodified**
|
||||
|
||||
Phase 5 feeds reviewers exactly four things: the Phase-3 unified diff, the triage map, the
|
||||
brief path, and the rule catalogue (`commands/trekreview.md`, Phase 5 input list).
|
||||
Implementation plans and commit messages are **not** fed; `git log` appears only to compute
|
||||
the diff range and to print a suggested narrowing command (`commands/trekreview.md:132`,
|
||||
`:181`). Again: correct today, unprotected tomorrow.
|
||||
|
||||
### (c) Fail-closed verdicts — **BEKREFTET. This is the one real defect.**
|
||||
|
||||
`agents/review-coordinator.md` computes the verdict mechanically from the findings that
|
||||
*survive* filtering:
|
||||
|
||||
- **Pass 2** (HubSpot judge) and **Pass 3** (Cloudflare reasonableness) **drop** findings
|
||||
that fail their tests (`agents/review-coordinator.md:87`, `:101`).
|
||||
- **Pass 4** counts the survivors: `BLOCKER ≥ 1 → BLOCK`; else `MAJOR ≥ 1 → WARN`; else
|
||||
`ALLOW` (`:132`). "Verdict is mechanical — never override."
|
||||
|
||||
There is no third state. Three consequences follow:
|
||||
|
||||
1. **A finding that cannot be substantiated is dropped, and a dropped finding contributes
|
||||
zero — which moves the verdict toward ALLOW.** Pass 3's non-existent-file test ("`file`
|
||||
does not exist in the working tree AND the diff does not show it as `(new file)`") has
|
||||
no inconclusive branch: unresolvable and refuted are treated identically. This is
|
||||
structurally the failure V2 attributes to its source case, where "not E2E testable"
|
||||
passed as approved.
|
||||
2. **A reviewer that returns nothing produces the same result.** Phase 5 has no
|
||||
empty-return handling — grep for `empty|no findings|did not return|abort` over
|
||||
`commands/trekreview.md` returns 0 (positive control: `Phase 5` = 7 hits, `reviewer` =
|
||||
24 hits, so the file and the query are both live). Zero findings from a silent reviewer
|
||||
is indistinguishable from zero findings from a clean diff. This is the same surface as
|
||||
the idle-agent hole recorded in STATE's open decisions.
|
||||
3. **One fail-closed branch already exists, and it is the pattern to copy.** When reviewer
|
||||
output fails schema validation, Phase 5 allows 2 bounded re-asks and then *stops*: "do
|
||||
not feed unvalidated findings to the coordinator." That is the correct shape, applied to
|
||||
one failure mode only.
|
||||
|
||||
**The defect is codified, not merely prose — which makes it testable.** The four passes
|
||||
exist deterministically in `lib/review/coordinator-contract.mjs`, and
|
||||
`computeVerdict(findings)` (`:184`–`:193`) counts *only* the findings handed to it.
|
||||
`runContract` hands it `reasoned.kept` (`:206`); `suppressed` and `skipped` are returned in
|
||||
the result object but **carry no weight in the verdict**. So a dropped finding is
|
||||
arithmetically identical to a finding that never existed.
|
||||
|
||||
**Minimal fix shape (not implemented here):** an `unverified` bucket that is neither kept
|
||||
nor dropped, plus one rule — *a non-empty `unverified` bucket forbids ALLOW* — and a
|
||||
Phase-5 check that a reviewer actually returned. Both are additive; neither touches the
|
||||
severity catalogue or the existing thresholds. Because the logic is deterministic ("No LLM,
|
||||
no network, no time, no randomness", `:23`) and already has
|
||||
`tests/lib/coordinator-contract.test.mjs`, this can be driven test-first under the Iron
|
||||
Law — a failing test asserting that an unverifiable BLOCKER-severity finding cannot yield
|
||||
ALLOW is writable before any production change.
|
||||
|
||||
### (d) Presume-failure framing — **ENDRET**
|
||||
|
||||
Both reviewers are already adversarially framed: "Adversarial reviewer" in each
|
||||
`description`, "You never praise", "You never say 'looks good'"
|
||||
(`agents/code-correctness-reviewer.md:36`, `agents/brief-conformance-reviewer.md:35`). What
|
||||
is absent is specifically the presume-failure formulation V2 names ("this agent was lazy —
|
||||
find out why"). This is a wording change with no measurement behind it in the corpus; treat
|
||||
it as optional polish, not a gap.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 3 — Lightweight lane — **ENDRET** (narrower gap than stated, already tracked)
|
||||
|
||||
A lightweight path exists: `--quick` on both `/trekbrief` and `/trekplan`, and
|
||||
`/trekplan --quick` already skips the Phase-5 exploration swarm
|
||||
(`commands/trekplan.md:122`, `:885`). What it does **not** do is bypass the 2.2 ceremony —
|
||||
`commands/trekbrief.md:126` states the framing question "is asked even in `--quick` mode",
|
||||
and a 2.2 brief still requires `framing` and a `## TL;DR` (`:472`, `:474`).
|
||||
|
||||
So the accurate gap is not "no lightweight path" but "the lightweight path still pays the
|
||||
2.2 toll". Already carried as open operator decision 1 in STATE — this is not a discovery.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 4 — Definition-of-done as an object — **BEKREFTET**
|
||||
|
||||
`commands/trekbrief.md` contains no notion of evidence, proof, verifier, residual risk, or
|
||||
next-step owner (grep for `evidence|proof|screenshot|verifier|residual risk|owner` → 0;
|
||||
positive control: `Success Criteri` = 5 hits). The gap is real.
|
||||
|
||||
**Cost note V2 does not carry:** the brief schema is **Handover 1**, a public contract
|
||||
(`docs/HANDOVER-CONTRACTS.md`). Adding required DoD fields is a breaking change for
|
||||
downstream consumers, i.e. a `brief_version` bump with a gate, not an edit. That moves it
|
||||
out of "cheap win" and into planned work.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 5 — Prune, don't build — **mostly ALREADY DECIDED**
|
||||
|
||||
- **`synthesis-agent`**: already dormant and labelled as such (`agents/synthesis-agent.md:3`),
|
||||
with the measurement that justified it (`docs/T1-synthesis-poc-results.md`, Δ≈0,
|
||||
DECLINED). Retiring versus keeping was decided in favour of keeping it as a
|
||||
re-measurable building block. V2 recommends a decision that has been made and recorded.
|
||||
- **24 all-opus agents**: 24 of 24 agent files carry `model: opus` (verified by count).
|
||||
This is an **operator pin**, not drift — commit `40d8742` "pin all sub-agents to Opus
|
||||
permanently (operator request)" — and the reconsideration V2 asks for already happened
|
||||
and is written down in `docs/voyage-vs-cc-balance-analysis.md` §10. Re-opening it is an
|
||||
operator decision, not an analysis task.
|
||||
- **`gemini-bridge`**: **BEKREFTET**, and the cheapest real item in this tiltak. It is
|
||||
already flagged `THIN_WRAP` / `DROP→NATIVE` in the same analysis (§V09, lines 81, 121,
|
||||
149) *and* it is broken at the engine (the `gemini-mcp` server fails deterministically on
|
||||
an SDK/API mismatch). Fix-or-drop is decidable today.
|
||||
|
||||
---
|
||||
|
||||
## Tiltak 6 — LSP as a harness component — **BEKREFTET**
|
||||
|
||||
`LSP` returns 0 hits across `commands/`, `agents/`, `lib/`, `docs/`. Voyage navigates with
|
||||
Glob/Grep, as V2 says. One qualifier: LSP is a harness-level capability available to
|
||||
agents, not a voyage feature — adoption is mostly permitting and prompting agents to use
|
||||
it, not code in this repo. Value is highest in large target repos, which is where voyage's
|
||||
exploration swarm actually runs.
|
||||
|
||||
---
|
||||
|
||||
## Bundle-gap
|
||||
|
||||
Checked against `bundle/concepts/docs-en-workflows.md`
|
||||
(`source_sha 363819ed9ec325275ca22023f6bb6b98fbbf6fcc12db0667478c5207e117751b`,
|
||||
timestamp 2026-08-15), plus `docs-en-agents.md` and `docs-en-sub-agents.md`.
|
||||
|
||||
**Confirmed by the bundle:**
|
||||
- The **2.1.154 floor** for dynamic workflows, on paid plans.
|
||||
- The **main-context relief mechanism**: "A workflow script holds the loop, the branching,
|
||||
and the intermediate results itself, so Claude's context holds only the final answer"
|
||||
(`:38`); "Intermediate results stay in script variables instead of landing in Claude's
|
||||
context" (`:316`). V2's mechanism claim is sound — it is the *magnitude* that is unmeasured.
|
||||
- **Per-stage model routing** (`:375`–`:378`).
|
||||
- **Cost caps**: agent caps, size guideline, and a `Large workflow` warning above 25 agents
|
||||
or 1.5M projected tokens (`:368`).
|
||||
|
||||
**Where V2 overstates, and the bundle wins:**
|
||||
- **"tokenbudsjetter" (token budgets).** The page documents *caps and size guidelines*, not
|
||||
a token budget — `budget` returns 0 hits on the page. V2's warning to "set an explicit
|
||||
token budget" has no documented primitive to point at in the official docs.
|
||||
- **"automatisk retry".** 0 hits. Not documented on the page.
|
||||
|
||||
**Limitation V2 omits, material for this repo:** resumability is **session-scoped** —
|
||||
"Resume works within the same Claude Code session. If you exit Claude Code while a workflow
|
||||
is running, the next session starts the workflow fresh" (`:360`). Further, stopping mid
|
||||
fan-out re-runs every agent that started after the stopped one (`:354`). Voyage is
|
||||
explicitly a *multi-session* tool, so resumability is a weaker argument here than V2 implies.
|
||||
|
||||
**Gap proper (feedback toward wiki v1.0):**
|
||||
1. The mirrored docs describe no programmatic **token-budget** primitive for workflows,
|
||||
while the Workflow tool's own runtime surface does expose a budget derived from an
|
||||
operator token directive. If that surface is real and stable, it is missing from the
|
||||
mirror. Flagged as a discrepancy, not asserted as a doc error.
|
||||
2. Nothing in the mirrored pages quantifies what a workflow's **final return** costs the
|
||||
orchestrating session. The relief mechanism is described qualitatively only, so a
|
||||
consumer cannot size the benefit from the docs alone — which is precisely why the
|
||||
measurement below cannot be replaced by reading.
|
||||
|
||||
---
|
||||
|
||||
## Recommended order (operator decides; nothing started)
|
||||
|
||||
1. **Fail-closed coordinator (tiltak 2c)** — smallest surface, highest value, no
|
||||
dependencies, and it closes a defect rather than adding a feature.
|
||||
2. **Measure Δ main-context on the `/trekplan` Phase-5 swarm (tiltak 1)** — Arm A vs Arm B
|
||||
against the pre-registered bars in `docs/T1-cc26-delegated-orchestration.md`. The port
|
||||
decision follows the number; it does not precede it.
|
||||
3. **`gemini-bridge` fix-or-drop (tiltak 5)** — already analysed, currently broken, cheap.
|
||||
|
||||
**Deferred with reasons:** tiltak 3 (already open operator decision 1) · tiltak 4 (requires
|
||||
a `brief_version` bump against a public contract) · tiltak 5's opus/synthesis items
|
||||
(decided; operator-pinned) · tiltak 6 (harness-level, not repo code).
|
||||
|
||||
---
|
||||
|
||||
## Verification log
|
||||
|
||||
| Claim | How verified |
|
||||
|---|---|
|
||||
| `/trekreview` Workflow port exists and was measured | `commands/trekreview.md:270`; `docs/T2-bakeoff-results.md` §Full run |
|
||||
| +4.4% is **subagent** tokens, main context uncounted | `docs/T2-bakeoff-results.md:154` (verbatim) |
|
||||
| Δ main-context ≈ 0 applies to Phase 7 only | `docs/T1-synthesis-poc-results.md:1`, `:100` |
|
||||
| Measurement design already exists | `docs/T1-cc26-delegated-orchestration.md:118`, `:125` |
|
||||
| Reviewers are not forks | `grep subagent_type\|fork commands/ agents/` → only `Explore` at `commands/trekplan.md:501` |
|
||||
| Plans/commit messages not fed to reviewers | Phase 5 input list; `commands/trekreview.md:132`, `:181` |
|
||||
| Verdict is computed from survivors only | `agents/review-coordinator.md:87`, `:101`, `:132` |
|
||||
| Same rule codified deterministically in `lib/` | `lib/review/coordinator-contract.mjs:184`–`:193`, `:206`; existing test `tests/lib/coordinator-contract.test.mjs` |
|
||||
| No empty-return handling in Phase 5 | grep → 0, positive control `Phase 5`=7, `reviewer`=24 |
|
||||
| Existing fail-closed branch on schema failure | `commands/trekreview.md`, Phase 5 bounded-retry paragraph |
|
||||
| `--quick` does not bypass 2.2 | `commands/trekbrief.md:126`, `:472`, `:474` |
|
||||
| No DoD fields in trekbrief | grep → 0, positive control `Success Criteri`=5 |
|
||||
| 24/24 agents pinned opus, by operator | file count 24/24; `git log 40d8742` |
|
||||
| gemini-bridge already flagged for drop | `docs/voyage-vs-cc-balance-analysis.md:81`, `:121`, `:149` |
|
||||
| No LSP usage | `grep -rn LSP commands/ agents/ lib/ docs/` → 0 |
|
||||
| Workflow floor, relief mechanism, model routing, caps | `bundle/concepts/docs-en-workflows.md:19`, `:38`, `:316`, `:375`, `:368` |
|
||||
| No documented token budget or retry in the bundle | `grep -i budget\|retry` on that page → 0 relevant |
|
||||
| Resume is session-scoped | `bundle/concepts/docs-en-workflows.md:354`, `:360` |
|
||||
| Fork inherits full conversation context | `bundle/concepts/docs-en-agents.md:37` |
|
||||
|
||||
**Not verified / stated as unverified:** V2's underlying video-corpus claims (the StrongDM
|
||||
bad-merge case, the 5.5-hour bake-off, the "$200 plan in 30 minutes" workflow) were not
|
||||
independently checked — they are cited here as V2's evidence, not as this repo's findings.
|
||||
Loading…
Add table
Add a link
Reference in a new issue