From 213cf388def61edb1608ede389e53a05d50cf370 Mon Sep 17 00:00:00 2001 From: Kjell Tore Guttormsen Date: Sat, 20 Jun 2026 10:18:38 +0200 Subject: [PATCH] =?UTF-8?q?chore(voyage):=20S34=20=E2=80=94=20V30=20econom?= =?UTF-8?q?y-profile=20self-declares=20experimental=20(uncalibrated=20Jacc?= =?UTF-8?q?ard=20floor)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The economy profile's cross-tier Jaccard floor (0.55) rests on parked-synthetic fixtures; empirical Step-17 calibration is v4.2-budget-gated ($60–120, unauthorized). Fork resolved as label-not-calibrate: the experimental status, previously prose-only in docs/profiles.md, now lives in the profile data and is machine-checked. No new user-facing capability — honest labeling + a guard. - lib/profiles/economy.yaml: add `experimental: true` (with rationale comment). - lib/validators/profile-validator.mjs: recognize `experimental` as an OPTIONAL boolean; non-boolean → PROFILE_INVALID_ENUM. Absent ⇒ tier is stable (premium/balanced unaffected, profile_version stays 1.0 — additive). - README.md + docs/operations.md + docs/profiles.md: flag the `economy` table row "⚠ Experimental (uncalibrated Jaccard floor)". - tests/synthetic/profile-jaccard-calibration.md + analysis §6/§10 + backlog plan §S34: cross-reference the marker; mark V30/S34 done. +5 tests (739 → 744, 742/2/0): economy declares experimental:true; premium and balanced do not; validator rejects non-boolean experimental; every profile-doc economy row is flagged; the flag tracks the calibration's parked-synthetic status (must drop in the same change that lands real calibration). Closes the balance backlog (4/4, S31–S34). claude plugin validate green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB --- README.md | 2 +- docs/balance-backlog-plan.md | 10 ++-- docs/operations.md | 2 +- docs/profiles.md | 16 +++--- docs/voyage-vs-cc-balance-analysis.md | 12 ++++- lib/profiles/economy.yaml | 5 ++ lib/validators/profile-validator.mjs | 10 ++++ tests/lib/doc-consistency.test.mjs | 40 ++++++++++++++ .../synthetic/profile-jaccard-calibration.md | 8 +++ tests/validators/profile-validator.test.mjs | 54 +++++++++++++++++++ 10 files changed, 146 insertions(+), 13 deletions(-) diff --git a/README.md b/README.md index 57c899f..8ddaf1a 100644 --- a/README.md +++ b/README.md @@ -755,7 +755,7 @@ Three built-in model profiles plus operator-defined `.yaml` (drop in `li | Profile | Brief | Research | Plan | Execute | Review | Continue | Use case | |---------|-------|----------|------|---------|--------|----------|----------| -| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) | +| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) | | `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (opt-in via `--profile balanced`) | | `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase (default since the 2026-05-13 operator decision) | diff --git a/docs/balance-backlog-plan.md b/docs/balance-backlog-plan.md index 0c64a6e..84ba2ae 100644 --- a/docs/balance-backlog-plan.md +++ b/docs/balance-backlog-plan.md @@ -55,13 +55,15 @@ Bundles the three items that D1–D3 turned into documentation: - **Verify:** `node --test`; counts/model claims coherent across all docs; `agent-frontmatter.test.mjs` unchanged + green (no model change). - **Depends-on:** none (settles the inventory baseline). -### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent] +### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent] ✅ DONE (2026-06-20) - **Goal:** the `economy`-profile Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step-17 calibration deferred). Either run the calibration against real fixtures and replace the floor, **or** clearly relabel `economy` as experimental/uncalibrated in docs + the profile. -- **Files:** `lib/parsers/profile-jaccard.mjs`, `tests/synthetic/profile-jaccard-calibration.md`, `docs/profiles.md`, `docs/operations.md`. -- **TDD:** a test pinning either the calibrated floor or the explicit experimental label. -- **Verify:** `node --test`; profile docs unambiguous about economy's status. +- **Resolution: Path B — label, not calibrate.** The empirical run is v4.2-budget-gated ($60–120, unauthorized), so the fork is forced. `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks the optional boolean); README + `docs/operations.md` + `docs/profiles.md` flag the `economy` row; the calibration doc cross-references the marker. (`lib/parsers/profile-jaccard.mjs` was untouched — it was listed for the calibration path only.) +- **TDD:** 5 tests — economy declares `experimental: true`, premium/balanced do not, validator rejects non-boolean `experimental`, every profile-doc economy row flagged experimental, and the flag tracks the calibration's `parked-synthetic` status. Baseline 739 → **744 (742/2/0)**. +- **Verify:** ✅ `node --test` green; ✅ `claude plugin validate` (1 accepted warning); economy's status now unambiguous + machine-checked. - **Depends-on:** none (lowest urgency; placed last). +**Backlog complete — 4/4 sessions shipped (S31–S34). All 8 §6 items disposed; 2 real code changes (V15 export-trim, V30 economy-label), the rest doc/audit.** + --- ## Verification (plan-level) diff --git a/docs/operations.md b/docs/operations.md index e42da4b..129cd7b 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -32,7 +32,7 @@ Three built-in model profiles plus operator-defined `.yaml`. Each profil | Profile | Brief | Research | Plan | Execute | Review | Continue | Use case | |---------|-------|----------|------|---------|--------|----------|----------| -| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) | +| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) | | `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (operator-opt-in via `--profile balanced`) | | `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase. Default since 2026-05-13 operator request; also the hardcoded resolver default returned by `resolveProfile()` in `lib/profiles/resolver.mjs` | diff --git a/docs/profiles.md b/docs/profiles.md index 0b69ef6..ea757f8 100644 --- a/docs/profiles.md +++ b/docs/profiles.md @@ -11,7 +11,7 @@ Three pre-defined tiers ship with v4.1, located at | Profile | Brief | Research | Plan | Execute | Review | Continue | Use case | |---------|-------|----------|------|---------|--------|----------|----------| -| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; small-scope tasks where you have high confidence the brief is right | +| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right | | `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` | | `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) | @@ -24,11 +24,15 @@ quality matters most — Plan synthesis + Review — and sonnet everywhere else) or `--profile economy` (sonnet everywhere) when cost or latency matters more than depth. -`economy` is *strictly experimental* in v4.1. The cross-tier Jaccard -floor (0.55) is grounded in parked-synthetic fixtures, not empirical -runs (Step 17 calibration was deferred — see -`tests/synthetic/profile-jaccard-calibration.md`). If you observe -economy-plan quality regressions, fall back to `balanced`. +`economy` is *strictly experimental* in v4.1, and says so in the profile +data itself: `lib/profiles/economy.yaml` carries `experimental: true`. The +cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not +empirical runs (Step 17 calibration was deferred — see +`tests/synthetic/profile-jaccard-calibration.md`). The flag is pinned: it must +stay `true` while the calibration status is `parked-synthetic`, and must be +dropped in the same change that lands empirical calibration +(`status: empirical`). If you observe economy-plan quality regressions, fall +back to `balanced`. ## Model & effort axes diff --git a/docs/voyage-vs-cc-balance-analysis.md b/docs/voyage-vs-cc-balance-analysis.md index a938c18..0b073e3 100644 --- a/docs/voyage-vs-cc-balance-analysis.md +++ b/docs/voyage-vs-cc-balance-analysis.md @@ -158,7 +158,7 @@ All items are **operator-gated** (scope-guard: analysis only). Tagged **[non-bre ### Tier 3 — surfaced risks (no action mandated) -7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly. +7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly. → **RESOLVED (S34): label, not calibrate.** Empirical calibration is v4.2-budget-gated ($60–120, unauthorized); instead `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks it) and every profile doc flags the `economy` row. Pinned in `profile-validator.test.mjs` + `doc-consistency.test.mjs` — the flag must track the calibration's `parked-synthetic` status. 8. **[non-breaking]** **V01** — minor: delegate the literal Q&A turn-taking to AskUserQuestion rather than a hand-rolled selection loop (internal hygiene; does not change the framing-gate or reviewer contract). ### Explicitly NOT recommended @@ -221,6 +221,16 @@ workflow and are not spawned as sub-agents. Of the 21 spawnable, one — `docs/T1-synthesis-poc-results.md`). Reconciled across `README.md`, `CLAUDE.md`, and this doc; pinned in `tests/lib/doc-consistency.test.mjs` (S33). +**V30 outcome (S34, the lone Tier-3 code item).** The economy-calibration fork +also resolved conservatively: **label, do not calibrate.** The empirical run is +v4.2-budget-gated ($60–120, unauthorized) and was never in scope here. Instead +`economy`'s experimental status — previously prose-only in `docs/profiles.md` — +now lives in the profile **data** (`lib/profiles/economy.yaml experimental: true`, +type-checked by the profile-validator) and on every profile-doc `economy` row, +pinned so the flag tracks the calibration's `parked-synthetic` status and is +dropped in the same change that lands real calibration. This closes the backlog: +all 8 §6 items disposed across S31–S34 (2 code: V15+V30; the rest doc/audit). + **Still explicitly NOT done** (out of scope, per the plan): no `gemini-bridge` removal (D1), no exporter deletion (D2), no model downgrade (D3), no Handover-1 change, no Workflow substrate swap (CC-27 declined), no delegated-orchestration diff --git a/lib/profiles/economy.yaml b/lib/profiles/economy.yaml index b5639cc..3a30e70 100644 --- a/lib/profiles/economy.yaml +++ b/lib/profiles/economy.yaml @@ -18,4 +18,9 @@ parallel_agents_min: 2 parallel_agents_max: 3 external_research_enabled: false brief_reviewer_iter_cap: 1 +# Experimental: the cross-tier Jaccard floor (0.55) rests on parked-synthetic +# fixtures — empirical Step-17 calibration is deferred to v4.2. Drop this flag +# in the SAME change that lands calibration (status: empirical). See +# tests/synthetic/profile-jaccard-calibration.md and docs/profiles.md. +experimental: true --- diff --git a/lib/validators/profile-validator.mjs b/lib/validators/profile-validator.mjs index c7fec26..37909e8 100644 --- a/lib/validators/profile-validator.mjs +++ b/lib/validators/profile-validator.mjs @@ -9,6 +9,10 @@ // parallel_agents_max : number (≥ parallel_agents_min) // external_research_enabled : boolean // brief_reviewer_iter_cap : number (≥ 1) +// experimental : boolean (OPTIONAL) — true marks a tier whose +// gating constants are not yet empirically +// calibrated (e.g. economy's parked-synthetic +// cross-tier Jaccard floor). Absent ⇒ stable. // // Issue codes: // PROFILE_MISSING_FIELD — required top-level frontmatter field absent @@ -99,6 +103,12 @@ export function validateProfileContent(text, opts = {}) { `brief_reviewer_iter_cap must be number (got ${typeof fm.brief_reviewer_iter_cap})`)); } + // Optional field: `experimental` (boolean). Absent ⇒ tier is stable. + if ('experimental' in fm && typeof fm.experimental !== 'boolean') { + errors.push(issue('PROFILE_INVALID_ENUM', + `experimental must be boolean (got ${typeof fm.experimental})`)); + } + // phase_models validation if ('phase_models' in fm) { if (!Array.isArray(fm.phase_models)) { diff --git a/tests/lib/doc-consistency.test.mjs b/tests/lib/doc-consistency.test.mjs index 450845e..926923c 100644 --- a/tests/lib/doc-consistency.test.mjs +++ b/tests/lib/doc-consistency.test.mjs @@ -961,6 +961,46 @@ test('S15: profile tables encode each built-in yaml phase_models exactly', () => } }); +// --- S34 (V30) — economy is self-declared experimental until the cross-tier +// Jaccard floor (0.55) is empirically calibrated (Step-17 calibration deferred +// to v4.2). The status must be visible in BOTH the profile data +// (lib/profiles/economy.yaml `experimental: true`) and every profile doc's +// economy row. Fix the SOURCE doc/profile, not these pins. +test('S34: economy profile row is flagged experimental across all profile docs', () => { + for (const doc of PROFILE_DOCS) { + const body = read(doc); + const row = body.split('\n').find((l) => /^\|\s*`economy`/.test(l)); + assert.ok(row, `${doc}: profile table missing an \`economy\` row`); + assert.match( + row, + /experimental/i, + `${doc}: the \`economy\` table row must flag it experimental ` + + `(uncalibrated Jaccard floor, S34/V30) — matches lib/profiles/economy.yaml experimental: true`, + ); + } +}); + +test('S34: economy experimental flag agrees with parked-synthetic calibration status', () => { + // economy.yaml experimental:true is honest ONLY while calibration is still + // parked-synthetic. When empirical calibration lands (status: empirical), + // drop experimental:true in the SAME change — this pin enforces the linkage. + const eco = parseDocument(read('lib/profiles/economy.yaml')).parsed.frontmatter; + const cal = parseDocument(read('tests/synthetic/profile-jaccard-calibration.md')).parsed.frontmatter; + if (cal.status === 'parked-synthetic') { + assert.equal( + eco.experimental, + true, + 'while calibration is parked-synthetic, economy.yaml must stay experimental: true', + ); + } else { + assert.notEqual( + eco.experimental, + true, + 'calibration is no longer parked-synthetic — drop economy experimental: true in the same change', + ); + } +}); + test('S15: README cost prose does not resurrect the false Sonnet-swarm claim', () => { // All 24 sub-agents are model: opus (operator-pinned, commit 40d8742) and the // model is uniform per phase — there is no "Opus orchestrates / Sonnet runs diff --git a/tests/synthetic/profile-jaccard-calibration.md b/tests/synthetic/profile-jaccard-calibration.md index 5dbc077..993574a 100644 --- a/tests/synthetic/profile-jaccard-calibration.md +++ b/tests/synthetic/profile-jaccard-calibration.md @@ -14,6 +14,14 @@ ramp_target: v4.2 ## Status: PARKED-SYNTHETIC +> **Profile linkage (S34/V30).** Because this calibration is still +> parked-synthetic, `lib/profiles/economy.yaml` carries `experimental: true` +> and every profile doc flags the `economy` row experimental. When the +> empirical calibration in *How to replace* below lands (`status: empirical`), +> drop `experimental: true` in the **same change** — the linkage is guarded by +> `tests/lib/doc-consistency.test.mjs` ("economy experimental flag agrees with +> parked-synthetic calibration status"). + Empirical Jaccard calibration was deferred from v4.1 because the four required `/trekplan` invocations cost an estimated $60-120 of LLM-budget that was not authorized for the v4.1-execute-4b session. Per Step 17 diff --git a/tests/validators/profile-validator.test.mjs b/tests/validators/profile-validator.test.mjs index c0e5792..37c3806 100644 --- a/tests/validators/profile-validator.test.mjs +++ b/tests/validators/profile-validator.test.mjs @@ -148,3 +148,57 @@ test('PROFILE_REQUIRED_PHASES export drift-pin', () => { 'PROFILE_REQUIRED_PHASES contract drift — pin contract', ); }); + +// S34 (V30) — economy carries an explicit `experimental` marker. +// The cross-tier Jaccard floor (0.55) rests on parked-synthetic fixtures; +// empirical Step-17 calibration is deferred to v4.2 ($60-120 LLM budget, +// unauthorized). Until a real calibration lands, `economy` is self-declared +// experimental in the profile DATA — not only in prose — so the status +// survives a docs rewrite. See tests/synthetic/profile-jaccard-calibration.md. + +test('S34: economy.yaml declares experimental: true (uncalibrated Jaccard floor)', () => { + const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', 'economy.yaml')); + assert.equal(r.valid, true, `economy.yaml should validate: ${JSON.stringify(r.errors)}`); + assert.equal(r.parsed.frontmatter.experimental, true, + 'economy.yaml must carry experimental: true until the Jaccard floor is empirically calibrated (S34/V30)'); +}); + +test('S34: premium + balanced are NOT flagged experimental', () => { + for (const name of ['premium', 'balanced']) { + const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', `${name}.yaml`)); + assert.equal(r.valid, true, `${name}.yaml should validate: ${JSON.stringify(r.errors)}`); + assert.notEqual(r.parsed.frontmatter.experimental, true, + `${name}.yaml must not be flagged experimental`); + } +}); + +test('S34: validator type-checks experimental as boolean (PROFILE_INVALID_ENUM)', () => { + const bad = `--- +profile_version: "1.0" +name: bad-experimental +phase_models: + - phase: brief + model: sonnet + - phase: research + model: sonnet + - phase: plan + model: sonnet + - phase: execute + model: sonnet + - phase: review + model: sonnet + - phase: continue + model: sonnet +parallel_agents_min: 2 +parallel_agents_max: 3 +external_research_enabled: false +brief_reviewer_iter_cap: 1 +experimental: "yes" +--- +`; + const r = validateProfileContent(bad); + assert.equal(r.valid, false, 'experimental: "yes" (string) must be rejected'); + const found = r.errors.find(e => e.code === 'PROFILE_INVALID_ENUM' && /experimental/.test(e.message)); + assert.ok(found, `expected PROFILE_INVALID_ENUM for experimental, got: ${JSON.stringify(r.errors)}`); + assert.match(found.message, /boolean/); +});