chore(voyage): S34 — V30 economy-profile self-declares experimental (uncalibrated Jaccard floor)

The economy profile's cross-tier Jaccard floor (0.55) rests on parked-synthetic
fixtures; empirical Step-17 calibration is v4.2-budget-gated ($60–120,
unauthorized). Fork resolved as label-not-calibrate: the experimental status,
previously prose-only in docs/profiles.md, now lives in the profile data and is
machine-checked. No new user-facing capability — honest labeling + a guard.

- lib/profiles/economy.yaml: add `experimental: true` (with rationale comment).
- lib/validators/profile-validator.mjs: recognize `experimental` as an OPTIONAL
  boolean; non-boolean → PROFILE_INVALID_ENUM. Absent ⇒ tier is stable
  (premium/balanced unaffected, profile_version stays 1.0 — additive).
- README.md + docs/operations.md + docs/profiles.md: flag the `economy` table
  row "⚠ Experimental (uncalibrated Jaccard floor)".
- tests/synthetic/profile-jaccard-calibration.md + analysis §6/§10 + backlog
  plan §S34: cross-reference the marker; mark V30/S34 done.

+5 tests (739 → 744, 742/2/0): economy declares experimental:true; premium and
balanced do not; validator rejects non-boolean experimental; every profile-doc
economy row is flagged; the flag tracks the calibration's parked-synthetic
status (must drop in the same change that lands real calibration).

Closes the balance backlog (4/4, S31–S34). claude plugin validate green.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
Kjell Tore Guttormsen 2026-06-20 10:18:38 +02:00
commit 213cf388de
10 changed files with 146 additions and 13 deletions

View file

@ -755,7 +755,7 @@ Three built-in model profiles plus operator-defined `<custom>.yaml` (drop in `li
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|---------|-------|----------|------|---------|--------|----------|----------|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) |
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (opt-in via `--profile economy`) |
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (opt-in via `--profile balanced`) |
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase (default since the 2026-05-13 operator decision) |

View file

@ -55,13 +55,15 @@ Bundles the three items that D1D3 turned into documentation:
- **Verify:** `node --test`; counts/model claims coherent across all docs; `agent-frontmatter.test.mjs` unchanged + green (no model change).
- **Depends-on:** none (settles the inventory baseline).
### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent]
### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent] ✅ DONE (2026-06-20)
- **Goal:** the `economy`-profile Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step-17 calibration deferred). Either run the calibration against real fixtures and replace the floor, **or** clearly relabel `economy` as experimental/uncalibrated in docs + the profile.
- **Files:** `lib/parsers/profile-jaccard.mjs`, `tests/synthetic/profile-jaccard-calibration.md`, `docs/profiles.md`, `docs/operations.md`.
- **TDD:** a test pinning either the calibrated floor or the explicit experimental label.
- **Verify:** `node --test`; profile docs unambiguous about economy's status.
- **Resolution: Path B — label, not calibrate.** The empirical run is v4.2-budget-gated ($60120, unauthorized), so the fork is forced. `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks the optional boolean); README + `docs/operations.md` + `docs/profiles.md` flag the `economy` row; the calibration doc cross-references the marker. (`lib/parsers/profile-jaccard.mjs` was untouched — it was listed for the calibration path only.)
- **TDD:** 5 tests — economy declares `experimental: true`, premium/balanced do not, validator rejects non-boolean `experimental`, every profile-doc economy row flagged experimental, and the flag tracks the calibration's `parked-synthetic` status. Baseline 739 → **744 (742/2/0)**.
- **Verify:** `node --test` green; ✅ `claude plugin validate` (1 accepted warning); economy's status now unambiguous + machine-checked.
- **Depends-on:** none (lowest urgency; placed last).
**Backlog complete — 4/4 sessions shipped (S31S34). All 8 §6 items disposed; 2 real code changes (V15 export-trim, V30 economy-label), the rest doc/audit.**
---
## Verification (plan-level)

View file

@ -32,7 +32,7 @@ Three built-in model profiles plus operator-defined `<custom>.yaml`. Each profil
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|---------|-------|----------|------|---------|--------|----------|----------|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (operator-opt-in via `--profile balanced`) |
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase. Default since 2026-05-13 operator request; also the hardcoded resolver default returned by `resolveProfile()` in `lib/profiles/resolver.mjs` |

View file

@ -11,7 +11,7 @@ Three pre-defined tiers ship with v4.1, located at
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|---------|-------|----------|------|---------|--------|----------|----------|
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; small-scope tasks where you have high confidence the brief is right |
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | **Experimental** (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right |
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` |
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) |
@ -24,11 +24,15 @@ quality matters most — Plan synthesis + Review — and sonnet everywhere else)
or `--profile economy` (sonnet everywhere) when cost or latency matters more
than depth.
`economy` is *strictly experimental* in v4.1. The cross-tier Jaccard
floor (0.55) is grounded in parked-synthetic fixtures, not empirical
runs (Step 17 calibration was deferred — see
`tests/synthetic/profile-jaccard-calibration.md`). If you observe
economy-plan quality regressions, fall back to `balanced`.
`economy` is *strictly experimental* in v4.1, and says so in the profile
data itself: `lib/profiles/economy.yaml` carries `experimental: true`. The
cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not
empirical runs (Step 17 calibration was deferred — see
`tests/synthetic/profile-jaccard-calibration.md`). The flag is pinned: it must
stay `true` while the calibration status is `parked-synthetic`, and must be
dropped in the same change that lands empirical calibration
(`status: empirical`). If you observe economy-plan quality regressions, fall
back to `balanced`.
## Model & effort axes

View file

@ -158,7 +158,7 @@ All items are **operator-gated** (scope-guard: analysis only). Tagged **[non-bre
### Tier 3 — surfaced risks (no action mandated)
7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly.
7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly.**RESOLVED (S34): label, not calibrate.** Empirical calibration is v4.2-budget-gated ($60120, unauthorized); instead `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks it) and every profile doc flags the `economy` row. Pinned in `profile-validator.test.mjs` + `doc-consistency.test.mjs` — the flag must track the calibration's `parked-synthetic` status.
8. **[non-breaking]** **V01** — minor: delegate the literal Q&A turn-taking to AskUserQuestion rather than a hand-rolled selection loop (internal hygiene; does not change the framing-gate or reviewer contract).
### Explicitly NOT recommended
@ -221,6 +221,16 @@ workflow and are not spawned as sub-agents. Of the 21 spawnable, one —
`docs/T1-synthesis-poc-results.md`). Reconciled across `README.md`,
`CLAUDE.md`, and this doc; pinned in `tests/lib/doc-consistency.test.mjs` (S33).
**V30 outcome (S34, the lone Tier-3 code item).** The economy-calibration fork
also resolved conservatively: **label, do not calibrate.** The empirical run is
v4.2-budget-gated ($60120, unauthorized) and was never in scope here. Instead
`economy`'s experimental status — previously prose-only in `docs/profiles.md`
now lives in the profile **data** (`lib/profiles/economy.yaml experimental: true`,
type-checked by the profile-validator) and on every profile-doc `economy` row,
pinned so the flag tracks the calibration's `parked-synthetic` status and is
dropped in the same change that lands real calibration. This closes the backlog:
all 8 §6 items disposed across S31S34 (2 code: V15+V30; the rest doc/audit).
**Still explicitly NOT done** (out of scope, per the plan): no `gemini-bridge`
removal (D1), no exporter deletion (D2), no model downgrade (D3), no Handover-1
change, no Workflow substrate swap (CC-27 declined), no delegated-orchestration

View file

@ -18,4 +18,9 @@ parallel_agents_min: 2
parallel_agents_max: 3
external_research_enabled: false
brief_reviewer_iter_cap: 1
# Experimental: the cross-tier Jaccard floor (0.55) rests on parked-synthetic
# fixtures — empirical Step-17 calibration is deferred to v4.2. Drop this flag
# in the SAME change that lands calibration (status: empirical). See
# tests/synthetic/profile-jaccard-calibration.md and docs/profiles.md.
experimental: true
---

View file

@ -9,6 +9,10 @@
// parallel_agents_max : number (≥ parallel_agents_min)
// external_research_enabled : boolean
// brief_reviewer_iter_cap : number (≥ 1)
// experimental : boolean (OPTIONAL) — true marks a tier whose
// gating constants are not yet empirically
// calibrated (e.g. economy's parked-synthetic
// cross-tier Jaccard floor). Absent ⇒ stable.
//
// Issue codes:
// PROFILE_MISSING_FIELD — required top-level frontmatter field absent
@ -99,6 +103,12 @@ export function validateProfileContent(text, opts = {}) {
`brief_reviewer_iter_cap must be number (got ${typeof fm.brief_reviewer_iter_cap})`));
}
// Optional field: `experimental` (boolean). Absent ⇒ tier is stable.
if ('experimental' in fm && typeof fm.experimental !== 'boolean') {
errors.push(issue('PROFILE_INVALID_ENUM',
`experimental must be boolean (got ${typeof fm.experimental})`));
}
// phase_models validation
if ('phase_models' in fm) {
if (!Array.isArray(fm.phase_models)) {

View file

@ -961,6 +961,46 @@ test('S15: profile tables encode each built-in yaml phase_models exactly', () =>
}
});
// --- S34 (V30) — economy is self-declared experimental until the cross-tier
// Jaccard floor (0.55) is empirically calibrated (Step-17 calibration deferred
// to v4.2). The status must be visible in BOTH the profile data
// (lib/profiles/economy.yaml `experimental: true`) and every profile doc's
// economy row. Fix the SOURCE doc/profile, not these pins.
test('S34: economy profile row is flagged experimental across all profile docs', () => {
for (const doc of PROFILE_DOCS) {
const body = read(doc);
const row = body.split('\n').find((l) => /^\|\s*`economy`/.test(l));
assert.ok(row, `${doc}: profile table missing an \`economy\` row`);
assert.match(
row,
/experimental/i,
`${doc}: the \`economy\` table row must flag it experimental ` +
`(uncalibrated Jaccard floor, S34/V30) — matches lib/profiles/economy.yaml experimental: true`,
);
}
});
test('S34: economy experimental flag agrees with parked-synthetic calibration status', () => {
// economy.yaml experimental:true is honest ONLY while calibration is still
// parked-synthetic. When empirical calibration lands (status: empirical),
// drop experimental:true in the SAME change — this pin enforces the linkage.
const eco = parseDocument(read('lib/profiles/economy.yaml')).parsed.frontmatter;
const cal = parseDocument(read('tests/synthetic/profile-jaccard-calibration.md')).parsed.frontmatter;
if (cal.status === 'parked-synthetic') {
assert.equal(
eco.experimental,
true,
'while calibration is parked-synthetic, economy.yaml must stay experimental: true',
);
} else {
assert.notEqual(
eco.experimental,
true,
'calibration is no longer parked-synthetic — drop economy experimental: true in the same change',
);
}
});
test('S15: README cost prose does not resurrect the false Sonnet-swarm claim', () => {
// All 24 sub-agents are model: opus (operator-pinned, commit 40d8742) and the
// model is uniform per phase — there is no "Opus orchestrates / Sonnet runs

View file

@ -14,6 +14,14 @@ ramp_target: v4.2
## Status: PARKED-SYNTHETIC
> **Profile linkage (S34/V30).** Because this calibration is still
> parked-synthetic, `lib/profiles/economy.yaml` carries `experimental: true`
> and every profile doc flags the `economy` row experimental. When the
> empirical calibration in *How to replace* below lands (`status: empirical`),
> drop `experimental: true` in the **same change** — the linkage is guarded by
> `tests/lib/doc-consistency.test.mjs` ("economy experimental flag agrees with
> parked-synthetic calibration status").
Empirical Jaccard calibration was deferred from v4.1 because the four
required `/trekplan` invocations cost an estimated $60-120 of LLM-budget
that was not authorized for the v4.1-execute-4b session. Per Step 17

View file

@ -148,3 +148,57 @@ test('PROFILE_REQUIRED_PHASES export drift-pin', () => {
'PROFILE_REQUIRED_PHASES contract drift — pin contract',
);
});
// S34 (V30) — economy carries an explicit `experimental` marker.
// The cross-tier Jaccard floor (0.55) rests on parked-synthetic fixtures;
// empirical Step-17 calibration is deferred to v4.2 ($60-120 LLM budget,
// unauthorized). Until a real calibration lands, `economy` is self-declared
// experimental in the profile DATA — not only in prose — so the status
// survives a docs rewrite. See tests/synthetic/profile-jaccard-calibration.md.
test('S34: economy.yaml declares experimental: true (uncalibrated Jaccard floor)', () => {
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', 'economy.yaml'));
assert.equal(r.valid, true, `economy.yaml should validate: ${JSON.stringify(r.errors)}`);
assert.equal(r.parsed.frontmatter.experimental, true,
'economy.yaml must carry experimental: true until the Jaccard floor is empirically calibrated (S34/V30)');
});
test('S34: premium + balanced are NOT flagged experimental', () => {
for (const name of ['premium', 'balanced']) {
const r = validateProfile(join(REPO_ROOT, 'lib', 'profiles', `${name}.yaml`));
assert.equal(r.valid, true, `${name}.yaml should validate: ${JSON.stringify(r.errors)}`);
assert.notEqual(r.parsed.frontmatter.experimental, true,
`${name}.yaml must not be flagged experimental`);
}
});
test('S34: validator type-checks experimental as boolean (PROFILE_INVALID_ENUM)', () => {
const bad = `---
profile_version: "1.0"
name: bad-experimental
phase_models:
- phase: brief
model: sonnet
- phase: research
model: sonnet
- phase: plan
model: sonnet
- phase: execute
model: sonnet
- phase: review
model: sonnet
- phase: continue
model: sonnet
parallel_agents_min: 2
parallel_agents_max: 3
external_research_enabled: false
brief_reviewer_iter_cap: 1
experimental: "yes"
---
`;
const r = validateProfileContent(bad);
assert.equal(r.valid, false, 'experimental: "yes" (string) must be rejected');
const found = r.errors.find(e => e.code === 'PROFILE_INVALID_ENUM' && /experimental/.test(e.message));
assert.ok(found, `expected PROFILE_INVALID_ENUM for experimental, got: ${JSON.stringify(r.errors)}`);
assert.match(found.message, /boolean/);
});