chore(voyage): S34 — V30 economy-profile self-declares experimental (uncalibrated Jaccard floor)
The economy profile's cross-tier Jaccard floor (0.55) rests on parked-synthetic fixtures; empirical Step-17 calibration is v4.2-budget-gated ($60–120, unauthorized). Fork resolved as label-not-calibrate: the experimental status, previously prose-only in docs/profiles.md, now lives in the profile data and is machine-checked. No new user-facing capability — honest labeling + a guard. - lib/profiles/economy.yaml: add `experimental: true` (with rationale comment). - lib/validators/profile-validator.mjs: recognize `experimental` as an OPTIONAL boolean; non-boolean → PROFILE_INVALID_ENUM. Absent ⇒ tier is stable (premium/balanced unaffected, profile_version stays 1.0 — additive). - README.md + docs/operations.md + docs/profiles.md: flag the `economy` table row "⚠ Experimental (uncalibrated Jaccard floor)". - tests/synthetic/profile-jaccard-calibration.md + analysis §6/§10 + backlog plan §S34: cross-reference the marker; mark V30/S34 done. +5 tests (739 → 744, 742/2/0): economy declares experimental:true; premium and balanced do not; validator rejects non-boolean experimental; every profile-doc economy row is flagged; the flag tracks the calibration's parked-synthetic status (must drop in the same change that lands real calibration). Closes the balance backlog (4/4, S31–S34). claude plugin validate green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LqBYc8Ltrk7LipyJmGxXiB
This commit is contained in:
parent
2df0cbb372
commit
213cf388de
10 changed files with 146 additions and 13 deletions
|
|
@ -55,13 +55,15 @@ Bundles the three items that D1–D3 turned into documentation:
|
|||
- **Verify:** `node --test`; counts/model claims coherent across all docs; `agent-frontmatter.test.mjs` unchanged + green (no model change).
|
||||
- **Depends-on:** none (settles the inventory baseline).
|
||||
|
||||
### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent]
|
||||
### S34 — V30: economy-profile Jaccard calibration · [CODE · non-breaking · pin-adjacent] ✅ DONE (2026-06-20)
|
||||
- **Goal:** the `economy`-profile Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step-17 calibration deferred). Either run the calibration against real fixtures and replace the floor, **or** clearly relabel `economy` as experimental/uncalibrated in docs + the profile.
|
||||
- **Files:** `lib/parsers/profile-jaccard.mjs`, `tests/synthetic/profile-jaccard-calibration.md`, `docs/profiles.md`, `docs/operations.md`.
|
||||
- **TDD:** a test pinning either the calibrated floor or the explicit experimental label.
|
||||
- **Verify:** `node --test`; profile docs unambiguous about economy's status.
|
||||
- **Resolution: Path B — label, not calibrate.** The empirical run is v4.2-budget-gated ($60–120, unauthorized), so the fork is forced. `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks the optional boolean); README + `docs/operations.md` + `docs/profiles.md` flag the `economy` row; the calibration doc cross-references the marker. (`lib/parsers/profile-jaccard.mjs` was untouched — it was listed for the calibration path only.)
|
||||
- **TDD:** 5 tests — economy declares `experimental: true`, premium/balanced do not, validator rejects non-boolean `experimental`, every profile-doc economy row flagged experimental, and the flag tracks the calibration's `parked-synthetic` status. Baseline 739 → **744 (742/2/0)**.
|
||||
- **Verify:** ✅ `node --test` green; ✅ `claude plugin validate` (1 accepted warning); economy's status now unambiguous + machine-checked.
|
||||
- **Depends-on:** none (lowest urgency; placed last).
|
||||
|
||||
**Backlog complete — 4/4 sessions shipped (S31–S34). All 8 §6 items disposed; 2 real code changes (V15 export-trim, V30 economy-label), the rest doc/audit.**
|
||||
|
||||
---
|
||||
|
||||
## Verification (plan-level)
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ Three built-in model profiles plus operator-defined `<custom>.yaml`. Each profil
|
|||
|
||||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||||
|---------|-------|----------|------|---------|--------|----------|----------|
|
||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
|
||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; high-confidence small-scope tasks (operator-opt-in via `--profile economy`) |
|
||||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (operator-opt-in via `--profile balanced`) |
|
||||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase. Default since 2026-05-13 operator request; also the hardcoded resolver default returned by `resolveProfile()` in `lib/profiles/resolver.mjs` |
|
||||
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ Three pre-defined tiers ship with v4.1, located at
|
|||
|
||||
| Profile | Brief | Research | Plan | Execute | Review | Continue | Use case |
|
||||
|---------|-------|----------|------|---------|--------|----------|----------|
|
||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | Lowest cost; small-scope tasks where you have high confidence the brief is right |
|
||||
| `economy` | sonnet | sonnet | sonnet | sonnet | sonnet | sonnet | ⚠ **Experimental** (uncalibrated Jaccard floor) — lowest cost; small-scope tasks where you have high confidence the brief is right |
|
||||
| `balanced` | sonnet | sonnet | opus | sonnet | opus | sonnet | Mixed — opus where reasoning depth pays off (plan synthesis + adversarial review); opt-in via `--profile balanced` |
|
||||
| `premium` (default) | opus | opus | opus | opus | opus | opus | Maximum quality — Opus on every phase + external research on (default since the 2026-05-13 operator decision) |
|
||||
|
||||
|
|
@ -24,11 +24,15 @@ quality matters most — Plan synthesis + Review — and sonnet everywhere else)
|
|||
or `--profile economy` (sonnet everywhere) when cost or latency matters more
|
||||
than depth.
|
||||
|
||||
`economy` is *strictly experimental* in v4.1. The cross-tier Jaccard
|
||||
floor (0.55) is grounded in parked-synthetic fixtures, not empirical
|
||||
runs (Step 17 calibration was deferred — see
|
||||
`tests/synthetic/profile-jaccard-calibration.md`). If you observe
|
||||
economy-plan quality regressions, fall back to `balanced`.
|
||||
`economy` is *strictly experimental* in v4.1, and says so in the profile
|
||||
data itself: `lib/profiles/economy.yaml` carries `experimental: true`. The
|
||||
cross-tier Jaccard floor (0.55) is grounded in parked-synthetic fixtures, not
|
||||
empirical runs (Step 17 calibration was deferred — see
|
||||
`tests/synthetic/profile-jaccard-calibration.md`). The flag is pinned: it must
|
||||
stay `true` while the calibration status is `parked-synthetic`, and must be
|
||||
dropped in the same change that lands empirical calibration
|
||||
(`status: empirical`). If you observe economy-plan quality regressions, fall
|
||||
back to `balanced`.
|
||||
|
||||
## Model & effort axes
|
||||
|
||||
|
|
|
|||
|
|
@ -158,7 +158,7 @@ All items are **operator-gated** (scope-guard: analysis only). Tagged **[non-bre
|
|||
|
||||
### Tier 3 — surfaced risks (no action mandated)
|
||||
|
||||
7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly.
|
||||
7. **[non-breaking][pinned]** **V30** — the `economy` profile's Jaccard floor (0.55) is grounded in parked synthetic fixtures (Step 17 calibration deferred). It is self-declared experimental; either calibrate or label clearly. → **RESOLVED (S34): label, not calibrate.** Empirical calibration is v4.2-budget-gated ($60–120, unauthorized); instead `lib/profiles/economy.yaml` now carries `experimental: true` (validator type-checks it) and every profile doc flags the `economy` row. Pinned in `profile-validator.test.mjs` + `doc-consistency.test.mjs` — the flag must track the calibration's `parked-synthetic` status.
|
||||
8. **[non-breaking]** **V01** — minor: delegate the literal Q&A turn-taking to AskUserQuestion rather than a hand-rolled selection loop (internal hygiene; does not change the framing-gate or reviewer contract).
|
||||
|
||||
### Explicitly NOT recommended
|
||||
|
|
@ -221,6 +221,16 @@ workflow and are not spawned as sub-agents. Of the 21 spawnable, one —
|
|||
`docs/T1-synthesis-poc-results.md`). Reconciled across `README.md`,
|
||||
`CLAUDE.md`, and this doc; pinned in `tests/lib/doc-consistency.test.mjs` (S33).
|
||||
|
||||
**V30 outcome (S34, the lone Tier-3 code item).** The economy-calibration fork
|
||||
also resolved conservatively: **label, do not calibrate.** The empirical run is
|
||||
v4.2-budget-gated ($60–120, unauthorized) and was never in scope here. Instead
|
||||
`economy`'s experimental status — previously prose-only in `docs/profiles.md` —
|
||||
now lives in the profile **data** (`lib/profiles/economy.yaml experimental: true`,
|
||||
type-checked by the profile-validator) and on every profile-doc `economy` row,
|
||||
pinned so the flag tracks the calibration's `parked-synthetic` status and is
|
||||
dropped in the same change that lands real calibration. This closes the backlog:
|
||||
all 8 §6 items disposed across S31–S34 (2 code: V15+V30; the rest doc/audit).
|
||||
|
||||
**Still explicitly NOT done** (out of scope, per the plan): no `gemini-bridge`
|
||||
removal (D1), no exporter deletion (D2), no model downgrade (D3), no Handover-1
|
||||
change, no Workflow substrate swap (CC-27 declined), no delegated-orchestration
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue