fix(storm-measure): check BOTH halves of the activation SC, not just the count delta
The SC asks two things of an effort: high run: that it discovered at least one dimension AND that the dimension list in the output brief is a TRUE SUPERSET of the interview-derived ones. activationCheck computed dimensions - dimensions_baseline >= 1 and returned ok on that alone. A count delta says nothing about membership: a run that dropped two interview dimensions and appended three discovered ones is +1 and passed the check while violating the second half outright. Supersetness was asserted only by Phase 4.5's prose contract that discovery appends - nothing read it. The stats record cannot carry the dimension names that would show it directly: names are free prose, and lib/exporters/field-allowlist.mjs denies prose by omission (question, project_dir, brief_path are already excluded for that reason). So the run attests membership with dimensions_baseline_preserved, a boolean set in a new Phase 4.5 step 5, and the gate refuses to call activation OK without it. An ABSENT attestation fails rather than falling back to the old count-only rule - otherwise legacy rows would keep passing on the defect. Producer side wired end to end: the record format, the measurement-fields prose (five fields -> six), the exporter allowlist, the jsonl-schemas fixture row, and the --activation-check comment in docs/storm-measurement.md. Review finding d2786604458207a5a73478cdcb6a54bbdb92141d (MINOR). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LuGhWAbWyRFBFeemfhxoVv
This commit is contained in:
parent
7d5e4fcbcc
commit
dc245af408
6 changed files with 145 additions and 14 deletions
|
|
@ -421,6 +421,14 @@ agent surfaced that no interview dimension claims.
|
|||
4. **Record the baseline.** Keep the interview-derived count as
|
||||
`dimensions_baseline` so the discovered delta is machine-readable against
|
||||
the final `dimensions` (Phase 8 stats).
|
||||
5. **Attest membership, not just the count.** Set
|
||||
`dimensions_baseline_preserved: true` only if EVERY interview-derived
|
||||
dimension is still on the final list — this phase appends, it never replaces.
|
||||
Set it `false` if any was dropped, merged away, or rewritten. A count delta
|
||||
cannot show this: dropping two interview dimensions and appending three
|
||||
discovered ones is `+1` and still not a superset, which is exactly what the
|
||||
Success Criterion forbids. `storm-measure.mjs --activation-check` reads the
|
||||
field and fails when it is absent, so omitting it is not the silent default.
|
||||
|
||||
Every outbound query generated from a discovered dimension passes
|
||||
`query-privacy-gate.mjs` before it leaves the machine — see the per-turn
|
||||
|
|
@ -737,6 +745,7 @@ Record format (one JSON line):
|
|||
"brief_path": "{brief_destination}",
|
||||
"dimensions": {N},
|
||||
"dimensions_baseline": {N},
|
||||
"dimensions_baseline_preserved": {true|false},
|
||||
"effort": "{low|standard|high}",
|
||||
"conv_turns": {N},
|
||||
"empty_turns": {N},
|
||||
|
|
@ -750,16 +759,20 @@ Record format (one JSON line):
|
|||
}
|
||||
```
|
||||
|
||||
**The five measurement fields (v5.10).** `effort` is the grouping key — the
|
||||
**The six measurement fields (v5.10).** `effort` is the grouping key — the
|
||||
resolved `phase_signal_result.effort` for the `research` phase, a
|
||||
low-cardinality label (`low|standard|high`), and the only axis on which a
|
||||
high-effort run can be compared against a standard one. The other four are
|
||||
numeric: `unique_sources` (distinct sources cited across the brief),
|
||||
high-effort run can be compared against a standard one. Four are numeric:
|
||||
`unique_sources` (distinct sources cited across the brief),
|
||||
`dimensions_baseline` (the interview-derived dimension count, so the Phase 4.5
|
||||
delta against `dimensions` is machine-readable), `conv_turns` (Phase 5 loop
|
||||
turns actually spent), and `empty_turns` (loop turns that returned no findings
|
||||
or no citations). On a standard run the loop never arms, so
|
||||
`dimensions_baseline == dimensions` and both turn counters are `0`.
|
||||
or no citations). The sixth is boolean: `dimensions_baseline_preserved`, the
|
||||
Phase 4.5 attestation (step 5) that every interview dimension survived onto the
|
||||
final list — the count delta cannot show membership, and the dimension NAMES
|
||||
that could are prose the exporter allowlist denies. On a standard run the loop
|
||||
never arms, so `dimensions_baseline == dimensions`, both turn counters are `0`,
|
||||
and `dimensions_baseline_preserved` is `true` (nothing touched the list).
|
||||
|
||||
If `${CLAUDE_PLUGIN_DATA}` is not set or not writable, skip tracking silently.
|
||||
|
||||
|
|
|
|||
|
|
@ -94,8 +94,13 @@ node scripts/storm-measure.mjs --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats
|
|||
# Machine-readable, for a decision record
|
||||
node scripts/storm-measure.mjs --json --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||||
|
||||
# SC activation check — did the latest high-effort run discover any dimension
|
||||
# beyond its interview baseline? Exit 0 = yes.
|
||||
# SC activation check — BOTH halves: did the latest high-effort run discover at
|
||||
# least one dimension, AND is its final list a true SUPERSET of the interview
|
||||
# ones? The second half is read from the run's own `dimensions_baseline_preserved`
|
||||
# attestation, because the dimension NAMES that would show it directly are prose
|
||||
# the exporter allowlist denies. A run that does not attest it FAILS — dropping
|
||||
# two interview dimensions and appending three discovered ones is a +1 count
|
||||
# delta and not a superset. Exit 0 = both halves hold.
|
||||
node scripts/storm-measure.mjs --activation-check --stats "${CLAUDE_PLUGIN_DATA}/trekresearch-stats.jsonl"
|
||||
```
|
||||
|
||||
|
|
|
|||
|
|
@ -34,14 +34,18 @@ const TREKBRIEF_ALLOWED = Object.freeze(new Set([
|
|||
// low-cardinality label (low|standard|high) and the grouping key the
|
||||
// measurement gate is computed on; `unique_sources`, `dimensions_baseline`,
|
||||
// `conv_turns` and `empty_turns` are plain counters. None of them carry prose
|
||||
// or paths.
|
||||
// or paths. `dimensions_baseline_preserved` is a boolean, and it exists BECAUSE
|
||||
// prose is denied here: the activation check needs to know that the final
|
||||
// dimension list is a superset of the interview-derived one, and the dimension
|
||||
// NAMES that would show it directly are free prose that must not reach the
|
||||
// exporter. A boolean attestation carries the fact without the payload.
|
||||
const TREKRESEARCH_ALLOWED = Object.freeze(new Set([
|
||||
'ts', 'slug', 'mode', 'scope', 'engine', 'dimensions', 'agents_local',
|
||||
'agents_external', 'gemini_used', 'confidence', 'contradictions',
|
||||
'open_questions', 'profile', 'parallel_agents',
|
||||
'external_research_enabled', 'profile_source',
|
||||
'effort', 'unique_sources', 'dimensions_baseline', 'conv_turns',
|
||||
'empty_turns',
|
||||
'empty_turns', 'dimensions_baseline_preserved',
|
||||
]));
|
||||
|
||||
// Source: tests/fixtures/jsonl-schemas.md row 3 (trekplan)
|
||||
|
|
|
|||
|
|
@ -185,8 +185,24 @@ export function decideVerdict(sourcesGain, dimensionsGain) {
|
|||
}
|
||||
|
||||
/**
|
||||
* SC activation check: did the most recent `effort: high` run actually discover
|
||||
* dimensions beyond its interview baseline?
|
||||
* SC activation check, BOTH halves.
|
||||
*
|
||||
* The SC asks two things of an `effort: high` run: that it discovered at least
|
||||
* one dimension, AND that the dimension list in the output brief is a TRUE
|
||||
* SUPERSET of the interview-derived ones. This function used to check only
|
||||
* `dimensions - dimensions_baseline >= 1`, which is a count delta and says
|
||||
* nothing about membership — a run that dropped two interview dimensions and
|
||||
* added three discovered ones passed while violating the second half.
|
||||
* Supersetness was asserted only by Phase 4.5's prose contract that discovery
|
||||
* APPENDS; nothing read it.
|
||||
*
|
||||
* The record cannot carry the dimension names: names are free prose, and
|
||||
* lib/exporters/field-allowlist.mjs denies prose by omission. So the run attests
|
||||
* membership with `dimensions_baseline_preserved`, a low-cardinality boolean set
|
||||
* in Phase 4.5, and this gate refuses to call activation OK without it. An
|
||||
* ABSENT attestation is not an attestation — legacy rows fail here rather than
|
||||
* passing on the old count-only rule.
|
||||
*
|
||||
* @param {object[]} records
|
||||
*/
|
||||
export function activationCheck(records) {
|
||||
|
|
@ -201,15 +217,40 @@ export function activationCheck(records) {
|
|||
return { ok: false, reason: 'latest high run lacks dimensions/dimensions_baseline', discovered_dimensions: null };
|
||||
}
|
||||
const discovered = d - b;
|
||||
return {
|
||||
ok: discovered >= 1,
|
||||
const preserved = last.dimensions_baseline_preserved;
|
||||
|
||||
const base = {
|
||||
ts: last.ts ?? null,
|
||||
dimensions: d,
|
||||
dimensions_baseline: b,
|
||||
discovered_dimensions: discovered,
|
||||
dimensions_baseline_preserved: preserved ?? null,
|
||||
conv_turns: Number(last.conv_turns ?? 0),
|
||||
empty_turns: Number(last.empty_turns ?? 0),
|
||||
};
|
||||
|
||||
if (typeof preserved !== 'boolean') {
|
||||
return {
|
||||
...base,
|
||||
ok: false,
|
||||
reason:
|
||||
'latest high run does not attest `dimensions_baseline_preserved`; the SC needs a true ' +
|
||||
'superset of the interview dimensions, and a count delta cannot show membership',
|
||||
};
|
||||
}
|
||||
if (preserved === false) {
|
||||
return {
|
||||
...base,
|
||||
ok: false,
|
||||
reason:
|
||||
`latest high run discovered ${discovered} dimension(s) but did NOT preserve its interview ` +
|
||||
'baseline, so the final list is not a superset of it',
|
||||
};
|
||||
}
|
||||
if (discovered < 1) {
|
||||
return { ...base, ok: false, reason: 'latest high run discovered no dimensions beyond its baseline' };
|
||||
}
|
||||
return { ...base, ok: true };
|
||||
}
|
||||
|
||||
// ---- CLI shim ----------------------------------------------------------------
|
||||
|
|
|
|||
2
tests/fixtures/jsonl-schemas.md
vendored
2
tests/fixtures/jsonl-schemas.md
vendored
|
|
@ -20,7 +20,7 @@
|
|||
| schema_id | fields | writer_path | line_ref | v4.1 additive | PII |
|
||||
|-----------|--------|-------------|----------|---------------|-----|
|
||||
| trekbrief-stats | ts, task, slug, mode, interview_turns, review_iterations, brief_quality, research_topics, auto_research, auto_result, project_dir | commands/trekbrief.md (orchestrator-emit Phase 7) | trekbrief.md:657-672 | profile, phase_models, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekresearch-stats | ts, question, mode, scope, engine, slug, project_dir, brief_path, dimensions, dimensions_baseline, dimensions_baseline_preserved, effort, conv_turns, empty_turns, unique_sources, agents_local, agents_external, gemini_used, confidence, contradictions, open_questions | commands/trekresearch.md (orchestrator-emit Stats tracking) | trekresearch.md:634-676 | profile, phase_models, parallel_agents, external_research_enabled, profile_source | none |
|
||||
| trekplan-stats | ts, task, mode, slug, brief_path, project_dir, codebase_size, codebase_files, agents_deployed, deep_dives, research_briefs_used, research_scout_used, critic_verdict, guardian_verdict, outcome | commands/trekplan.md (orchestrator-emit Phase 12) | trekplan.md:805-826 | profile, phase_models, parallel_agents, profile_source | none |
|
||||
| trekexecute-stats (Phase 9 record) | ts, plan, plan_type, mode, result, steps_total, steps_passed, steps_failed, steps_skipped, failed_at_step | commands/trekexecute.md (orchestrator-emit Phase 9) | trekexecute.md:1479-1494 | profile, phase_models, profile_source | none |
|
||||
| trekexecute-stats (autonomy events) | ts, event, known_event, payload | lib/stats/event-emit.mjs `emit()` | event-emit.mjs:64-86 | payload.profile, payload.phase_models, payload.profile_source | none |
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@ import {
|
|||
partitionEligible,
|
||||
measure,
|
||||
decideVerdict,
|
||||
activationCheck,
|
||||
ADOPT_THRESHOLD,
|
||||
DECLINE_THRESHOLD,
|
||||
} from '../../scripts/storm-measure.mjs';
|
||||
|
|
@ -235,6 +236,73 @@ test('decideVerdict: a null gain is insufficient data, never a decline', () => {
|
|||
assert.equal(decideVerdict(0.4, null), 'insufficient-data');
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// activationCheck — BOTH halves of the SC, not just the count
|
||||
//
|
||||
// The SC requires that an `effort: high` run discovers at least one dimension
|
||||
// AND that the dimension list in the output brief is a TRUE SUPERSET of the
|
||||
// interview-derived ones. activationCheck only computed dimensions -
|
||||
// dimensions_baseline >= 1, so a run that replaced two interview dimensions with
|
||||
// three discovered ones passed the check while violating the SC's second half.
|
||||
// Supersetness was asserted only by Phase 4.5's prose contract that discovery
|
||||
// appends; nothing read it.
|
||||
//
|
||||
// The stats record carries counts, not names — names are free prose and
|
||||
// field-allowlist.mjs denies prose by omission — so the run attests membership
|
||||
// with a low-cardinality boolean instead, and the gate refuses to call
|
||||
// activation OK without it.
|
||||
// ---------------------------------------------------------------------------
|
||||
function highRun(over = {}) {
|
||||
return JSON.stringify({
|
||||
ts: '2026-08-12T00:00:00.000Z',
|
||||
effort: 'high',
|
||||
unique_sources: 13,
|
||||
dimensions: 8,
|
||||
dimensions_baseline: 5,
|
||||
conv_turns: 3,
|
||||
empty_turns: 0,
|
||||
dimensions_baseline_preserved: true,
|
||||
...over,
|
||||
});
|
||||
}
|
||||
|
||||
test('activationCheck: discovery plus a preserved baseline is activation', () => {
|
||||
const r = activationCheck(parseStats(jsonl(highRun())).records);
|
||||
assert.equal(r.ok, true);
|
||||
assert.equal(r.discovered_dimensions, 3);
|
||||
assert.equal(r.dimensions_baseline_preserved, true);
|
||||
});
|
||||
|
||||
test('activationCheck: a REPLACED baseline is not activation, however many were discovered', () => {
|
||||
const r = activationCheck(parseStats(jsonl(
|
||||
highRun({ dimensions: 8, dimensions_baseline: 5, dimensions_baseline_preserved: false }),
|
||||
)).records);
|
||||
assert.equal(r.ok, false, 'a count delta of +3 says nothing about which dimensions survived');
|
||||
assert.match(r.reason, /superset|baseline/i);
|
||||
});
|
||||
|
||||
test('activationCheck: a run that does not attest baseline membership cannot pass', () => {
|
||||
const rec = JSON.parse(highRun());
|
||||
delete rec.dimensions_baseline_preserved;
|
||||
const r = activationCheck(parseStats(jsonl(JSON.stringify(rec))).records);
|
||||
assert.equal(r.ok, false, 'an absent attestation is not an attestation');
|
||||
assert.match(r.reason, /dimensions_baseline_preserved/);
|
||||
});
|
||||
|
||||
test('activationCheck: no discovery is still not activation even with the baseline preserved', () => {
|
||||
const r = activationCheck(parseStats(jsonl(
|
||||
highRun({ dimensions: 5, dimensions_baseline: 5 }),
|
||||
)).records);
|
||||
assert.equal(r.ok, false);
|
||||
assert.equal(r.discovered_dimensions, 0);
|
||||
});
|
||||
|
||||
test('activationCheck: no effort:high run at all is reported as such', () => {
|
||||
const r = activationCheck(parseStats(jsonl(...STANDARD)).records);
|
||||
assert.equal(r.ok, false);
|
||||
assert.match(r.reason, /high/);
|
||||
});
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// pre-registration — the thresholds are the doc's, and the doc is committed first
|
||||
// ---------------------------------------------------------------------------
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue