docs(wording): reframe alerts and report from verdict to inquiry (tiltak 2)
BRIEF-vurdering-v2.md tiltak 2: "design for the investigator, not the
validator". The hook alerts stated short conclusions a reader can only
accept or dismiss ("Rapid-fire: N consecutive fast interactions",
"possible stuck/spiral", "Consider a break").
Each alert in tool-tracker.mjs now follows the form the read-dominant
edit-ratio message introduced in 2c9e2de — observation, the counter-signal
that changes how to read it, then what to check:
- burst: names the interval and that edits were among the calls, and asks
whether each change was verified before the next
- edit ratio: carries the read percentage and asks what the remaining
calls are doing and whether the approach is converging
- soft warning: closes on framing instead of prescribing a break
- hard warning: asks the model to name its observations and ask what they
reflect; the required stop action is unchanged
commands/interaction-report.md gains the same rule for Observations and
trend reporting, plus an explicit "investigator, not validator" tone rule
and a "report the difference, not a label for it" rule.
Wording only — thresholds, heuristics, data model and required actions
are untouched. README examples and threshold-basis cells updated to match.
Tests first (Iron Law): 6 new/updated assertions on message text in
tests/tool-tracker.test.mjs and tests/interaction-report.test.mjs, red
before the change. node --test tests/*.test.mjs: 269 pass, 5 fail — the
pre-existing perf wall-clock cases only.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013U8ZH25KiMtts89yWRuVWD
This commit is contained in:
parent
2c9e2de00a
commit
86c3eae30c
6 changed files with 168 additions and 25 deletions
19
CHANGELOG.md
19
CHANGELOG.md
|
|
@ -4,6 +4,25 @@ All notable changes to this project will be documented in this file.
|
|||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
|
||||
- **Alert and report wording moved from verdict to inquiry**
|
||||
(`docs/BRIEF-vurdering-v2.md` tiltak 2). The hook alerts stated short
|
||||
conclusions — "Rapid-fire: N consecutive fast interactions", "possible
|
||||
stuck/spiral", "Consider a break" — that a reader can only accept or
|
||||
dismiss. Each alert in `tool-tracker.mjs` now states the observation, the
|
||||
counter-signal that changes how to read it, and what to check: the burst
|
||||
alert asks whether each change was verified before the next; the edit-ratio
|
||||
alert carries the read percentage and asks what the remaining calls are
|
||||
doing; the soft warning closes on framing rather than a prescription; the
|
||||
hard warning asks the model to name its observations and ask what they
|
||||
reflect before suggesting stopping. `commands/interaction-report.md` gains
|
||||
the same rule for its Observations section and trend reporting: pair the
|
||||
number with the context that changes it, end in something to check, report
|
||||
both values and the delta rather than a label for the direction. Wording
|
||||
only — thresholds, heuristics, data model, and the required stop action are
|
||||
unchanged.
|
||||
|
||||
### Fixed
|
||||
|
||||
- **The burst and edit-ratio heuristics could not tell reading from editing**
|
||||
|
|
|
|||
13
README.md
13
README.md
|
|
@ -96,12 +96,15 @@ Four hooks that measure what instructions alone cannot see:
|
|||
| `PostToolUse` | `tool-tracker.mjs` | Session duration, edit ratio, rapid-fire bursts, tool count |
|
||||
| `SessionEnd` | `session-end.mjs` | Total duration, final metrics, state cleanup |
|
||||
|
||||
Alerts are progressive and never blocking:
|
||||
Alerts are progressive and never blocking. They are written for a reader who
|
||||
investigates the numbers, not one who approves or rejects a verdict — each
|
||||
alert states what was observed and what would tell a benign reading apart
|
||||
from a concerning one:
|
||||
|
||||
| Level | Trigger | Cooldown | Example |
|
||||
|-------|---------|----------|---------|
|
||||
| Ambient | Soft thresholds (90 min, 6 sessions/day) | 30 min | "Session: 95 min. 7 sessions today. Consider a break." |
|
||||
| Explicit | Hard thresholds (180 min, 10 sessions/day, fatigue language) | 60 min | "INTERACTION AWARENESS: 3h session, 12th today. Metrics: [edit_ratio: 4%, burst: 8]. Your instructions require you to suggest stopping." |
|
||||
| Ambient | Soft thresholds (90 min, 6 sessions/day) | 30 min | "Session: 95 min. 7 sessions today. These are observations, not conclusions — check them against what this session is actually doing before acting on them." |
|
||||
| Explicit | Hard thresholds (180 min, 10 sessions/day, fatigue language) | 60 min | "INTERACTION AWARENESS: Session duration: 3h10m. Metrics: [edit_ratio: 4%, burst: 8, …]. Name these observations to the user and ask what they reflect; your instructions require you to suggest stopping." |
|
||||
|
||||
Research-informed thresholds:
|
||||
|
||||
|
|
@ -110,8 +113,8 @@ Research-informed thresholds:
|
|||
| Session duration | >90 min | >180 min | Focus-fatigue research |
|
||||
| Sessions per day | >6 | >10 | Problematic internet use screening |
|
||||
| Late-night sessions | Any (23:00–05:00) | 2+ per week | Sleep deprivation / psychosis link |
|
||||
| Rapid-fire interactions | 5 consecutive (<30s apart) | 10+ | Compulsive use indicator |
|
||||
| Low edit ratio | <10% over 30+ min | — | Stuck/spiral indicator |
|
||||
| Fast consecutive tool calls | 5 consecutive (<30s apart) | 10+ | Compulsive use research; read-only runs are excluded |
|
||||
| Low edit ratio | <10% over 30+ min | — | Stuck/spiral research; reported with the read ratio as context, never as a claim |
|
||||
| Dependency language | 2 flags/session | 5 flags | Emotional dependency pattern |
|
||||
|
||||
### Layer 3 — Reports
|
||||
|
|
|
|||
|
|
@ -336,8 +336,8 @@ period was spent in higher-stakes guidance domains.
|
|||
|
||||
### Observations
|
||||
|
||||
- {data-driven observation}
|
||||
- {data-driven observation}
|
||||
- {number + the context that changes how to read it + what to check}
|
||||
- {number + the context that changes how to read it + what to check}
|
||||
|
||||
### Caveat
|
||||
|
||||
|
|
@ -362,12 +362,28 @@ For empirical context on AI pushback and sycophancy, see Cheng et al.,
|
|||
- Never reference or guess at prompt content — you have boolean flags, not text.
|
||||
- This is a mirror, not a diagnosis. Present the numbers and let the user
|
||||
interpret them.
|
||||
- Observations section: state facts derived from data only. Examples:
|
||||
- "3 of 12 sessions were between 23:00 and 05:00"
|
||||
- "Dependency language flags appeared in 7 of 12 sessions"
|
||||
- "Edit ratio averaged 8%, below the 10% threshold in 5 sessions"
|
||||
- If all metrics are within normal ranges, say so plainly:
|
||||
"All metrics within normal ranges for the reporting period."
|
||||
- **Investigator, not validator.** The report exists to prompt examination,
|
||||
not to hand the user a verdict to approve or reject. Pair each number with
|
||||
the context that would change how to read it, and end in something to
|
||||
check — never in a conclusion about the user.
|
||||
- **Report the difference, not a label for it.** Where a metric moved between
|
||||
periods, report both values and the delta ("42 min → 56 min, +14") rather
|
||||
than naming the direction ("sessions got worse"). The same applies to a
|
||||
metric crossing a threshold: give the value and the threshold, not a
|
||||
judgment about crossing it.
|
||||
- Observations section: state facts derived from data only, each followed by
|
||||
what to check. Examples:
|
||||
- "3 of 12 sessions were between 23:00 and 05:00. Worth checking whether
|
||||
those were a deadline week or the usual pattern."
|
||||
- "Dependency language flags appeared in 7 of 12 sessions. The flags are
|
||||
regex matches on phrasing — check whether those sessions shared a topic."
|
||||
- "Edit ratio averaged 8%, below the 10% threshold in 5 sessions; reads
|
||||
were 71% of tool calls. Check which of those sessions were research and
|
||||
which were stalled."
|
||||
- Not: "Edit ratio was low, suggesting difficulty making progress."
|
||||
- If no metric crossed a threshold, say so plainly, and say what that does and
|
||||
does not mean: "No metric crossed its configured threshold this period. The
|
||||
thresholds are the plugin's, not a norm — nothing here rules anything out."
|
||||
- Omit any section that has no data (e.g., skip Trend if no previous period,
|
||||
skip Tool Usage if events.jsonl was missing).
|
||||
|
||||
|
|
|
|||
|
|
@ -120,21 +120,25 @@ if (dayCount >= THRESHOLD_HARD_SESSIONS) {
|
|||
messages.push(`${dayCount} sessions today.`);
|
||||
}
|
||||
|
||||
// Burst — a run of read-only tools is bulk reading, not rapid-fire work
|
||||
// Burst — a run of read-only tools is bulk reading, not fast editing. What
|
||||
// remains is reported as the observation plus what would tell a fast-but-
|
||||
// deliberate run apart from an unchecked one — never as a verdict label.
|
||||
const burstMessage = `Fast tool calls: ${burstCount} in a row under ${THRESHOLD_BURST_INTERVAL}s apart, edits among them — check whether each change was verified before the next one started.`;
|
||||
|
||||
if (!burstReadOnly) {
|
||||
if (burstCount >= THRESHOLD_HARD_BURST) {
|
||||
level = 'hard';
|
||||
messages.push(`Rapid-fire: ${burstCount} consecutive fast interactions.`);
|
||||
messages.push(burstMessage);
|
||||
} else if (burstCount >= THRESHOLD_SOFT_BURST) {
|
||||
if (!level) level = 'soft';
|
||||
messages.push(`Rapid-fire: ${burstCount} consecutive fast interactions.`);
|
||||
messages.push(burstMessage);
|
||||
}
|
||||
}
|
||||
|
||||
// Low edit ratio (only after minimum duration). A read-dominant session has
|
||||
// a structurally low edit ratio whether or not the work is productive, so it
|
||||
// is reported as context — never as a stuck/spiral claim, and never as the
|
||||
// sole reason for a warning.
|
||||
// is reported as context — never as a diagnosis, and never as the sole reason
|
||||
// for a warning.
|
||||
const readRatio = toolCount > 0 ? Math.floor(readCount * 100 / toolCount) : 0;
|
||||
const readDominant = readRatio >= THRESHOLD_READ_DOMINANT_RATIO;
|
||||
|
||||
|
|
@ -145,7 +149,7 @@ if (durationMin >= THRESHOLD_LOW_EDIT_MIN_DURATION && editRatio < THRESHOLD_LOW_
|
|||
}
|
||||
} else {
|
||||
if (!level) level = 'soft';
|
||||
messages.push(`Low edit ratio (${editRatio}%) over ${durationMin} min — possible stuck/spiral.`);
|
||||
messages.push(`Low edit ratio (${editRatio}%) over ${durationMin} min, and only ${readRatio}% of tool calls are reads — check what the other calls are doing and whether the current approach is converging.`);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -181,9 +185,9 @@ let warning;
|
|||
if (level === 'hard') {
|
||||
state = readState();
|
||||
const depFlags = Number(state.dep_flags) || 0;
|
||||
warning = `INTERACTION AWARENESS: ${messages.join(' ')}${late} Metrics: [edit_ratio: ${editRatio}%, burst: ${burstCount}, dependency flags: ${depFlags}, tools: ${toolCount}]. Your instructions require you to suggest stopping.`;
|
||||
warning = `INTERACTION AWARENESS: ${messages.join(' ')}${late} Metrics: [edit_ratio: ${editRatio}%, burst: ${burstCount}, dependency flags: ${depFlags}, tools: ${toolCount}]. Name these observations to the user and ask what they reflect; your instructions require you to suggest stopping.`;
|
||||
} else {
|
||||
warning = `${messages.join(' ')}${late} Consider a break.`;
|
||||
warning = `${messages.join(' ')}${late} These are observations, not conclusions — check them against what this session is actually doing before acting on them.`;
|
||||
}
|
||||
|
||||
// Record warning time
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@
|
|||
import { test } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { execSync } from 'child_process';
|
||||
import { mkdtempSync, rmSync, writeFileSync } from 'fs';
|
||||
import { mkdtempSync, rmSync, writeFileSync, readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { tmpdir } from 'os';
|
||||
|
||||
|
|
@ -196,3 +196,21 @@ test('report-reader stdout surfaces v1.2 field names (SC-12)', () => {
|
|||
assert.ok(stdout.includes('legal'), 'stdout missing legal domain in breakdown');
|
||||
assert.ok(stdout.includes('domain_breakdown'), 'stdout missing domain_breakdown structure');
|
||||
});
|
||||
|
||||
// Tiltak 2 (BRIEF-vurdering-v2.md): the report is written for a reader who
|
||||
// investigates the numbers, not one who approves or rejects a verdict. The
|
||||
// tone rules must say so explicitly, and the Observations template must carry
|
||||
// the context + check form rather than a bare conclusion.
|
||||
test('interaction-report.md instructs an investigative, non-verdict report', () => {
|
||||
const cmd = readFileSync(
|
||||
join(import.meta.dirname, '..', 'commands', 'interaction-report.md'), 'utf8');
|
||||
|
||||
assert.ok(cmd.includes('Investigator, not validator'),
|
||||
'tone rules missing the investigator framing');
|
||||
assert.ok(cmd.includes('never in a conclusion about the user'),
|
||||
'tone rules do not forbid concluding about the user');
|
||||
assert.ok(cmd.includes('report both values and the delta'),
|
||||
'tone rules do not require reporting the difference over a label');
|
||||
assert.ok(cmd.includes('what to check'),
|
||||
'Observations template does not ask for something to check');
|
||||
});
|
||||
|
|
|
|||
|
|
@ -1,5 +1,6 @@
|
|||
import { describe, it, afterEach } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
import { readFileSync } from 'fs';
|
||||
import { join } from 'path';
|
||||
import { runHook, setupTestDir, cleanupTestDir, createStateFile, readState, readJsonl } from './test-helper.mjs';
|
||||
|
||||
|
|
@ -109,7 +110,7 @@ describe('tool-tracker — task-type calibration', () => {
|
|||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'b1', tool_name: 'Read' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(!ctx.includes('Rapid-fire'), `expected no rapid-fire alert, got: ${ctx}`);
|
||||
assert.ok(!ctx.includes('Fast tool calls'), `expected no burst alert, got: ${ctx}`);
|
||||
assert.equal(out.continue, true);
|
||||
});
|
||||
|
||||
|
|
@ -123,7 +124,7 @@ describe('tool-tracker — task-type calibration', () => {
|
|||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'b2', tool_name: 'Edit' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(ctx.includes('Rapid-fire'), `expected rapid-fire alert, got: ${ctx}`);
|
||||
assert.ok(ctx.includes('Fast tool calls'), `expected burst alert, got: ${ctx}`);
|
||||
});
|
||||
|
||||
it('does not call a read-dominant session stuck/spiral', () => {
|
||||
|
|
@ -151,7 +152,7 @@ describe('tool-tracker — task-type calibration', () => {
|
|||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'e2', tool_name: 'Bash' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(ctx.includes('stuck/spiral'), `expected stuck/spiral claim, got: ${ctx}`);
|
||||
assert.ok(ctx.includes('Low edit ratio'), `expected low-edit-ratio observation, got: ${ctx}`);
|
||||
});
|
||||
|
||||
it('counts read tools in read_count and leaves it alone for others', () => {
|
||||
|
|
@ -165,3 +166,85 @@ describe('tool-tracker — task-type calibration', () => {
|
|||
assert.equal(readState(dir, 'r1').read_count, 2);
|
||||
});
|
||||
});
|
||||
|
||||
// Tiltak 2 (BRIEF-vurdering-v2.md): alert wording must read as an invitation to
|
||||
// investigate, not as a verdict to approve or reject. Every alert states what
|
||||
// was observed and what would distinguish a benign reading from a concerning
|
||||
// one — the form the read-dominant edit-ratio message already uses.
|
||||
describe('tool-tracker — investigative alert wording', () => {
|
||||
it('states what to check instead of labelling the burst', () => {
|
||||
dir = setupTestDir();
|
||||
createStateFile(dir, 'w1', freshState({
|
||||
last_event_epoch: Math.floor(Date.now() / 1000) - 5,
|
||||
burst_count: 9,
|
||||
tool_count: 30,
|
||||
read_count: 29,
|
||||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'w1', tool_name: 'Edit' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(!ctx.includes('Rapid-fire'), `burst alert still carries a verdict label: ${ctx}`);
|
||||
assert.ok(ctx.includes('check whether each change was verified'),
|
||||
`burst alert names no check: ${ctx}`);
|
||||
});
|
||||
|
||||
it('offers the counter-signal and a check instead of a stuck/spiral diagnosis', () => {
|
||||
dir = setupTestDir();
|
||||
createStateFile(dir, 'w2', freshState({
|
||||
start_epoch: Math.floor(Date.now() / 1000) - 40 * 60,
|
||||
tool_count: 49,
|
||||
edit_count: 2,
|
||||
read_count: 5,
|
||||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'w2', tool_name: 'Bash' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(!ctx.includes('stuck/spiral'), `edit-ratio alert still diagnoses: ${ctx}`);
|
||||
assert.ok(ctx.includes('% of tool calls are reads'),
|
||||
`edit-ratio alert omits the counter-signal: ${ctx}`);
|
||||
assert.ok(ctx.includes('check what the other calls are doing'),
|
||||
`edit-ratio alert names no check: ${ctx}`);
|
||||
});
|
||||
|
||||
it('closes a soft warning with framing, not a prescription', () => {
|
||||
dir = setupTestDir();
|
||||
// 95 min (soft duration), edit ratio 20% — isolates the closing sentence
|
||||
createStateFile(dir, 'w3', freshState({
|
||||
start_epoch: Math.floor(Date.now() / 1000) - 95 * 60,
|
||||
tool_count: 24,
|
||||
edit_count: 5,
|
||||
read_count: 10,
|
||||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'w3', tool_name: 'Bash' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(ctx.includes('Session: 95 min'), `soft warning did not fire: ${ctx}`);
|
||||
assert.ok(!ctx.includes('Consider a break'), `soft warning still prescribes: ${ctx}`);
|
||||
assert.ok(ctx.includes('observations, not conclusions'),
|
||||
`soft warning omits the framing: ${ctx}`);
|
||||
});
|
||||
|
||||
it('asks the hard warning to name its observations before suggesting stopping', () => {
|
||||
dir = setupTestDir();
|
||||
// 190 min — hard duration threshold
|
||||
createStateFile(dir, 'w4', freshState({
|
||||
start_epoch: Math.floor(Date.now() / 1000) - 190 * 60,
|
||||
tool_count: 24,
|
||||
edit_count: 5,
|
||||
read_count: 10,
|
||||
}));
|
||||
const out = runHook('tool-tracker.mjs', { session_id: 'w4', tool_name: 'Bash' }, dir);
|
||||
const ctx = out.hookSpecificOutput?.additionalContext || '';
|
||||
assert.ok(ctx.includes('INTERACTION AWARENESS'), `hard warning did not fire: ${ctx}`);
|
||||
assert.ok(ctx.includes('Name these observations to the user and ask what they reflect'),
|
||||
`hard warning omits the investigative framing: ${ctx}`);
|
||||
// The required action is unchanged — this is a wording change, not a behaviour change.
|
||||
assert.ok(ctx.includes('require you to suggest stopping'),
|
||||
`hard warning dropped the required action: ${ctx}`);
|
||||
});
|
||||
|
||||
it('leaves no verdict labels in the hook source', () => {
|
||||
const src = readFileSync(
|
||||
join(import.meta.dirname, '..', 'hooks', 'scripts', 'tool-tracker.mjs'), 'utf8');
|
||||
for (const label of ['Rapid-fire', 'stuck/spiral', 'Consider a break']) {
|
||||
assert.ok(!src.includes(label), `tool-tracker.mjs still contains verdict label: ${label}`);
|
||||
}
|
||||
});
|
||||
});
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue