Kjell Tore Guttormsen
33b58f460a
1. A step of only tests can never be proven red. One reading, stepTestsOnly (lib/parsers/plan-schema.mjs, test paths from the new lib/util/test-path.mjs): the plan validator warns PLAN_STEP_TESTS_ONLY, red-proof --plan/--step adds tests_only to a not-red-at-base, and /trekexecute and the step helper send such a step straight up to the plan instead of into fix rounds that cannot succeed. /trekplan, the plan template and plan-critic (dimension 8, still 12) say the test belongs in the step that builds the behaviour. 2. Timing tests under load: the hook helper's kill limit is 30 s and per call overridable; SC #14 passes on the fastest of up to three runs against the same 800 ms limit (tests/helpers/timing.mjs). 3. The example plan's On failure lines open with a K11 word. 4. /trekreview saves the reviewer's whole answer for the findings check and names FINDING_PROSE_LINE_OUT_OF_RANGE (error: re-ask; warning: report). 5. The criteria runner's commands run without NODE_TEST_CONTEXT (commandEnv, moved to lib/util/command-env.mjs). 6. The resolver returns effort only when the brief's phase signal sets one; the four commands said it always did. Also: the K2 hook tests used fixed session ids in a per-user temp dir, so two suites running at once took each other's approval records; the ids are now unique per run. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
63 lines
3 KiB
JavaScript
63 lines
3 KiB
JavaScript
// tests/lib/timing-under-load.test.mjs
|
|
// The two timing tests that went red under load in the wave sessions (order 10,
|
|
// point 2; operator 2026-10-09): P7 (post-edit-check) was killed by the hook
|
|
// helper's 5 s limit at load 95-177 and was 8/8 green alone; SC #14
|
|
// (otel-export) demands < 800 ms for ONE cold run.
|
|
// - The helper's kill limit is longer (HOOK_TIMEOUT_MS) and overridable per
|
|
// call. P7 checks the hook's answer, not its speed, so that weakens nothing.
|
|
// - SC #14 keeps its 800 ms limit and measures up to three runs, passing on
|
|
// the fastest. Load only ever ADDS time, so the fastest run is the best
|
|
// reading of the real cost: a real slowness is slow every time and still
|
|
// fails; one run slowed by a busy machine does not.
|
|
|
|
import { test } from 'node:test';
|
|
import { strict as assert } from 'node:assert';
|
|
import { mkdtempSync, rmSync, writeFileSync } from 'node:fs';
|
|
import { tmpdir } from 'node:os';
|
|
import { join } from 'node:path';
|
|
import { fastestWithin } from '../helpers/timing.mjs';
|
|
import { HOOK_TIMEOUT_MS, runHookWithEnv } from '../helpers/hook-helper.mjs';
|
|
|
|
/** A measure that returns the given durations in turn (no real waiting). */
|
|
const stub = (durations) => {
|
|
let i = 0;
|
|
return async () => durations[i++];
|
|
};
|
|
|
|
test('timing: a stub that is slow every time fails the limit (a real slowness is felled)', async () => {
|
|
const r = await fastestWithin(800, stub([900, 950, 1200]));
|
|
assert.equal(r.ok, false);
|
|
assert.deepEqual(r.runs, [900, 950, 1200]);
|
|
assert.equal(r.fastest, 900);
|
|
});
|
|
|
|
test('timing: one slow run of three is tolerated - the next run under the limit passes', async () => {
|
|
const r = await fastestWithin(800, stub([1500, 300, 2000]));
|
|
assert.equal(r.ok, true);
|
|
assert.deepEqual(r.runs, [1500, 300], 'stops at the first run under the limit');
|
|
});
|
|
|
|
test('timing: the limit itself is not loosened - a run AT the limit is not under it', async () => {
|
|
const r = await fastestWithin(800, stub([800, 800, 800]));
|
|
assert.equal(r.ok, false);
|
|
});
|
|
|
|
test('timing: a first run under the limit passes at once (one run on a quiet machine)', async () => {
|
|
const r = await fastestWithin(800, stub([120]));
|
|
assert.equal(r.ok, true);
|
|
assert.deepEqual(r.runs, [120]);
|
|
});
|
|
|
|
test('hook helper: the kill limit is far above 5 s by default, and a call can set its own', async () => {
|
|
assert.ok(HOOK_TIMEOUT_MS >= 30_000, `HOOK_TIMEOUT_MS = ${HOOK_TIMEOUT_MS}`);
|
|
const dir = mkdtempSync(join(tmpdir(), 'hook-helper-'));
|
|
try {
|
|
const slow = join(dir, 'slow-hook.mjs');
|
|
writeFileSync(slow, "setTimeout(() => { process.stdout.write('done'); process.exit(0); }, 400);\n");
|
|
const killed = await runHookWithEnv(slow, '{}', {}, { timeoutMs: 100 });
|
|
assert.notEqual(killed.code, 0, 'a per-call limit below the hook\'s run time kills it');
|
|
const done = await runHookWithEnv(slow, '{}', {});
|
|
assert.equal(done.code, 0);
|
|
assert.equal(done.stdout, 'done');
|
|
} finally { rmSync(dir, { recursive: true, force: true }); }
|
|
});
|