fix(migration): operator-window SC2 must be regression-relative, not strict

The operator window's step [a] called 40-validate-standalone.sh directly (strict
exit-code), so it STOPped on the first target carrying pre-existing in-repo test
red — voyage (2 doc-consistency drifts re phase_models/phase_signals, content moved
to docs/operations.md) and ai-psychosis (1). But the migration's ratified contract,
the one the Step-11 dry-run validated as PASS 11/11, is 'introduce no regression':
pre-existing in-repo red is the plugin's own concern, not a migration regression.
The window enforced a STRICTER gate than the contract the dry-run signed off.

Fix: new 41-validate-or-regression.sh — the single per-target gate the window calls
in [a]. It runs 40 strict, then on failure passes iff the standalone failing-test
NAME set is a SUBSET of the live in-repo set (the exact decision 99-dryrun.sh makes),
reusing sc2-regression.sh. A genuine extraction-introduced regression still STOPs the
window; a structure-validator fail and the config-audit gate stay strict.

Single-source the failing-name capture: extract capture_fails into capture-fails.sh
(mirrors the sc2-regression.sh extraction) so the live gate and the dry-run agree on
what 'failing' means; 99-dryrun.sh now delegates to it (behaviour identical).

Verified end-to-end on the real extracts: 40 strict fails voyage+ai-psychosis while
41 passes them 'N pre-existing, regression-relative'; clean targets (llm-security,
graceful-handoff) still pass via the strict path. New hermetic tests: capture-fails
3/3, 41 6/6 (strict-pass, regression-relative-pass, genuine-regression-fail,
structure-not-eligible, gate pass/fail). RUNBOOK per-repo step updated to 41.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-06-18 06:25:55 +02:00
commit 3403648c6c
7 changed files with 336 additions and 12 deletions

View file

@ -0,0 +1,84 @@
#!/usr/bin/env bash
# Step 6b — per-target SC2/SC7 gate with the migration's REGRESSION-RELATIVE contract baked in.
#
# WHY THIS EXISTS: 40-validate-standalone.sh is STRICT — any standalone test failure exits non-zero. The
# migration's RATIFIED contract — the one the Step-11 dry-run (99-dryrun.sh) validated and signed off as
# "PASS 11/11" — is weaker and correct: a target passes iff the extraction introduces NO NEW failure.
# Pre-existing in-repo red (voyage's 2 doc-consistency drifts re phase_models/phase_signals; ai-psychosis's
# 1) is the PLUGIN's own concern, not a migration regression. The operator window (run-operator-window.sh
# step [a]) must enforce THAT contract, not a stricter one — otherwise it STOPs on the first target carrying
# pre-existing red even though the dry-run blessed it (the bug this script fixes). This is the single
# per-target gate the window calls; it mirrors 99-dryrun.sh's SC2 decision exactly, reusing capture-fails.sh
# (the failing-name capture) + sc2-regression.sh (the subset decision). NULL push (D8): read-only validation.
#
# CONTRACT:
# - sc2_gate target (config-audit): the dedicated gate is deterministic — run it STRICT (PASS/FAIL).
# - else: run 40-validate-standalone.sh STRICT. PASS => exit 0.
# On FAIL, only a `node --test` suite is regression-eligible: its standalone failing-NAME set must be
# a SUBSET of the live in-repo failing-NAME set (regression-relative PASS, "PASS (N pre-existing)").
# A structure-validator (validate-plugin*.sh / bash -c) FAIL is a genuine structural defect — NOT
# softened. A genuine regression (standalone-only failure) or a missing capture => exit non-zero.
# Escalate, never mask: any real regression or structural failure exits non-zero so the window STOPs.
#
# Usage: 41-validate-or-regression.sh <target-key>
set -uo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
MAP="$SCRIPT_DIR/plugin-map.json"
REPO_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
WORK="${WORK:-/tmp/polyrepo-migration}"
[ -f "$MAP" ] || { echo "plugin-map.json missing at $MAP" >&2; exit 2; }
key="${1:?usage: 41-validate-or-regression.sh <target-key>}"
mapget() { python3 -c "import json;print(json.load(open('$MAP'))['targets']['$key'].get('$1',''))"; }
sc2_gate="$(mapget sc2_gate)"
test_cmd="$(mapget test_cmd)"
# --- gate target (config-audit): deterministic, strict by design ---
if [ -n "$sc2_gate" ]; then
if WORK="$WORK" bash "$SCRIPT_DIR/$sc2_gate" >/dev/null 2>&1; then
echo "$key: PASS ($sc2_gate)"; exit 0
fi
echo "$key: FAIL ($sc2_gate)" >&2; exit 1
fi
# --- strict standalone validation first (SC2 + SC7 + per-target structural checks) ---
if WORK="$WORK" bash "$SCRIPT_DIR/40-validate-standalone.sh" "$key" >/dev/null 2>&1; then
echo "$key: PASS (standalone strict)"; exit 0
fi
# Strict failed. Only a node:test suite can fail regression-relative-acceptably; a structure validator
# failing is a genuine structural defect — do not soften it.
case "$test_cmd" in
*node\ --test*) : ;;
*) echo "$key: FAIL (standalone strict; structure validator — not regression-eligible)" >&2; exit 1 ;;
esac
dest="$WORK/$key"
[ -d "$dest/.git" ] || { echo "$key: FAIL (no prepped extract at $dest — run 40-validate-standalone.sh first)" >&2; exit 1; }
# Regression-relative: the standalone failing-NAME set must be a SUBSET of the live in-repo failing-NAME set.
# Capture the standalone set from the PREPPED EXTRACT ($WORK/$key), NOT 40's /tmp/claude-$key side-effect
# clean room (4e494c8: that coupling masked a real regression when the dir was absent). The subset decision
# is delegated to sc2-regression.sh; a missing capture FILE there is a hard error, never a false PASS.
sf="$(mktemp)" && bf="$(mktemp)" || { echo "$key: FAIL (mktemp)" >&2; exit 1; }
trap 'rm -f "$sf" "$bf"' EXIT
bash "$SCRIPT_DIR/capture-fails.sh" "$dest" "$test_cmd" > "$sf" 2>/dev/null
bash "$SCRIPT_DIR/capture-fails.sh" "$REPO_ROOT/plugins/$key" "$test_cmd" > "$bf" 2>/dev/null
pre="$(wc -l < "$bf" | tr -d '[:space:]')"
if regr="$(bash "$SCRIPT_DIR/sc2-regression.sh" "$sf" "$bf")"; then
echo "$key: PASS (${pre} pre-existing, regression-relative)"
exit 0
else
rc=$?
if [ "$rc" -ge 2 ]; then
echo "$key: FAIL (sc2-regression error — missing capture file)" >&2
else
echo "$key: FAIL (regression: $(printf '%s' "$regr" | grep -c .) new failure(s) absent from in-repo baseline):" >&2
printf '%s\n' "$regr" >&2
fi
exit 1
fi

View file

@ -0,0 +1,141 @@
// Coverage for 41-validate-or-regression.sh — the per-target gate the operator window calls in step [a].
// It enforces the migration's RATIFIED contract ("introduce no regression"), NOT the stricter zero-failure
// gate that made run-operator-window.sh STOP on voyage's / ai-psychosis's pre-existing in-repo red. 41 is
// thin glue over already-tested detectors (sc2-regression.sh — sc-checks.test.mjs; capture-fails.sh —
// capture-fails.test.mjs) and over 40-validate-standalone.sh (strict). This suite drives the REAL 41 inside
// a self-contained fake migration tree: the heavy extract pipeline (40) is STUBBED so the regression-branch
// glue is exercised hermetically, with no /tmp/polyrepo-migration state and no real plugin extracts.
//
// Branches covered: strict-PASS passthrough; regression-relative PASS (standalone ⊆ in-repo); genuine
// regression FAIL (standalone-only failure); structure-validator FAIL is NOT regression-eligible; the
// sc2_gate target stays strict (PASS + FAIL).
import test from 'node:test';
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync, copyFileSync, chmodSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const here = path.dirname(fileURLToPath(import.meta.url));
// One node:test file = imports ONCE + a failing test() per name. (Concatenating per-name single-file
// strings would duplicate `import test` → SyntaxError → a file-level not-ok instead of per-test names.)
const failSuite = (names) =>
"import test from 'node:test';\nimport assert from 'node:assert/strict';\n" +
names.map((n) => `test(${JSON.stringify(n)}, () => assert.equal(1, 2));`).join('\n') +
'\n';
function write(p, content, mode) {
mkdirSync(path.dirname(p), { recursive: true });
writeFileSync(p, content);
if (mode) chmodSync(p, mode);
}
// Build a fake migration tree: SCRIPT_DIR nested 3 deep so 41's REPO_ROOT ($SCRIPT_DIR/../../..) is the root.
function buildTree() {
const root = mkdtempSync(path.join(os.tmpdir(), 'mig41-'));
const mig = path.join(root, 'a', 'b', 'c'); // SCRIPT_DIR; ../../.. === root
const work = path.join(root, '_work'); // $WORK
mkdirSync(mig, { recursive: true });
// real scripts under test + reused detectors
for (const f of ['41-validate-or-regression.sh', 'capture-fails.sh', 'sc2-regression.sh']) {
copyFileSync(path.join(here, f), path.join(mig, f));
}
// stub strict 40-validate-standalone.sh: passes ONLY for the designated strict-pass target.
write(path.join(mig, '40-validate-standalone.sh'),
'#!/usr/bin/env bash\n[ "${1:-}" = "strictpass" ] && exit 0\nexit 1\n', 0o755);
// sc2_gate stubs
write(path.join(mig, 'gate-pass.sh'), '#!/usr/bin/env bash\nexit 0\n', 0o755);
write(path.join(mig, 'gate-fail.sh'), '#!/usr/bin/env bash\nexit 1\n', 0o755);
const NT = "node --test 'tests/**/*.test.mjs'";
const map = {
targets: {
strictpass: { test_cmd: NT },
subset: { test_cmd: NT },
regress: { test_cmd: NT },
structure: { test_cmd: 'bash validate-plugin.sh' },
gatepass: { test_cmd: NT, sc2_gate: 'gate-pass.sh' },
gatefail: { test_cmd: NT, sc2_gate: 'gate-fail.sh' },
},
};
write(path.join(mig, 'plugin-map.json'), JSON.stringify(map, null, 2));
// node:test suites for the regression-eligible targets. dest = $WORK/<key> (needs .git); baseline = root/plugins/<key>.
const suite = (key, names) => {
mkdirSync(path.join(work, key, '.git'), { recursive: true });
write(path.join(work, key, 'tests', 'x.test.mjs'), failSuite(names.standalone));
write(path.join(root, 'plugins', key, 'tests', 'x.test.mjs'), failSuite(names.baseline));
};
suite('subset', { standalone: ['shared red'], baseline: ['shared red', 'other in-repo red'] });
suite('regress', { standalone: ['shared red', 'extract regressed'], baseline: ['shared red'] });
return { root, mig, work };
}
function run41(tree, key) {
// Strip NODE_TEST_CONTEXT: 41 → capture-fails → `node --test`, which emits no TAP if it inherits the
// harness's nested-test context (Step-6 env-clean gotcha). The real window runs as plain bash, unnested.
const env = { ...process.env, WORK: tree.work };
delete env.NODE_TEST_CONTEXT;
return spawnSync('bash', [path.join(tree.mig, '41-validate-or-regression.sh'), key],
{ encoding: 'utf8', env });
}
test('strict-PASS passthrough: 40 passes → 41 PASS (standalone strict), exit 0, no capture', () => {
const tree = buildTree();
try {
const r = run41(tree, 'strictpass');
assert.equal(r.status, 0, `strict pass must exit 0: ${r.stderr}`);
assert.match(r.stdout, /PASS \(standalone strict\)/, r.stdout);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});
test('regression-relative PASS: standalone failing-set ⊆ in-repo → exit 0, reports N pre-existing', () => {
const tree = buildTree();
try {
const r = run41(tree, 'subset');
assert.equal(r.status, 0, `a subset of in-repo red is not a migration regression: ${r.stdout}${r.stderr}`);
assert.match(r.stdout, /pre-existing, regression-relative/, r.stdout);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});
test('genuine regression FAIL: a standalone-only failure (absent in-repo) → non-zero, names the regression', () => {
const tree = buildTree();
try {
const r = run41(tree, 'regress');
assert.notEqual(r.status, 0, 'a new failure introduced by extraction must STOP the window');
assert.match(r.stderr, /regression/, r.stderr);
assert.match(r.stderr, /extract regressed/, `the regressing test name must be surfaced: ${r.stderr}`);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});
test('structure-validator FAIL is NOT regression-eligible: bash validator fail → non-zero, never softened', () => {
const tree = buildTree();
try {
const r = run41(tree, 'structure');
assert.notEqual(r.status, 0, 'a deterministic structure-validator failure is a genuine defect');
assert.match(r.stderr, /not regression-eligible/, r.stderr);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});
test('sc2_gate target stays strict: gate PASS → exit 0', () => {
const tree = buildTree();
try {
const r = run41(tree, 'gatepass');
assert.equal(r.status, 0, r.stderr);
assert.match(r.stdout, /PASS \(gate-pass\.sh\)/, r.stdout);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});
test('sc2_gate target stays strict: gate FAIL → non-zero', () => {
const tree = buildTree();
try {
const r = run41(tree, 'gatefail');
assert.notEqual(r.status, 0, 'a failing dedicated gate must STOP the window');
assert.match(r.stderr, /FAIL \(gate-fail\.sh\)/, r.stderr);
} finally { rmSync(tree.root, { recursive: true, force: true }); }
});

View file

@ -59,14 +59,9 @@ live_files() {
echo "$total"
}
# Failing-test NAME set for a node:test suite, via the stable TAP reporter (locale-independent).
# $1 = dir to run in, $2 = the plugin's node --test command.
capture_fails() {
local dir="$1" cmd="$2" tapcmd
tapcmd="${cmd/node --test/node --test --test-reporter=tap}"
( cd "$dir" && eval "$tapcmd" ) 2>&1 \
| grep -E '^not ok ' | sed -E 's/^not ok [0-9]+ - //; s/ #.*$//' | sort -u
}
# Failing-test NAME set for a node:test suite — delegated to capture-fails.sh so the SAME capture feeds both
# this rehearsal and the live operator-window gate (41-validate-or-regression.sh). $1 = dir, $2 = node --test cmd.
capture_fails() { bash "$SCRIPT_DIR/capture-fails.sh" "$1" "$2"; }
[ -f "$MAP" ] || fail "plugin-map.json missing at $MAP"
mkdir -p "$WORK"

View file

@ -99,11 +99,15 @@ Order (lower risk last so problems surface while the blast radius is small):
```bash
KEY=<target> # e.g. voyage
# a. extract + validate standalone (config-audit uses its dedicated SC2 gate)
# a. extract + validate standalone (config-audit uses its dedicated SC2 gate).
# SC2 is REGRESSION-RELATIVE (the contract the Step-11 dry-run validated): 41 runs 40 strict, then on
# failure passes iff the standalone failing-test set is a SUBSET of the live in-repo set — so pre-existing
# in-repo red (e.g. voyage's 2 doc-consistency drifts, ai-psychosis's 1) does NOT STOP the rollout, while
# a genuine extraction-introduced regression still does. (Strict 40 alone would STOP on that blessed red.)
if [ "$KEY" = "config-audit" ]; then
WORK=$WORK bash MIG/50-config-audit-sc2.sh # config-audit: SC2 PASS (...)
else
WORK=$WORK bash MIG/40-validate-standalone.sh "$KEY" # <KEY>: PASS (...)
WORK=$WORK bash MIG/41-validate-or-regression.sh "$KEY" # <KEY>: PASS (standalone strict | N pre-existing)
fi
# b. create the Forgejo repo (auto_init:false, public)

View file

@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Failing-test NAME set for a node:test suite, via the stable TAP reporter (locale-independent).
# Extracted from 99-dryrun.sh's inline capture_fails (mirrors the sc2-regression.sh extraction, 5d112cb) so
# the SAME capture feeds BOTH the dry-run rehearsal (99-dryrun.sh) and the live operator-window gate
# (41-validate-or-regression.sh) — a single source of truth prevents the gate and the rehearsal from
# disagreeing on what "failing" means (the exact class of drift that let the strict window STOP on red the
# dry-run had blessed).
#
# Prints sorted-unique failing test names, one per line. Empty output = the suite reported no `not ok` lines.
# The caller distinguishes "ran clean" from "did not run": sc2-regression.sh treats a missing capture FILE
# as a hard error, but an empty-yet-present file is a legitimate zero-failure set. No pipefail: a no-match
# grep (zero failures) must exit 0, not leak a spurious non-zero up to the caller.
#
# Usage: capture-fails.sh <dir> <node --test command>
set -u
dir="${1:?dir}"
cmd="${2:?node --test command}"
tapcmd="${cmd/node --test/node --test --test-reporter=tap}"
( cd "$dir" && eval "$tapcmd" ) 2>&1 \
| grep -E '^not ok ' | sed -E 's/^not ok [0-9]+ - //; s/ #.*$//' | sort -u

View file

@ -0,0 +1,73 @@
// Coverage for capture-fails.sh — the failing-test-NAME capture extracted from 99-dryrun.sh so a SINGLE
// source feeds both the dry-run rehearsal and the live operator-window gate (41-validate-or-regression.sh).
// Proves it reports exactly the failing names (sorted, deduped across files) and stays exit-0 on an all-pass
// suite — a no-match grep must NOT leak a non-zero status up to the caller (which would masquerade as a
// failed capture). Drives the real script against tiny node:test fixtures — hermetic, no migration state.
import test from 'node:test';
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import { mkdtempSync, rmSync, writeFileSync, mkdirSync } from 'node:fs';
import os from 'node:os';
import path from 'node:path';
import { fileURLToPath } from 'node:url';
const here = path.dirname(fileURLToPath(import.meta.url));
const CAP = path.join(here, 'capture-fails.sh');
const CMD = "node --test 'tests/**/*.test.mjs'";
// One node:test file = imports ONCE + a test() per entry. entry: { name, pass }. (Concatenating
// per-name single-file strings would duplicate `import test` → SyntaxError → a file-level not-ok, masking
// the per-test capture under test.)
const suiteFile = (entries) =>
"import test from 'node:test';\nimport assert from 'node:assert/strict';\n" +
entries.map((e) => `test(${JSON.stringify(e.name)}, () => assert.equal(1, ${e.pass ? 1 : 2}));`).join('\n') +
'\n';
function sandbox(files) {
const dir = mkdtempSync(path.join(os.tmpdir(), 'capfail-'));
for (const [rel, content] of Object.entries(files)) {
const p = path.join(dir, rel);
mkdirSync(path.dirname(p), { recursive: true });
writeFileSync(p, content);
}
return dir;
}
function cap(dir) {
// Strip NODE_TEST_CONTEXT: capture-fails spawns `node --test`, which misbehaves (emits no TAP) when it
// inherits the harness's nested-test context (Step-6 env-clean gotcha). The real operator window runs as
// plain bash, never nested under node:test, so this only matters inside this harness.
const env = { ...process.env };
delete env.NODE_TEST_CONTEXT;
return spawnSync('bash', [CAP, dir, CMD], { encoding: 'utf8', env });
}
test('capture-fails reports the failing test name (and only it), exit 0', () => {
const dir = sandbox({ 'tests/a.test.mjs': suiteFile([{ name: 'good', pass: true }, { name: 'bad apple', pass: false }]) });
try {
const r = cap(dir);
assert.equal(r.status, 0, `capture must exit 0 even with failures: ${r.stderr}`);
assert.equal(r.stdout.trim(), 'bad apple', `only the failing name expected, got: ${JSON.stringify(r.stdout)}`);
} finally { rmSync(dir, { recursive: true, force: true }); }
});
test('capture-fails on an all-pass suite → empty output, exit 0 (no-match grep must not leak non-zero)', () => {
const dir = sandbox({ 'tests/a.test.mjs': suiteFile([{ name: 'good', pass: true }, { name: 'also good', pass: true }]) });
try {
const r = cap(dir);
assert.equal(r.status, 0, `all-pass must exit 0: ${r.stderr}`);
assert.equal(r.stdout.trim(), '', `no failing names expected, got: ${JSON.stringify(r.stdout)}`);
} finally { rmSync(dir, { recursive: true, force: true }); }
});
test('capture-fails sorts + dedups failing names across files', () => {
const dir = sandbox({
'tests/a.test.mjs': suiteFile([{ name: 'zeta', pass: false }, { name: 'alpha', pass: false }]),
'tests/b.test.mjs': suiteFile([{ name: 'alpha', pass: false }]), // duplicate name across files collapses to one
});
try {
const r = cap(dir);
assert.equal(r.status, 0, r.stderr);
assert.equal(r.stdout.trim(), 'alpha\nzeta', `sorted+deduped expected, got: ${JSON.stringify(r.stdout)}`);
} finally { rmSync(dir, { recursive: true, force: true }); }
});

View file

@ -81,12 +81,19 @@ rollout_one() {
say ""
say "==== $key (tag $tag) ===="
# (a) validate the extract — self-extracts if $WORK/$key is absent
# (a) validate the extract — self-extracts if $WORK/$key is absent.
# SC2 is REGRESSION-RELATIVE (the contract the Step-11 dry-run validated): a target passes iff the
# extraction introduces NO NEW failure. Pre-existing in-repo red (voyage's 2 doc-consistency drifts,
# ai-psychosis's 1) is the plugin's own concern — 41-validate-or-regression.sh enforces that exact
# contract (strict 40 first, then standalone-failing ⊆ in-repo-failing) so the window does NOT STOP on
# red the rehearsal blessed. config-audit keeps its dedicated deterministic gate.
say " [a] validate extract…"
if [ "$key" = "config-audit" ]; then
WORK="$WORK" bash "$MIG/50-config-audit-sc2.sh" >/dev/null 2>&1 || die "$key SC2 gate FAILED — STOP"
else
WORK="$WORK" bash "$MIG/40-validate-standalone.sh" "$key" >/dev/null 2>&1 || die "$key standalone validation FAILED — STOP"
sc2out="$(WORK="$WORK" bash "$MIG/41-validate-or-regression.sh" "$key" 2>&1)" \
|| die "$key standalone validation FAILED (incl. regression-relative SC2) — STOP${sc2out:+ :: $sc2out}"
case "$sc2out" in *pre-existing*) say " ${sc2out#*: }";; esac
fi
# (b) create the Forgejo repo (201 created | 409 already exists)