test(round-builder): the determinism arm builds twice in two processes, not twice in one
Found while killing mutants: an ordering made to depend on hash() survived test_two_builds_of_the_same_outbox_are_byte_identical untouched. One interpreter has one hash seed, so the two builds agreed with each other and with nothing else -- the arm could not tell a deterministic report from a report that is merely consistent within a process. The ordering arm caught that mutant, so the seam was witnessed; this one was not witnessing it. The second build now runs as a subprocess with PYTHONHASHSEED=1. Re-measured with the same mutant: both arms red, where before only one was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
db38cfc1a4
commit
86c61a343b
1 changed files with 30 additions and 2 deletions
|
|
@ -14,6 +14,7 @@ it cannot parse.
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
||||||
import json
|
import json
|
||||||
|
import os
|
||||||
import shutil
|
import shutil
|
||||||
import subprocess
|
import subprocess
|
||||||
import sys
|
import sys
|
||||||
|
|
@ -450,10 +451,37 @@ def test_two_builds_of_the_same_outbox_are_byte_identical(tmp_path: Path) -> Non
|
||||||
|
|
||||||
Row 4 counts content lines IN ORDER, so a report that reshuffles between builds would read as
|
Row 4 counts content lines IN ORDER, so a report that reshuffles between builds would read as
|
||||||
an expert's edit. Both the whole-file bytes and the heading sequence are checked, because two
|
an expert's edit. Both the whole-file bytes and the heading sequence are checked, because two
|
||||||
files can differ in bytes for a reason that is not order."""
|
files can differ in bytes for a reason that is not order.
|
||||||
|
|
||||||
|
The second build runs in a SUBPROCESS with a different ``PYTHONHASHSEED``, and that is not
|
||||||
|
decoration: measured 19.09, an ordering made to depend on ``hash()`` survived a second
|
||||||
|
in-process build untouched — one interpreter has one hash seed, so the two builds agreed with
|
||||||
|
each other and with nothing else. Two processes is the cheapest way to be a witness rather
|
||||||
|
than a coincidence."""
|
||||||
source = _outbox(tmp_path)
|
source = _outbox(tmp_path)
|
||||||
first = _build(tmp_path, outbox=source, rounds_dir=tmp_path / "a").round_dir
|
first = _build(tmp_path, outbox=source, rounds_dir=tmp_path / "a").round_dir
|
||||||
second = _build(tmp_path, outbox=source, rounds_dir=tmp_path / "b").round_dir
|
second = tmp_path / "b" / "0"
|
||||||
|
env = {**os.environ, "PYTHONHASHSEED": "1"}
|
||||||
|
other = subprocess.run(
|
||||||
|
[
|
||||||
|
sys.executable,
|
||||||
|
"-m",
|
||||||
|
"portfolio_optimiser.evals.round_builder",
|
||||||
|
"--outbox",
|
||||||
|
str(source),
|
||||||
|
"--round",
|
||||||
|
"0",
|
||||||
|
"--rounds-dir",
|
||||||
|
str(tmp_path / "b"),
|
||||||
|
"--ran-at",
|
||||||
|
_RAN_AT,
|
||||||
|
],
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
cwd=Path(rb.__file__).resolve().parents[3],
|
||||||
|
env=env,
|
||||||
|
)
|
||||||
|
assert other.returncode == 0, other.stderr
|
||||||
for name in ("report.md", "outcome.json"):
|
for name in ("report.md", "outcome.json"):
|
||||||
assert (first / name).read_bytes() == (second / name).read_bytes(), name
|
assert (first / name).read_bytes() == (second / name).read_bytes(), name
|
||||||
headings = [
|
headings = [
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue