test(round-builder): the determinism arm builds twice in two processes, not twice in one
Found while killing mutants: an ordering made to depend on hash() survived test_two_builds_of_the_same_outbox_are_byte_identical untouched. One interpreter has one hash seed, so the two builds agreed with each other and with nothing else -- the arm could not tell a deterministic report from a report that is merely consistent within a process. The ordering arm caught that mutant, so the seam was witnessed; this one was not witnessing it. The second build now runs as a subprocess with PYTHONHASHSEED=1. Re-measured with the same mutant: both arms red, where before only one was. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
db38cfc1a4
commit
86c61a343b
1 changed files with 30 additions and 2 deletions
|
|
@ -14,6 +14,7 @@ it cannot parse.
|
|||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
|
|
@ -450,10 +451,37 @@ def test_two_builds_of_the_same_outbox_are_byte_identical(tmp_path: Path) -> Non
|
|||
|
||||
Row 4 counts content lines IN ORDER, so a report that reshuffles between builds would read as
|
||||
an expert's edit. Both the whole-file bytes and the heading sequence are checked, because two
|
||||
files can differ in bytes for a reason that is not order."""
|
||||
files can differ in bytes for a reason that is not order.
|
||||
|
||||
The second build runs in a SUBPROCESS with a different ``PYTHONHASHSEED``, and that is not
|
||||
decoration: measured 19.09, an ordering made to depend on ``hash()`` survived a second
|
||||
in-process build untouched — one interpreter has one hash seed, so the two builds agreed with
|
||||
each other and with nothing else. Two processes is the cheapest way to be a witness rather
|
||||
than a coincidence."""
|
||||
source = _outbox(tmp_path)
|
||||
first = _build(tmp_path, outbox=source, rounds_dir=tmp_path / "a").round_dir
|
||||
second = _build(tmp_path, outbox=source, rounds_dir=tmp_path / "b").round_dir
|
||||
second = tmp_path / "b" / "0"
|
||||
env = {**os.environ, "PYTHONHASHSEED": "1"}
|
||||
other = subprocess.run(
|
||||
[
|
||||
sys.executable,
|
||||
"-m",
|
||||
"portfolio_optimiser.evals.round_builder",
|
||||
"--outbox",
|
||||
str(source),
|
||||
"--round",
|
||||
"0",
|
||||
"--rounds-dir",
|
||||
str(tmp_path / "b"),
|
||||
"--ran-at",
|
||||
_RAN_AT,
|
||||
],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
cwd=Path(rb.__file__).resolve().parents[3],
|
||||
env=env,
|
||||
)
|
||||
assert other.returncode == 0, other.stderr
|
||||
for name in ("report.md", "outcome.json"):
|
||||
assert (first / name).read_bytes() == (second / name).read_bytes(), name
|
||||
headings = [
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue