test(loadbearing): measure the last 8 all-negative tests instead of assuming them
Point 2 of the vacuity sweep is now MEASURED, not paired-by-reading. Every one of the 8 got a mutation that detaches the seam it claims to guard, run through a harness that asserts the anchor is unique before mutating, restores in `finally`, and sha256-verifies the restore. Six were value-proven — the negative itself went RED under its detach: hitl :260 load_routing invents a default -> RED hitl :298 route_pending hardcodes a fallback -> RED step7 :145 the is_dir() guard deleted -> RED step7 :171 the §4.2 vocabulary filter deleted -> RED step7 :253 the id grammar off the model -> RED prov :111 sdk_version becomes required -> RED Two did not, and both are fixed here. hitl :197 — the guard it appeared to prove is DEAD. Deleting the `is_dir()` early-out from load_outbox_proposals leaves all 711 tests green: the tolerance comes from `Path.glob`, which yields nothing on a missing directory and never raises. The contrast is the finding: load_inbox carries an identically-shaped guard that IS load-bearing, because it walks with `Path.iterdir`, which DOES raise (measured both ways). Same guard, opposite verdict, and the difference is the stdlib call behind it — the point-3 lesson one level out, where the default being pinned belongs to the standard library rather than the SDK. The stdlib baseline is now anchored explicitly, so a Python that makes glob raise turns this red instead of quietly promoting a dead line to a seam. What the test always did prove is kept and stated: replacing the early-out with a raise turns it red, so it does hold tolerance. portfolio :175 — the negative asserted over an unheld population. Measured, it is real today (6 prompts), so the test is not vacuous now; nothing in it says so, and a run_portfolio that stopped prompting would leave it green while proving nothing. A positive control now runs first. Value-proven: green before, red after the same mutation (iterate no projects), and it is that assertion which fails, not an import. The sibling repo sent the same rule from the other stack this week, arrived at independently via its B4 empty-negative: on a negative assert, prove FIRST that the event happened. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qr6TwWrHDHeukHy3bL4hgb
This commit is contained in:
parent
e6bc1832a2
commit
82d46148e2
2 changed files with 26 additions and 0 deletions
|
|
@ -195,6 +195,26 @@ class TestPendingRegistry:
|
|||
assert [p.run_id for p in pending] == ["r-001", "r-002"] # sorted by run_id
|
||||
|
||||
def test_missing_outbox_dir_is_empty(self, tmp_path: Path) -> None:
|
||||
# HONEST LIMIT (measured this session): this does NOT prove the
|
||||
# ``is_dir()`` early-out in load_outbox_proposals is load-bearing.
|
||||
# Deleting that guard leaves all 711 tests green — the tolerance comes
|
||||
# from ``Path.glob``, not from us: glob over a missing directory yields
|
||||
# nothing and never raises.
|
||||
#
|
||||
# The contrast is what makes the shape untrustworthy on sight:
|
||||
# load_inbox carries an identically-shaped guard that IS load-bearing,
|
||||
# because it walks the folder with ``Path.iterdir``, which DOES raise.
|
||||
# Same guard, opposite verdict, and the only difference is the stdlib
|
||||
# call standing behind it.
|
||||
#
|
||||
# So pin the stdlib baseline the tolerance actually rests on. Should a
|
||||
# future Python make glob raise on a missing directory, the guard
|
||||
# becomes the seam — and this says so instead of going quiet.
|
||||
assert list((tmp_path / "nope").glob("*.json")) == [] # the stdlib baseline
|
||||
|
||||
# What the assertion below DOES prove: the read boundary TOLERATES a
|
||||
# missing outbox rather than raising. An intolerant implementation
|
||||
# (measured: replace the early-out with a raise) turns it red.
|
||||
assert pending_proposals(tmp_path / "nope", tmp_path / "inbox") == []
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -179,6 +179,12 @@ class TestPortfolioInboxFold:
|
|||
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
|
||||
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
|
||||
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
|
||||
# Positive control FIRST — prove the event happened before asserting it
|
||||
# carried nothing. A negative over an EMPTY population is green for the
|
||||
# wrong reason: were run_portfolio to stop prompting the model at all,
|
||||
# the assertion below would keep passing while proving nothing about
|
||||
# the marker's channel. Measured today: 6 prompts.
|
||||
assert client.calls, "no prompt was issued — the negative below would be vacuous"
|
||||
assert not any(INBOX_MARKER in prompt for _role, prompt in client.calls)
|
||||
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue