open/portfolio-optimiser-claude
1
0
Fork 0
Code Issues Releases 1 Activity

test(loadbearing): measure the last 8 all-negative tests instead of assuming them

Browse source 
Point 2 of the vacuity sweep is now MEASURED, not paired-by-reading. Every
one of the 8 got a mutation that detaches the seam it claims to guard, run
through a harness that asserts the anchor is unique before mutating, restores
in `finally`, and sha256-verifies the restore.

Six were value-proven — the negative itself went RED under its detach:
  hitl :260  load_routing invents a default        -> RED
  hitl :298  route_pending hardcodes a fallback    -> RED
  step7 :145 the is_dir() guard deleted            -> RED
  step7 :171 the §4.2 vocabulary filter deleted    -> RED
  step7 :253 the id grammar off the model          -> RED
  prov  :111 sdk_version becomes required          -> RED

Two did not, and both are fixed here.

hitl :197 — the guard it appeared to prove is DEAD. Deleting the `is_dir()`
early-out from load_outbox_proposals leaves all 711 tests green: the tolerance
comes from `Path.glob`, which yields nothing on a missing directory and never
raises. The contrast is the finding: load_inbox carries an identically-shaped
guard that IS load-bearing, because it walks with `Path.iterdir`, which DOES
raise (measured both ways). Same guard, opposite verdict, and the difference
is the stdlib call behind it — the point-3 lesson one level out, where the
default being pinned belongs to the standard library rather than the SDK.
The stdlib baseline is now anchored explicitly, so a Python that makes glob
raise turns this red instead of quietly promoting a dead line to a seam.
What the test always did prove is kept and stated: replacing the early-out
with a raise turns it red, so it does hold tolerance.

portfolio :175 — the negative asserted over an unheld population. Measured, it
is real today (6 prompts), so the test is not vacuous now; nothing in it says
so, and a run_portfolio that stopped prompting would leave it green while
proving nothing. A positive control now runs first. Value-proven: green before,
red after the same mutation (iterate no projects), and it is that assertion
which fails, not an import.

The sibling repo sent the same rule from the other stack this week, arrived at
independently via its B4 empty-negative: on a negative assert, prove FIRST that
the event happened.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qr6TwWrHDHeukHy3bL4hgb
This commit is contained in:
Kjell Tore Guttormsen 2026-08-09 10:08:12 +02:00
commit 82d46148e2
2 changed files with 26 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

6
tests/test_portfolio_learning_loadbearing.py
View file

@ -179,6 +179,12 @@ class TestPortfolioInboxFold:
projects = _config([("BYGG-KONTOR-NORD", LED_BUNDLE), ("PUMPE-SOR", VFD_BUNDLE)])
client = ScriptedClient(replies=_happy_replies([LED_BUNDLE, VFD_BUNDLE]))
run_portfolio(projects, client, _meter(), top_k=3, max_debate_rounds=3, max_attempts=3)
# Positive control FIRST — prove the event happened before asserting it
# carried nothing. A negative over an EMPTY population is green for the
# wrong reason: were run_portfolio to stop prompting the model at all,
# the assertion below would keep passing while proving nothing about
# the marker's channel. Measured today: 6 prompts.
assert client.calls, "no prompt was issued — the negative below would be vacuous"
assert not any(INBOX_MARKER in prompt for _role, prompt in client.calls)