test(loadbearing): positive controls for the static-guard half of the sibling-vacuity class

Point 2 of the sweep, enumerated rather than assumed. STATE's total was right and
its distribution was not: 86 hits confirmed (`assert not X` 41 / `== []` 42 /
`== {}` 2 / `== set()` 1), but per file measured `test_cli_paritet` 13 (STATE said
19), `test_preflight` 10 (11), `test_step7` 4 (6).

AST triage split the 86: 55 hits sit in 50 tests whose assertions are ALL
negative; the other 31 already have a positive sibling assert in the same test.

Two negative results worth recording, because they bound the remaining work:

- The `test_preflight` "clears" family (`_check_credentials(...) == []` and
  friends) is NOT vacuous. Each sits beside a sibling in the same class that
  asserts refusals are non-empty, so a no-op checker turns the sibling red.
  Class-level pairing is a real control; these need no change.
- `test_method_spec_loadbearing.py` already models the right pattern for
  detectors — explicit `test_guard_red_when_*` red-proofs against a mutated COPY.

This commit fixes the class that had no control at all: static/AST guards that
assert an absence without ever showing the scanner can detect a presence.

1. TAUTOLOGICAL RED-PROOFS (both spec guards). `test_guard_red_when_spec_missing`
   asserted a file is absent from a fresh `tmp_path` — true by construction of the
   fixture, and it never called the guard it is named for. It would have stayed
   green with `test_spec_is_present` deleted outright. Both now exercise the same
   `_spec_is_present` predicate the guard calls, in both directions.

2. MISSING RED-PROOF. `test_spec_keeps_structure_markers` had none, unlike its
   toolkit and contract-field siblings: with `_STRUCTURE_MARKERS` emptied or
   `_missing_markers` stubbed to `[]` it reported green forever. Added
   `test_guard_red_when_marker_removed`, parametrized over all 21 markers.

3. BLIND IMPORT SCANNERS (costsim x2, okf, preflight, notify). Every one asserted
   `not names & {forbidden}` or `outside == set()` with nothing showing `names`
   was non-empty — an empty scan satisfies them exactly as well as real purity.
   `test_okf_is_pure_stdlib`'s subset check is likewise trivially true of the
   empty set, so it did not guard its neighbour either. Each now asserts a
   known-present module first. The notify guard gets the strongest form
   available: it proves the detector DOES match a network import inside the seam,
   so the matcher itself is shown to work rather than only its silence.

Value-proved, not merely detach-proved. Seven vacuity mutations run against the
NEW tests: all seven RED, each dying on the intended control line. The same
mutations run against the PRE-CHANGE tests (session edits stashed): all five
applicable ones GREEN — blind to the vacuity they were meant to catch. Green
before, red after, same mutation, is the value-proof.

Harness held original bytes in memory, restored in `finally`, sha256-verified
every restore, and checked each run ACTUALLY RAN (a wrong test id yields rc!=0
and mimics red). `git status` clean before and after.

Remaining in the class and NOT closed here: ~45 all-negative tests, mostly CLI
refusal (`calls == []` after a refused invocation) and empty-default
(`missing dir -> []`). Listed in STATE, not silently dropped.

Suite 690 -> 711.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DJmse16bEkaSBtvXhncEUc
This commit is contained in:
Kjell Tore Guttormsen 2026-08-01 20:01:21 +02:00
commit fae5b22578
6 changed files with 64 additions and 4 deletions

View file

@ -155,6 +155,11 @@ class TestCostsimPurity:
def test_costsim_never_imports_an_agent_toolkit(self) -> None:
names = _imported_module_names(SRC_PKG / "costsim.py")
# Positive control: the scanner actually RESOLVED imports from this file.
# Without it, an empty `names` — a renamed module, a parse that yielded
# nothing, a scanner narrowed to the wrong node types — satisfies the
# intersection below just as well as real purity does.
assert "json" in names
assert not names & {"claude_agent_sdk", "anthropic"}
@ -221,4 +226,7 @@ class TestBundledExampleAndCli:
def test_no_network_import_in_costsim() -> None:
# Belt on the offline invariant: not even the stdlib socket/urllib slip in.
names = _imported_module_names(SRC_PKG / "costsim.py")
# Positive control — the scan is non-empty, so the absence below is measured
# rather than inherited from a scanner that found nothing at all.
assert "json" in names
assert not names & {"socket", "urllib", "http", "requests", "httpx"}

View file

@ -112,9 +112,14 @@ def _row_renamed(text: str, field: str) -> str:
# --- The guard itself (against the real spec) ---------------------------------------
def _spec_is_present(path: Path) -> bool:
"""The presence predicate itself, so the red-proof can exercise THE SAME one."""
return path.is_file()
def test_spec_is_present() -> None:
# RED if the spec goes missing (the layer stops being implementable from spec alone).
assert SPEC.is_file(), "ingest-spec.md missing — subtree pull the commons contract"
assert _spec_is_present(SPEC), "ingest-spec.md missing — subtree pull the commons contract"
def test_spec_names_no_agent_toolkit() -> None:
@ -142,7 +147,12 @@ def test_spec_documents_contract_field(field: str) -> None:
def test_guard_red_when_spec_missing(tmp_path: Path) -> None:
assert not (tmp_path / "ingest-spec.md").is_file()
# Was VACUOUS — see the twin in ``test_method_spec_loadbearing.py``: it asserted
# a file is absent from a fresh ``tmp_path``, true by construction, and never
# touched the guard it is named for. Now it exercises THE SAME predicate the
# guard calls, both directions, positive control first.
assert _spec_is_present(SPEC)
assert not _spec_is_present(tmp_path / "ingest-spec.md")
@pytest.mark.parametrize("toolkit", _FORBIDDEN_TOOLKITS)

View file

@ -115,9 +115,14 @@ def _undocumented_fields(text: str) -> list[str]:
# --- The guard itself (against the real spec) ---------------------------------------
def _spec_is_present(path: Path) -> bool:
"""The presence predicate itself, so the red-proof can exercise THE SAME one."""
return path.is_file()
def test_spec_is_present() -> None:
# RED if the spec goes missing (the method stops being implementable from spec alone).
assert SPEC.is_file(), "method-spec.md missing — subtree pull the commons contract"
assert _spec_is_present(SPEC), "method-spec.md missing — subtree pull the commons contract"
def test_spec_keeps_structure_markers() -> None:
@ -142,7 +147,26 @@ def test_spec_documents_contract_field(field: str) -> None:
def test_guard_red_when_spec_missing(tmp_path: Path) -> None:
assert not (tmp_path / "method-spec.md").is_file()
# Was VACUOUS: it asserted a file is absent from a fresh ``tmp_path`` — true by
# construction of the fixture, and it never touched the guard it is named for.
# It would have stayed green with ``test_spec_is_present`` deleted outright.
# Now it exercises THE SAME predicate the guard calls, both directions.
# Positive control first: without it, "False for a missing path" would also
# hold for a predicate that is False for everything.
assert _spec_is_present(SPEC)
assert not _spec_is_present(tmp_path / "method-spec.md")
@pytest.mark.parametrize("marker", _STRUCTURE_MARKERS)
def test_guard_red_when_marker_removed(tmp_path: Path, marker: str) -> None:
# The structure-marker guard had NO red-proof, unlike its toolkit and
# contract-field siblings: with ``_STRUCTURE_MARKERS`` emptied or
# ``_missing_markers`` stubbed to ``[]``, it would report green forever.
# Positive control: the REAL spec is missing no marker, so the detection
# below is the mutation being caught, not a spec that was already broken.
assert _missing_markers(SPEC.read_text(encoding="utf-8")) == []
mutated = SPEC.read_text(encoding="utf-8").replace(marker, "")
assert marker in _missing_markers(mutated)
@pytest.mark.parametrize("toolkit", _FORBIDDEN_TOOLKITS)

View file

@ -164,6 +164,16 @@ class TestNoSocketOutsideSeam:
n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == _SEAM_FUNC
)
seam_node_ids = {id(n) for n in _import_nodes(seam)}
# Positive control: the seam DOES import a network module, and this very
# machinery detects it. Without this, `outside == set()` would hold just
# as well if `_import_nodes` returned nothing, `_modules_of` resolved no
# names, or `_NETWORK_MODULES` were empty — i.e. if the detector were
# incapable of ever flagging anything. It proves the matcher, not just
# the absence.
inside: set[str] = set()
for node in _import_nodes(seam):
inside |= _modules_of(node) & _NETWORK_MODULES
assert inside, "the seam should carry the ONLY network import; detector matched none"
outside: set[str] = set()
for node in _import_nodes(tree):
if id(node) not in seam_node_ids:

View file

@ -418,6 +418,11 @@ class TestContextSeamPurity:
@pytest.mark.parametrize("module", ["okf.py", "experience.py"])
def test_context_seam_never_imports_an_agent_toolkit(self, module: str) -> None:
names = _imported_module_names(SRC_PKG / module)
# Positive control: the scanner resolved real imports from this module.
# An empty `names` satisfies the intersection below exactly as well as
# purity does — and `test_okf_is_pure_stdlib`'s subset check is likewise
# trivially true of the empty set, so neither guards the other.
assert "dataclasses" in names
assert not names & {"claude_agent_sdk", "anthropic"}
def test_okf_is_pure_stdlib(self) -> None:

View file

@ -248,6 +248,9 @@ class TestPreflightIsOffline:
# The offline seam (grep-guard, AST form): no socket/httpx path exists —
# importing any of these is the detach that turns this RED.
names = _imported_module_names(SRC_PKG / "preflight.py")
# Positive control: the scan resolved real imports, so the absence below
# is a measured property of preflight.py and not an empty scan.
assert "importlib" in names
assert not names & {"socket", "urllib", "http", "requests", "httpx", "anthropic"}
def test_no_sdk_completion_is_called(self) -> None: