open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 1 Activity

test(docs): gate constants cited in live docs against the code (kø-(h))

Browse source 
docs/extending.md claimed SEMANTIC_WEIGHT_DEFAULT = 0.5 long after the code
lowered it to 0.25. It was found by accident while editing the neighbouring
line; nothing in the suite would ever have caught it. This is that gate.

Both bounds were MEASURED, not assumed:

- SCREAMING_CASE discriminates exactly. Across every numeric `name = value`
  code span in docs/ it selects the 2 real module constants and rejects all
  10 kwargs/locals (max_attempts=3, concurrency=3, realiseringsgrad=0.79).
  Ordinary prose stays freely editable, so the gate has no reason to be
  switched off.
- Dated documents are observations, not contract. A spike finding or a July
  review records what was true when measured; rewriting it to track the code
  would falsify the record.

Measuring also rewrote the ambiguity rule: SEMANTIC_WEIGHT_DEFAULT is bound
in both semretrieval and run (a re-export), so a "same name in two modules"
check would have been RED on today's code. Only DIVERGENT values are refused.

Fail-closed throughout, per write_concept_file / read_spend: an unknown
constant name is an error rather than a skip, and a document that is neither
listed live nor recognisably archived goes RED asking to be classified —
otherwise a new guide would be silently unguarded.

Load-bearing MEASURED against the whole 638-test suite:
- drift the DOC (the original defect) -> only this gate goes red; the other
  637 stay green, so it covers ground nothing else did
- drift the CODE -> this gate and the semretrieval weight gate both go red
- make an unknown name tolerant -> red
- drop the only citing doc from the live list -> red (twice: coverage and
  classification)
- add a new unclassified guide -> red
- (control) remove this gate entirely, code still drifted -> the adjacent
  semretrieval gate still goes red, so nothing is masked in either direction

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01D9AAyWtMqr4HjftKaegTtS
This commit is contained in:
Kjell Tore Guttormsen 2026-08-03 22:22:43 +02:00
commit 1515a474d6
1 changed files with 183 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

183
tests/test_doc_constant_sync_loadbearing.py Normal file
View file

@ -0,0 +1,183 @@
"""Load-bearing tests for doc↔code constant sync (kø-(h)).
``docs/extending.md`` claimed ``SEMANTIC_WEIGHT_DEFAULT = 0.5`` long after the code had lowered it
to ``0.25``. The drift was found BY ACCIDENT while editing the neighbouring line; nothing in the
suite would ever have caught it. This module is the gate that would have.
**Scope is deliberately narrow (90 %).** Only SCREAMING_CASE module constants cited with their
value inside an inline code span, and only in LIVE documents. Both bounds were MEASURED, not
assumed:
* **SCREAMING_CASE discriminates exactly.** Across every numeric ``name = value`` code span in
``docs/`` it selects the 2 real module constants (``SEMANTIC_WEIGHT_DEFAULT``,
``MAX_SAVING_FRACTION``) and rejects all 10 keyword arguments and locals (``max_attempts=3``,
``concurrency=3``, ``realiseringsgrad=0.79``, …). SCREAMING_CASE *is* the claim "this is a module
constant"; ordinary prose and kwargs stay editable and never trip the gate.
* **Dated documents are observations, not contract.** A spike finding or a July review records what
was true when it was measured; rewriting it to track the code would falsify the record. LIVE
documents describe the framework as it is NOW, which is what a reader acts on.
The gate REFUSES rather than skips, in this repo's fail-closed tradition (``write_concept_file``,
``VerdictFrontmatterError``, ``read_spend``): an unknown constant name is an error, never a silent
pass, because a typo that quietly disables a guard is exactly the masking class this repo hunts.
For the same reason document classification is fail-closed — a new document that is neither listed
LIVE nor recognisably archived goes RED asking to be classified, so coverage cannot quietly rot.
Ambiguity is narrowed to what actually makes a lookup unsafe: a name bound in several modules is
normal Python re-export (``run`` re-exports ``SEMANTIC_WEIGHT_DEFAULT`` from ``semretrieval``), so
only genuinely DIVERGENT values are refused. Measuring this rewrote the rule — a
"same name in two modules" check would have been RED on today's code.
"""
from __future__ import annotations
import importlib
import pkgutil
import re
from pathlib import Path
import pytest
import portfolio_optimiser
REPO_ROOT = Path(__file__).resolve().parents[1]
DOCS_DIR = REPO_ROOT / "docs"
# Documents that describe the framework AS IT IS NOW. Explicit rather than derived: the companion
# classification test below goes RED on any document that is neither listed here nor archived, so
# this list cannot silently fall behind.
_LIVE_DOCS = (
"README.md",
"docs/extending.md",
"docs/knowledge-base-recipe.md",
)
# A dated path segment marks a point-in-time record. ``docs/fase1-spikes/`` is the one archive whose
# directory name carries no date — the phase-1 spike findings are dated by their phase, not by
# filename.
_DATED = re.compile(r"\d{4}-\d{2}")
_ARCHIVE_DIRS = ("fase1-spikes",)
# A constant citation: SCREAMING_CASE bound to a numeric literal, inside an inline code span.
_CITATION = re.compile(r"`\s*([A-Z][A-Z0-9_]{2,})\s*=\s*(-?\d+(?:\.\d+)?)\s*`")
def _is_archive(rel_path: str) -> bool:
"""A document is archived when its path carries a date or sits in a known archive directory."""
return bool(_DATED.search(rel_path)) or any(f"/{d}/" in f"/{rel_path}" for d in _ARCHIVE_DIRS)
def _module_constants() -> dict[str, set[float]]:
"""Map every public numeric SCREAMING_CASE constant in the package to the DISTINCT values it is
bound to. A name re-exported across modules yields one value and is unambiguous; only divergent
values make a lookup unsafe."""
constants: dict[str, set[float]] = {}
for info in pkgutil.iter_modules(portfolio_optimiser.__path__):
module = importlib.import_module(f"portfolio_optimiser.{info.name}")
for name in dir(module):
if not name.isupper() or name.startswith("_"):
continue
value = getattr(module, name)
if isinstance(value, bool) or not isinstance(value, (int, float)):
continue
constants.setdefault(name, set()).add(float(value))
return constants
def _check_citations(text: str, label: str) -> list[str]:
"""Return one message per citation that does NOT match the code. Unknown and ambiguous names are
failures, never skips — the whole point is that a citation cannot quietly stop being checked."""
constants = _module_constants()
problems: list[str] = []
for lineno, line in enumerate(text.splitlines(), 1):
for name, literal in _CITATION.findall(line):
where = f"{label}:{lineno}"
values = constants.get(name)
if values is None:
problems.append(
f"{where}: cites `{name} = {literal}` but no public numeric constant named "
f"{name} exists in portfolio_optimiser"
)
elif len(values) > 1:
problems.append(
f"{where}: cites `{name}` but it is bound to divergent values "
f"{sorted(values)} — the citation cannot be checked safely"
)
elif float(literal) != next(iter(values)):
problems.append(
f"{where}: says `{name} = {literal}` but the code says {next(iter(values))}"
)
return problems
def _live_doc_paths() -> list[Path]:
return [REPO_ROOT / rel for rel in _LIVE_DOCS]
@pytest.mark.parametrize("rel_path", _LIVE_DOCS)
def test_live_docs_cite_constants_that_match_the_code(rel_path: str) -> None:
"""THE GATE. Every constant a live document cites with its value must equal what the code
actually holds. Goes RED on precisely the drift that shipped undetected: ``extending.md``
saying 0.5 while ``semretrieval.SEMANTIC_WEIGHT_DEFAULT`` is 0.25."""
path = REPO_ROOT / rel_path
assert path.is_file(), f"{rel_path} is listed LIVE but does not exist"
problems = _check_citations(path.read_text(encoding="utf-8"), rel_path)
assert not problems, "documentation cites constants the code contradicts:\n" + "\n".join(
problems
)
def test_a_drifted_citation_is_caught() -> None:
"""The gate must FAIL on drift, not merely pass on a clean tree. Uses the real constant so the
check is anchored to live code rather than to a fixture's idea of it."""
real = _module_constants()["SEMANTIC_WEIGHT_DEFAULT"]
assert len(real) == 1, "test premise: the constant must be unambiguous"
wrong = next(iter(real)) + 1.0
problems = _check_citations(f"blends `SEMANTIC_WEIGHT_DEFAULT = {wrong}` today", "synthetic.md")
assert len(problems) == 1, problems
assert "but the code says" in problems[0]
def test_an_unknown_constant_name_is_refused_not_skipped() -> None:
"""Fail-closed: a name that resolves to nothing is an error. If unknown names were skipped, a
typo in a doc — or a constant deleted from the code — would silently retire the check."""
problems = _check_citations("see `NO_SUCH_CONSTANT_HERE = 7` for details", "synthetic.md")
assert len(problems) == 1, problems
assert "no public numeric constant" in problems[0]
def test_prose_and_keyword_arguments_do_not_trip_the_gate() -> None:
"""The measured bound: ordinary prose, kwargs and lowercase locals stay freely editable. A gate
that made routine documentation edits brittle would be switched off, catching nothing."""
text = (
"run with `max_attempts=3` and `concurrency=1`, giving `realiseringsgrad=0.79` "
"against a p50 of `p50 = 95443.98966314227`"
)
assert _check_citations(text, "synthetic.md") == []
def test_every_document_is_classified_live_or_archive() -> None:
"""Fail-closed coverage: a new document must be deliberately classified. Without this, adding a
live guide would leave it silently unguarded — a hole indistinguishable from a passing gate."""
live = set(_LIVE_DOCS)
unclassified = [
rel
for path in [REPO_ROOT / "README.md", *sorted(DOCS_DIR.rglob("*.md"))]
if (rel := path.relative_to(REPO_ROOT).as_posix()) not in live and not _is_archive(rel)
]
assert not unclassified, (
"these documents are neither listed in _LIVE_DOCS nor recognisable as archive — classify "
"them (a live guide belongs in _LIVE_DOCS; a point-in-time record needs a dated path):\n"
+ "\n".join(unclassified)
)
def test_the_gate_actually_reaches_a_real_citation() -> None:
"""Coverage, not vocabulary: a gate that scans only documents containing no citations would be
green forever. At least one live document must genuinely exercise the comparison."""
cited = [
(rel, name)
for path, rel in ((p, p.relative_to(REPO_ROOT).as_posix()) for p in _live_doc_paths())
for name, _ in _CITATION.findall(path.read_text(encoding="utf-8"))
]
assert cited, "no live document cites any constant — the gate would pass vacuously"