open/portfolio-optimiser
1
0
Fork 0
Code Issues Releases 3 Activity

fix(docs): rett README-statusen — to usanne paastander paa offentlig flate (ORDRE 20260825T122454Z)

Browse source 
"no live-model run yet" var falskt siden 14.08 (docs/2026-08-14-fase1b-forste-levende-kjoring.md):
en live koersel mot gpt-4.1-mini konkluderte "rejected" — korrekt utfall, men modellen fant opp
en kostkode. "sibling built in parallel" stemte ikke: po-claude er PARKERT (STATE.md). Begge
punkt verifisert mot primaerkilde foer retting (Verifiseringsloven ansikt 3).

Statusavsnittet er ogsaa gjort mer noekternt: ingen validerte forslag mot levende modell enna,
ingen ekte ekspertdommer i treet (kun AI-forfattede froe, merket som saadan).

Utvidet test_public_surface_claims_loadbearing.py med to nye rode->groenne gater (Iron Law):
begge paastandene kan ikke gjeninnfores usett. Fant og fikset en linjebrudd-felle i egen test
underveis (blockquote-wrap gjorde "no live-model run yet" usynlig for et raatt substring-soek).

Ingen kodeendring i src/. Ingen push til open (Azure-gatet, egen ordre). 1025 passed / 5 skipped.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RF8QXXBWJ917Uj9zMf18nz
This commit is contained in:
Kjell Tore Guttormsen 2026-08-25 15:07:08 +02:00
commit 932ece345b
2 changed files with 89 additions and 18 deletions
Download patch file Download diff file Expand all files Collapse all files

24
README.md
View file

@ -261,16 +261,24 @@ per amount, before anything is summed.
- **Not a model benchmark.** The end-to-end proof runs offline against a scripted stand-in client:
it shows that the loop closes, not how well a given LLM proposes or judges.
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests, and the
> end-to-end proof is an **offline simulation** with a scripted stand-in client — no live-model
> run yet. The **ingest layer** (real data sources) is implemented — file/CSV and SQL on both
> stacks with bit-identical golden extractions from the shared spec, plus HTTP as a MAF-only
> demonstrated extension point against a local mock — but exercised only against committed
> fixtures: no bundle has yet been materialized from a live source. Ingested content passes a
> **content gate** (`materialize_gated`) that scans every generated concept with
> **Status:** the full 8-step agentic loop is wired and proven with load-bearing tests. The
> end-to-end proof is primarily an **offline simulation** with a scripted stand-in client, but one
> **live run** against a real endpoint (`gpt-4.1-mini`, 2026-08-14) has also completed: it ended
> in a correct `rejected` outcome — the deterministic validator caught a cost line the model had
> invented outright (a code absent from the knowledge base), on the tolerance gate rather than the
> stricter existence gate, because the bundle it ran against ships no cost baseline to anchor
> against. No run has yet produced a **validated** proposal against a live model, and every
> human-in-the-loop verdict currently seeded into the knowledge base is a synthetic, AI-authored
> seed marked as such — no genuine expert verdict has entered the tree yet. The **ingest layer**
> (real data sources) is implemented — file/CSV and SQL on both stacks with bit-identical golden
> extractions from the shared spec, plus HTTP as a MAF-only demonstrated extension point against a
> local mock — but exercised only against committed fixtures: no bundle has yet been materialized
> from a live source. Ingested content passes a **content gate** (`materialize_gated`) that scans
> every generated concept with
> [`llm-ingestion-guard`](https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security)
> before any of it reaches the bundle; a refused run writes nothing. A sibling implementation of
> the same method on the **Claude Agents SDK** is built in parallel from the same shared spec.
> the same method on the **Claude Agents SDK** exists in a separate repository but is currently
> **parked**, not developed in parallel.
> **Disclaimer — technical framework only.** Deploying organizations own their processing
> purposes and assessments (DPIA, risk/ROS, security review). The framework ships the technical

81
tests/test_public_surface_claims_loadbearing.py
View file

@ -1,7 +1,7 @@
"""Load-bearing gates on two claims the PUBLISHED surface makes about itself.
"""Load-bearing gates on claims the PUBLISHED surface makes about itself.
AAA+ criterion A5 is that no claim on the public surface is untrue. Two of this repo's claims are
made in prose that no test could see, and both drift silently:
AAA+ criterion A5 is that no claim on the public surface is untrue. Four of this repo's claims are
made in prose that no test could see, and all four drift silently:
1. ``env.template`` tells the reader which credential the AZURE profile resolves. It said
``DefaultAzureCredential`` while :mod:`portfolio_optimiser.backends` has never constructed one —
@ -10,12 +10,22 @@ made in prose that no test could see, and both drift silently:
2. ``README.md`` publishes a wheel-install command that spells the wheel's FILENAME, and a wheel
filename carries the version. A version bump moves the file the build produces without touching
the README, leaving a stranger with an install command for a file that does not exist.
3. ``README.md``'s status paragraph said "no live-model run yet". Measured false as of the F1
review (2026-08-25): a live run against ``gpt-4.1-mini`` completed 2026-08-14 (see
``docs/2026-08-14-fase1b-forste-levende-kjoring.md``) and ended in a correct ``rejected``
outcome.
4. ``README.md``'s status paragraph said the Claude Agents SDK sibling is "built in parallel".
Measured false the same day: ``STATE.md`` records it as parked.
Both gates read the source artefacts as RAW TEXT, because that is the only thing that can see prose.
Both are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES ``DefaultAzureCredential``
four times in the comments that explain why it is not used, so a whole-file substring check would be
red on exactly the prose it protects (this repo's 08-09 defect class, and the reason the handover
package's python-only gate matches archive member NAMES rather than document prose).
Most gates read the source artefacts as RAW TEXT, because that is the only thing that can see
prose. Gates 1 and 2 are LINE-ANCHORED rather than substring-matched: ``backends.py`` NAMES
``DefaultAzureCredential`` four times in the comments that explain why it is not used, so a
whole-file substring check would be red on exactly the prose it protects (this repo's 08-09 defect
class, and the reason the handover package's python-only gate matches archive member NAMES rather
than document prose). Gates 3 and 4 are literal regression guards on the specific wording measured
false — ``STATE.md`` is local-only (gitignored, not shipped to a fresh clone) and cannot serve as a
portable ground-truth source for gate 4, so that gate cannot verify the CURRENT sibling status,
only refuse the specific false claim already made once.
Each positive assertion is paired with a CONTROL that the thing being searched for is actually
present. An extractor that silently finds nothing makes a gate that can only ever be green, which
@ -34,6 +44,7 @@ _ENV_TEMPLATE = _REPO_ROOT / "env.template"
_README = _REPO_ROOT / "README.md"
_PYPROJECT = _REPO_ROOT / "pyproject.toml"
_BACKENDS = _REPO_ROOT / "src" / "portfolio_optimiser" / "backends.py"
_LIVE_RUN_EVIDENCE = _REPO_ROOT / "docs" / "2026-08-14-fase1b-forste-levende-kjoring.md"
# The credential is chosen on ONE assignment statement. Reading the credential names off that line —
# rather than off the whole module — is what keeps the explanatory comments out of the measurement.
@ -43,6 +54,9 @@ _CREDENTIAL_CALL = re.compile(r"(\w*Credential)\s*\(")
# A wheel filename spells the distribution, the version and the tags. The version is the drifting part.
_WHEEL_FILENAME = re.compile(r"portfolio_optimiser-(?P<version>[0-9][^-\s]*)-py3-none-any\.whl")
# The completed-live-run record on the evidence doc, tolerant of the exact column width used there.
_LIVE_RUN_OUTCOME = re.compile(r"outcome_type:\s+rejected")
# `[project]`'s own version line: the value hatchling stamps into the wheel filename.
_PROJECT_VERSION = re.compile(r'^version\s*=\s*"(?P<version>[^"]+)"', re.MULTILINE)
@ -133,7 +147,56 @@ def test_readme_wheel_command_cites_the_built_version() -> None:
)
@pytest.mark.parametrize("artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS])
# --------------------------------------------------------------------------------------------
# F1 (2026-08-25) — the README status paragraph does not repeat two claims measured false
# --------------------------------------------------------------------------------------------
def test_live_run_evidence_records_a_completed_run() -> None:
"""CONTROL. The negative assertion below is only meaningful while this run is on record."""
text = _LIVE_RUN_EVIDENCE.read_text(encoding="utf-8")
assert _LIVE_RUN_OUTCOME.search(text), (
f"{_LIVE_RUN_EVIDENCE} no longer records a completed live run with outcome_type: "
"rejected — the claim the status paragraph must not contradict has nothing to guard it"
)
def _readme_prose() -> str:
"""``README.md``'s status paragraph is a markdown blockquote wrapped across several lines,
each continued with ``"> "``. Rendered markdown collapses those soft line breaks into one
paragraph, so a raw substring search must normalise the same way first — otherwise a claim
that happens to wrap across a line boundary is invisible to the gate (measured: "no
live-model run yet" wraps as "no live-model\\n> run yet" in the raw file).
"""
text = _README.read_text(encoding="utf-8")
return re.sub(r"\n>\s*", " ", text)
def test_readme_status_does_not_claim_no_live_run() -> None:
"""A live run against a real endpoint completed 2026-08-14 (see ``_LIVE_RUN_EVIDENCE``); the
status paragraph may not say otherwise, even though no run has yet produced a *validated*
proposal against a live model.
"""
assert "no live-model run yet" not in _readme_prose(), (
"README.md claims no live-model run has happened, but one completed on 2026-08-14 "
f"(see {_LIVE_RUN_EVIDENCE.name})"
)
def test_readme_status_does_not_claim_sibling_built_in_parallel() -> None:
"""The Claude Agents SDK sibling is parked (``STATE.md``: 'po-claude PARKERT'); the status
paragraph may not claim active parallel development. ``STATE.md`` is local-only and is not
read here — this is a literal regression guard on the specific claim measured false on
2026-08-25, not a check against the sibling's current status.
"""
assert "built in parallel" not in _readme_prose(), (
"README.md claims the Claude Agents SDK sibling is built in parallel, but it is parked"
)
@pytest.mark.parametrize(
"artefact", [_ENV_TEMPLATE, _README, _PYPROJECT, _BACKENDS, _LIVE_RUN_EVIDENCE]
)
def test_guarded_artefacts_exist(artefact: Path) -> None:
"""CONTROL. A missing artefact must fail here rather than turn a gate into a no-op."""
assert artefact.is_file(), f"{artefact} is missing; the gates above would read nothing"