Kjell Tore Guttormsen
2d1264088e
Fase 1b siste trinn: forste bundne levende kjoring over HELE run_project-stien mot Foundry (gpt-4-1-mini). Okt 37s uttalte aerlighets-grense er lukket ved maling: ingen -parse-failures.json i outboksen, altsa parset hvert eneste genererings-svar. assumptions-normaliseringen virket ende-til-ende. Utfall: rejected pa P90 (claimed 34500 > feasible 11488), checker approve. Kjoringen KONKLUDERTE - per pre-registreringen det bestatte utfallet. De to falsifisererne skilte lag for forste gang mot en levende modell. FUNN storre enn den gronne testen: modellen fant opp kostkoden EL-LIGHTING-OP-HR (null treff i kunnskapsbasen). Avvisningen var riktig men skjedde pa 30%-cap-en, ikke stage 0 - bundelen shipper ingen cost-baseline.json, sa S4.0-forankringen var inaktiv. Ko-fort, ikke rettet her. Gate-designet er ovis beslutning, tatt for koding: - TREDJE distinkt opt-in PORTFOLIO_LIVE_FULL_RUN (truthiness, 4b-invarianten). Begge eksisterende live-tester gater pa SAMME to Foundry-variabler, sa gjenbruk ville latt den billige proben fyre den dyre kjoringen - stigen i maleprotokollen ville kollapset til ett trinn. MALT: den dyre SKIPPET med begge Foundry-variablene satt. - Asserten i EN kopi (conftest.assert_full_run_contract, ko-(p)), smal med vilje: fravaer av parse-failures-artefaktet + at validatoren avgjorde. En rejected BESTAR - pastanden er schema-aksept, ikke modell-dommekraft. - Iron Law uten a betale to ganger: diskrimineringen bevist OFFLINE av test_live_full_run_contract.py. To mutasjoner, hver sin signatur: detach artefakt-sjekken (T1 rod ALENE) - raise ubetinget (T2 rod ALENE). STATE-premiss korrigert: "test_foundry_profile_live dekker KUN klient-nivaet" var upresist - test_portfolio_live.py dekket allerede fan-outen, men dens len(runs)==1 kan ikke skille validert fra avvist og bar derfor ikke pastanden. 869 passed / 5 skipped (fra 867/4), ruff+format+mypy rene. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GF7va4cpRiuf79kTzAi3vW
86 lines
4.2 KiB
Python
86 lines
4.2 KiB
Python
"""Fase 1b, last step — GATED live run over the WHOLE ``run_project`` path (måleprotokoll §4.4).
|
|
|
|
NOT default CI, and NOT gated like its two siblings. This is the expensive arm: it drives the
|
|
complete vertical slice — bundle navigation, the maker/checker debate, generation under
|
|
``response_format``, and the deterministic validator — against a real Foundry deployment.
|
|
|
|
**Why a THIRD environment variable, and why it is load-bearing.** ``test_foundry_profile_live.py``
|
|
(client-level probe) and ``test_portfolio_live.py`` (``run_portfolio`` fan-out) both skip on exactly
|
|
``PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT`` + ``PORTFOLIO_FOUNDRY_DEPLOYMENT``. Reusing that pair here
|
|
would mean the moment an operator exports the two variables to run the CHEAP one-word probe, this
|
|
full run fires too — collapsing the måleprotokoll's whole point (§1: *"bevis så mye som mulig før
|
|
det dyre trinnet, så en feil er attribuerbar"*) into a single step, and spending money on a rung
|
|
whose predecessors have not been shown green. ``PORTFOLIO_LIVE_FULL_RUN`` is therefore a separate,
|
|
deliberate opt-in, read on **truthiness, not presence** (the Fase 4b invariant: an exported-but-empty
|
|
value is a shell accident, not a decision).
|
|
|
|
``PORTFOLIO_MODEL_MAP`` is part of the skip condition for a different reason — attribution. ``run.py``
|
|
stamps provenance with the deployment NAME before any client is built (målt 4e), so without the map
|
|
the run fails for a CONFIGURATION reason while looking exactly like a model failure. Skipping is
|
|
honest; failing there would misattribute.
|
|
|
|
**What this asserts is narrow on purpose** — see ``conftest.assert_full_run_contract``. The claim
|
|
being felled is *"the emitted structured schema is accepted by the live endpoint"* (økt 37's stated
|
|
honesty limit), NOT "the model proposes well". A validator REJECTION passes this test: the run
|
|
reached the deterministic gate with a parsed candidate, which is the whole question. The contract's
|
|
ability to discriminate is proven offline and for free by
|
|
``tests/test_live_full_run_contract.py`` — the paid call here is the measurement, not the proof that
|
|
the instrument works.
|
|
|
|
Outcomes are pre-registered in ``docs/2026-08-14-fase1b-forste-levende-kjoring.md`` §5, written
|
|
BEFORE the run, so the write-up cannot be negotiated after the fact.
|
|
|
|
The round/token caps are the SAME ones the first live run died on. They are deliberately not
|
|
raised: if the ledger fires again that is information, and raising it spends more on a path that may
|
|
still be broken.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
from pathlib import Path
|
|
|
|
import pytest
|
|
from conftest import assert_full_run_contract
|
|
|
|
from portfolio_optimiser.run import RunResult, run_project
|
|
from portfolio_optimiser.verdicts import VerdictStore
|
|
|
|
BUNDLE_DIR = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
|
_PROJECT_ID = "BYGG-KONTOR-NORD"
|
|
|
|
_ENDPOINT = os.environ.get("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT")
|
|
_DEPLOYMENT = os.environ.get("PORTFOLIO_FOUNDRY_DEPLOYMENT")
|
|
_MODEL_MAP = os.environ.get("PORTFOLIO_MODEL_MAP")
|
|
#: Truthiness, not presence (Fase 4b): ``PORTFOLIO_LIVE_FULL_RUN=`` must NOT arm a paid run.
|
|
_OPTED_IN = bool(os.environ.get("PORTFOLIO_LIVE_FULL_RUN"))
|
|
|
|
_SKIP = not (_ENDPOINT and _DEPLOYMENT and _MODEL_MAP and _OPTED_IN)
|
|
|
|
|
|
@pytest.mark.skipif(
|
|
_SKIP,
|
|
reason=(
|
|
"paid full run not armed (set PORTFOLIO_LIVE_FULL_RUN=1 alongside "
|
|
"PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT + PORTFOLIO_FOUNDRY_DEPLOYMENT + PORTFOLIO_MODEL_MAP)"
|
|
),
|
|
)
|
|
async def test_full_run_reaches_the_validator_on_a_live_model(tmp_path: Path) -> None:
|
|
"""The whole slice against a real deployment: every generation reply must come back in the
|
|
requested shape, and the deterministic validator must decide on it."""
|
|
outbox_dir = tmp_path / "outbox"
|
|
run_id = "live-full-001"
|
|
|
|
result = await run_project(
|
|
_PROJECT_ID,
|
|
"azure",
|
|
docs_dir=str(BUNDLE_DIR),
|
|
bundle_dir=str(BUNDLE_DIR),
|
|
verdict_input={"decision": "approved", "rationale": "expert reviewed (live 1b)"},
|
|
store=VerdictStore(verdicts=[]),
|
|
outbox_dir=str(outbox_dir),
|
|
run_id=run_id,
|
|
)
|
|
|
|
assert isinstance(result, RunResult)
|
|
assert_full_run_contract(result, outbox_dir, run_id)
|