feat(1b): skjemaet ER akseptert av det levende endepunktet - malt, ikke resonnert [skip-docs]
Fase 1b siste trinn: forste bundne levende kjoring over HELE run_project-stien mot Foundry (gpt-4-1-mini). Okt 37s uttalte aerlighets-grense er lukket ved maling: ingen -parse-failures.json i outboksen, altsa parset hvert eneste genererings-svar. assumptions-normaliseringen virket ende-til-ende. Utfall: rejected pa P90 (claimed 34500 > feasible 11488), checker approve. Kjoringen KONKLUDERTE - per pre-registreringen det bestatte utfallet. De to falsifisererne skilte lag for forste gang mot en levende modell. FUNN storre enn den gronne testen: modellen fant opp kostkoden EL-LIGHTING-OP-HR (null treff i kunnskapsbasen). Avvisningen var riktig men skjedde pa 30%-cap-en, ikke stage 0 - bundelen shipper ingen cost-baseline.json, sa S4.0-forankringen var inaktiv. Ko-fort, ikke rettet her. Gate-designet er ovis beslutning, tatt for koding: - TREDJE distinkt opt-in PORTFOLIO_LIVE_FULL_RUN (truthiness, 4b-invarianten). Begge eksisterende live-tester gater pa SAMME to Foundry-variabler, sa gjenbruk ville latt den billige proben fyre den dyre kjoringen - stigen i maleprotokollen ville kollapset til ett trinn. MALT: den dyre SKIPPET med begge Foundry-variablene satt. - Asserten i EN kopi (conftest.assert_full_run_contract, ko-(p)), smal med vilje: fravaer av parse-failures-artefaktet + at validatoren avgjorde. En rejected BESTAR - pastanden er schema-aksept, ikke modell-dommekraft. - Iron Law uten a betale to ganger: diskrimineringen bevist OFFLINE av test_live_full_run_contract.py. To mutasjoner, hver sin signatur: detach artefakt-sjekken (T1 rod ALENE) - raise ubetinget (T2 rod ALENE). STATE-premiss korrigert: "test_foundry_profile_live dekker KUN klient-nivaet" var upresist - test_portfolio_live.py dekket allerede fan-outen, men dens len(runs)==1 kan ikke skille validert fra avvist og bar derfor ikke pastanden. 869 passed / 5 skipped (fra 867/4), ruff+format+mypy rene. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GF7va4cpRiuf79kTzAi3vW
This commit is contained in:
parent
986fc19350
commit
2d1264088e
5 changed files with 408 additions and 0 deletions
|
|
@ -192,3 +192,52 @@ def docs_dir(tmp_path) -> str:
|
|||
encoding="utf-8",
|
||||
)
|
||||
return str(d)
|
||||
|
||||
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
# Fase 1b — the FULL-RUN contract, in ONE copy (kø-(p): a second copy of an assertion drifts).
|
||||
# ------------------------------------------------------------------------------------------------
|
||||
|
||||
|
||||
#: What the paid run must fell, stated as an assertion rather than as prose. The open honesty limit
|
||||
#: after økt 37 is narrow and specific: *"that the emitted structured schema IS accepted by the LIVE
|
||||
#: endpoint is NOT verified — the tests prove conformance with the DOCUMENTED subset, not
|
||||
#: acceptance."* So the contract asserts schema ACCEPTANCE, never model JUDGEMENT.
|
||||
#:
|
||||
#: The discriminator is an artefact this repo already owns: ``{run_id}-parse-failures.json`` is
|
||||
#: written if and ONLY if some reply failed to parse (økt 35 invariant — "the file's presence is the
|
||||
#: signal"). Its ABSENCE beside a RunResult therefore proves that every generation reply came back
|
||||
#: in the requested shape, which is exactly what "the live endpoint honoured the schema" means.
|
||||
#:
|
||||
#: ``validator_decision`` is the second half: it mirrors the VALIDATOR alone (never the checker), so
|
||||
#: reading it proves the deterministic gate actually ran on a parsed candidate. Both ``validated``
|
||||
#: and ``rejected`` satisfy the contract — a P90 rejection is a run that CONCLUDED, and demanding
|
||||
#: ``validated`` would be asserting that the model reasons well, which no schema can promise and
|
||||
#: which one paid run could not establish anyway.
|
||||
def assert_full_run_contract(result, outbox_dir, run_id: str) -> None:
|
||||
"""Assert the Fase 1b full-run contract on a completed ``run_project`` result.
|
||||
|
||||
Two things, and deliberately nothing else:
|
||||
|
||||
1. **No parse-failure artefact** — every reply parsed, i.e. the live endpoint accepted the
|
||||
emitted ``response_format`` schema. This is the honesty limit being felled.
|
||||
2. **The validator was reached and decided** — ``provenance.validator_decision`` is one of the
|
||||
two decisions the deterministic gate emits.
|
||||
|
||||
Deliberately NOT asserted: ``checker_verdict``, token counts, ``validated_count``, or the
|
||||
content of the proposal. Those are model-judgement claims, and one run cannot carry them.
|
||||
"""
|
||||
from pathlib import Path
|
||||
|
||||
artefact = Path(outbox_dir) / f"{run_id}-parse-failures.json"
|
||||
if artefact.exists():
|
||||
# Quote the evidence in the failure message: the whole point of økt 35 was that the operator
|
||||
# should never again have to guess WHY a reply did not parse.
|
||||
raise AssertionError(
|
||||
f"the endpoint did NOT honour the structured schema — {artefact.name} exists.\n"
|
||||
f"{artefact.read_text(encoding='utf-8')[:2000]}"
|
||||
)
|
||||
assert result.provenance.validator_decision in {"validated", "rejected"}, (
|
||||
"the deterministic validator never decided — the run did not reach the gate with a "
|
||||
f"parsed candidate (validator_decision={result.provenance.validator_decision!r})"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue