Kallsted (2) av oerkt 57s fire. explore_prompt + explore_contract whitelistes paa den hostede flaten. De er ikke run_project-parametre - utforskningen kjoerer FOERST og gir run_project et Mandate - saa de KONSUMERES i stedet for aa videresendes, og whitelisten er dermed en TREDELING (_REQUIRED / _OPTIONAL / _CONSUMED). Ett sted (_run_kwargs) avgjoer hvilke felt som naar signaturen: et konsumert felt som blir liggende i kwargs er et argument run_project ikke har, som containeren svarer 500 paa for det som egentlig er en wiring-feil. FASE 4e-REGELEN UTVIDET MED EN NEGATIV HALVDEL, og det var paakrevd: 4e-testen sender HVERT whitelistet felt gjennom den EKTE run_project og asserterer dekning mot _ALLOWED_FIELDS - en assert et konsumert felt ALDRI kan oppfylle. Den positive halvdelen dekker naa _REQUIRED|_OPTIONAL (hvert felt maalt mot inspect.signature(run_project)), og den negative at _CONSUMED er DISJUNKT fra samme signatur. Uten den ville et felt som glir fra konsumert til videresendt vaere nettopp driften 4e finnes for. Fire nekter, alle ved navn og alle paa KALLERENS kanal (400): explore_contract uten explore_prompt - explore_prompt uten explore_contract - explore_prompt uten bundle_dir - enable_plan_review=true. Den siste er nektet HER og ikke i explore(), som ogsaa nekter den: ExplorationError er en RuntimeError, saa aa overlate den til sloeyfa ville svart en kallers konfigurasjonsfeil paa KRASJ-kanalen (500) - samme sammenblanding BudgetExceeded fikk sin egen 429 for aa avslutte. U13-doera er dessuten SYNKRON: den blokkerer sloeyfa paa et menneske, og en HTTP-forespoersel har ingen - invocationen ville hengt i stedet for aa svare. TracingConfigError er derimot en ValueError, saa 400- armen dekket den alt (verifisert, ikke antatt). Load-bearing MAALT (tests/test_explore_callsites_loadbearing.py, 6 nye tester), fire mutasjoner alle roede mot HELE suiten + groenn kontroll 996/5: videresend de konsumerte feltene igjen (2 roede) - detach mandate=-wiringen (1 roed) - detach enable_plan_review-nekten (1) - detach bundle_dir-kravet (1). Golden-transkriptet byte-uendret (ea8c534773acdbe41ae68f2c55724d69aaf8be4f). mypy + ruff rene. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01YRZhBJcxqTcqWyMW6hBttx
562 lines
26 KiB
Python
562 lines
26 KiB
Python
"""Fase 4d hosted entrypoint (``hosting.py`` + root ``main.py``): the Foundry hosted-agent
|
|
runtime contract implemented DIRECTLY around ``run_project`` — the wrapper form BOTH 13.08
|
|
measurements prescribe (hosting-pakka: ``InvocationsHostServer`` finnes kun i bygg som krever
|
|
core>=1.13.0; gjenbrukt workflow: kall-serie [2, 0, 0] — single-use på 1.9.0).
|
|
|
|
Load-bearing surface pinned here:
|
|
|
|
* ``GET /readiness`` → 200. We use NO protocol library (the measurement above), so the endpoint
|
|
the platform health-checks is OURS to serve — nothing serves it for us.
|
|
* ``POST /invocations`` wires the payload's whitelisted fields into ``run_project`` and returns
|
|
the outbox-shaped outcome payload. ``profile`` defaults to ``"azure"`` on THIS surface (a
|
|
hosted container has no local endpoint; run_project's own default stays LOCAL).
|
|
* Validation, NEVER repair: an unknown field is a 400 naming the field — the permissive-schema
|
|
trap (valg-doc §0: "en fil som ser konfigurert ut og ikke er det") applied to our own surface.
|
|
* Honest error mapping: ``ValueError`` (incl. pydantic contract violations) → 400; run failures
|
|
→ 500 ``{error_type, error}`` (mirrors ``RunFailure``); a ``Rejection`` is a SUCCESSFUL run →
|
|
200 — the negative outcome belongs to the payload, never to the transport.
|
|
* The server is asyncio on the ONE loop (NG1: ``test_no_thread_or_process_path_exists_under_src``
|
|
ratchets src/ thread-free) — these tests run client and server as coroutines on the SAME loop,
|
|
which only works because nothing in the server blocks it.
|
|
* Root ``main.py`` is the ONE process entry (``python main.py``, the command DEPLOY.md prints): the
|
|
subprocess test is the ONLY test that catches a detached shim or a detached SIGTERM handler
|
|
(P4-presedensen: entry-point-mutasjoner fanges aldri av in-process-tester).
|
|
|
|
Fase 4e closed two gaps the above leaves open, and both are about things a stand-in cannot see:
|
|
|
|
* **The whitelist composes with the REAL ``run_project``.** Every test above hands ``invoke`` a
|
|
stand-in that swallows ``**kwargs``, so the whitelist could name a field ``run_project`` does not
|
|
take — or hand it the same argument twice — and every one of them would stay green while a live
|
|
container answered 500. The end-to-end test drives a whole run (bundle navigation → debate →
|
|
deterministic validator → checker gate → verdict) against a SCRIPTED backend. The seam is
|
|
``run._default_factory``, not the payload: ``client_factory`` is refused by the whitelist ON
|
|
PURPOSE (a caller must never choose the server's model client), so patching the factory the run
|
|
falls back to is the only injection point this surface leaves — the same argument
|
|
``test_run_cli_loadbearing`` makes for ``main()``.
|
|
* **The deployment artifacts were raw-text-gated** — until 14.08, when the operator directive
|
|
after an external trial made the delivery runnable Python and the two artifacts were removed
|
|
from the tree. The gate is deleted, not weakened; see the note where it stood, below the
|
|
end-to-end test. The start command now has exactly one copy left, in DEPLOY.md, and
|
|
``tests/test_handover_package_loadbearing.py`` is what keeps it there.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import asyncio
|
|
import json
|
|
import os
|
|
import signal
|
|
import socket
|
|
import subprocess
|
|
import sys
|
|
import time
|
|
import urllib.error
|
|
import urllib.request
|
|
from pathlib import Path
|
|
from typing import Any
|
|
|
|
import pytest
|
|
|
|
from portfolio_optimiser import hosting
|
|
from portfolio_optimiser import run as run_module
|
|
from portfolio_optimiser.budget import BudgetExceeded
|
|
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
|
from portfolio_optimiser.provenance import Citation, ProvenanceStamp
|
|
from portfolio_optimiser.retrieval import TextSpan
|
|
from portfolio_optimiser.run import RunResult
|
|
from portfolio_optimiser.simulation import scripted_factory
|
|
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
|
from portfolio_optimiser.verdicts import ProposalFeatures, Verdict, VerdictStore
|
|
|
|
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
|
_BUNDLE_DIR = _REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro"
|
|
_BUNDLE_PID = "BYGG-KONTOR-NORD"
|
|
|
|
# The proposer's scripted reply for the end-to-end run. Its cost line IS the bundle's own code, and
|
|
# the claim (30 000) sits under the degenerate Monte Carlo P90 of 90 000 that the DETERMINISTIC
|
|
# validator computes from it — so the validated outcome is the validator's arithmetic, not a
|
|
# scripted string. bygg-energi-mikro ships no ``cost-baseline.json``, so the S4.0 stage-0 anchoring
|
|
# is legitimately absent here (a commons-owned golden predates the amendment).
|
|
_BUNDLE_PROPOSER_REPLY = (
|
|
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
|
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
|
)
|
|
_BUNDLE_P90 = 90000.0
|
|
|
|
_PROPOSAL = SavingsProposal(
|
|
project_id="P1",
|
|
measure="LED-retrofit av kontorbelysning",
|
|
affected_items=[AffectedItem(code="ENERGI-TOTAL-EL", quantity=1000.0, unit_cost=1.0)],
|
|
claimed_saving_nok=200.0,
|
|
assumptions={"ENERGI-TOTAL-EL": (0.8, 1.2)},
|
|
)
|
|
_PROVENANCE = ProvenanceStamp(
|
|
citations=[
|
|
Citation(file="f.md", locator=TextSpan(start_index=0, end_index=5), snippet="hello")
|
|
],
|
|
model="synthetic",
|
|
role="proposer",
|
|
validator_decision="validated",
|
|
token_usage=8,
|
|
# These fixtures stand in for an ordinary complete run; the road path is anchored by
|
|
# construction, so ``True`` is the honest value here. The un-anchored case has its own file.
|
|
cost_baseline_anchored=True,
|
|
)
|
|
_VALIDATED = ValidatedProposal(
|
|
proposal=_PROPOSAL, p10=100.0, p50=150.0, p90=200.0, nominal_feasible=180.0
|
|
)
|
|
_REJECTION = Rejection(proposal=_PROPOSAL, reason="stage 0: unknown cost code")
|
|
_VERDICT = Verdict(
|
|
id="vid-hosted",
|
|
proposal_features=ProposalFeatures(
|
|
affected_codes=frozenset({"ENERGI-TOTAL-EL"}),
|
|
measure_type="LED-retrofit av kontorbelysning",
|
|
claimed_saving_nok=200.0,
|
|
),
|
|
decision="approved",
|
|
rationale="expert reviewed",
|
|
)
|
|
|
|
_PAYLOAD = {
|
|
"project_id": "P1",
|
|
"docs_dir": "docs",
|
|
"verdict_input": {"decision": "approved", "rationale": "expert"},
|
|
}
|
|
|
|
|
|
def _result(outcome: ValidatedProposal | Rejection) -> RunResult:
|
|
return RunResult(
|
|
outcome=outcome,
|
|
provenance=_PROVENANCE,
|
|
verdict=_VERDICT,
|
|
retrieved=[],
|
|
store=VerdictStore(verdicts=[]),
|
|
debate_output="debate",
|
|
checker_verdict="approve",
|
|
)
|
|
|
|
|
|
class _Recorder:
|
|
"""An async stand-in for ``run_project``: records every (args, kwargs), then returns the
|
|
configured RunResult or raises the configured error."""
|
|
|
|
def __init__(
|
|
self,
|
|
result: RunResult | None = None,
|
|
error: Exception | None = None,
|
|
) -> None:
|
|
self.calls: list[tuple[tuple[Any, ...], dict[str, Any]]] = []
|
|
self._run_result = result
|
|
self._error = error
|
|
|
|
async def __call__(self, *args: Any, **kwargs: Any) -> RunResult:
|
|
self.calls.append((args, kwargs))
|
|
if self._error is not None:
|
|
raise self._error
|
|
assert self._run_result is not None
|
|
return self._run_result
|
|
|
|
|
|
@pytest.fixture()
|
|
async def served() -> Any:
|
|
"""The contract server on THIS test's loop (port 0 → ephemeral). Client and server share
|
|
the loop, so a server that blocked it would hang these tests — the fixture is itself a
|
|
check that nothing in the request path blocks."""
|
|
server = await hosting.start_server("127.0.0.1", 0)
|
|
port = server.sockets[0].getsockname()[1]
|
|
yield f"127.0.0.1:{port}"
|
|
server.close()
|
|
await server.wait_closed()
|
|
|
|
|
|
async def _request(
|
|
base: str, method: str, path: str, body: bytes | None = None
|
|
) -> tuple[int, bytes]:
|
|
"""A minimal HTTP/1.1 client on the same loop (urllib would block the shared loop)."""
|
|
host, port_text = base.split(":")
|
|
reader, writer = await asyncio.open_connection(host, int(port_text))
|
|
head = f"{method} {path} HTTP/1.1\r\nHost: {base}\r\nConnection: close\r\n"
|
|
if body is not None:
|
|
head += f"Content-Type: application/json\r\nContent-Length: {len(body)}\r\n"
|
|
writer.write(head.encode("latin-1") + b"\r\n" + (body or b""))
|
|
await writer.drain()
|
|
raw = await reader.read()
|
|
writer.close()
|
|
await writer.wait_closed()
|
|
status = int(raw.split(b" ", 2)[1])
|
|
return status, raw.split(b"\r\n\r\n", 1)[1]
|
|
|
|
|
|
async def _get(base: str, path: str) -> tuple[int, bytes]:
|
|
return await _request(base, "GET", path)
|
|
|
|
|
|
async def _post(base: str, path: str, payload: Any) -> tuple[int, dict[str, Any]]:
|
|
body = payload if isinstance(payload, bytes) else json.dumps(payload).encode("utf-8")
|
|
status, raw = await _request(base, "POST", path, body)
|
|
return status, json.loads(raw.decode("utf-8"))
|
|
|
|
|
|
async def test_readiness_returns_200(served: str) -> None:
|
|
"""The platform health-checks GET /readiness; with no protocol library, serving it is ours."""
|
|
status, _body = await _get(served, "/readiness")
|
|
assert status == 200
|
|
|
|
|
|
async def test_unknown_paths_are_404(served: str) -> None:
|
|
status, _ = await _get(served, "/other")
|
|
assert status == 404
|
|
status, _ = await _post(served, "/other", _PAYLOAD)
|
|
assert status == 404
|
|
|
|
|
|
async def test_invocations_wires_payload_into_run_project(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""The wiring seam: the payload's fields reach run_project as its own arguments, the hosted
|
|
default profile is 'azure', and the response carries the outbox-shaped outcome + proposal +
|
|
provenance. RED if the handler detaches the runner or drops the field mapping."""
|
|
recorder = _Recorder(result=_result(_VALIDATED))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
status, body = await _post(served, "/invocations", {**_PAYLOAD, "max_rounds": 2})
|
|
|
|
assert status == 200
|
|
assert recorder.calls, "run_project was never called"
|
|
args, kwargs = recorder.calls[0]
|
|
assert args == ("P1",)
|
|
assert kwargs["docs_dir"] == "docs"
|
|
assert kwargs["verdict_input"] == {"decision": "approved", "rationale": "expert"}
|
|
assert kwargs["max_rounds"] == 2
|
|
# Hosted default: a container has no local OpenAI-compatible endpoint (run_project's own
|
|
# default stays LOCAL — the two defaults are different on purpose, and this pins OURS).
|
|
assert kwargs["profile"] == "azure"
|
|
assert body["outcome_type"] == "validated"
|
|
assert body["p90"] == 200.0
|
|
assert body["checker_verdict"] == "approve"
|
|
assert body["verdict_id"] == "vid-hosted"
|
|
assert body["proposal"]["measure"] == "LED-retrofit av kontorbelysning"
|
|
assert body["provenance"]["model"] == "synthetic"
|
|
|
|
|
|
async def test_profile_in_payload_overrides_hosted_default(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
recorder = _Recorder(result=_result(_VALIDATED))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
status, _body = await _post(served, "/invocations", {**_PAYLOAD, "profile": "local"})
|
|
|
|
assert status == 200
|
|
_args, kwargs = recorder.calls[0]
|
|
assert kwargs["profile"] == "local"
|
|
|
|
|
|
async def test_unknown_field_is_refused_never_repaired(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Validation, never repair: an unknown field is a 400 naming the field and the runner is
|
|
never reached — dropping it silently would be the permissive-schema failure mode (valg §0).
|
|
The control arm (same payload minus the key) proves the refusal is FOR the key."""
|
|
recorder = _Recorder(result=_result(_VALIDATED))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
status, body = await _post(served, "/invocations", {**_PAYLOAD, "outbox_dir": "/x"})
|
|
assert status == 400
|
|
assert "outbox_dir" in body["error"]
|
|
assert recorder.calls == [] # refused BEFORE the runner — never repaired-and-run
|
|
|
|
control_status, _ = await _post(served, "/invocations", _PAYLOAD)
|
|
assert control_status == 200
|
|
assert len(recorder.calls) == 1
|
|
|
|
|
|
@pytest.mark.parametrize("missing", ["project_id", "docs_dir", "verdict_input"])
|
|
async def test_missing_required_field_is_400(
|
|
served: str, monkeypatch: pytest.MonkeyPatch, missing: str
|
|
) -> None:
|
|
recorder = _Recorder(result=_result(_VALIDATED))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
payload = {k: v for k, v in _PAYLOAD.items() if k != missing}
|
|
status, body = await _post(served, "/invocations", payload)
|
|
|
|
assert status == 400
|
|
assert missing in body["error"]
|
|
assert recorder.calls == []
|
|
|
|
|
|
async def test_non_object_or_invalid_json_body_is_400(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
recorder = _Recorder(result=_result(_VALIDATED))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
status, _ = await _post(served, "/invocations", b"[1, 2]")
|
|
assert status == 400
|
|
status, _ = await _post(served, "/invocations", b"not json")
|
|
assert status == 400
|
|
assert recorder.calls == []
|
|
|
|
|
|
async def test_rejected_outcome_is_200_with_reason(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""A Rejection is a successful run with a negative outcome — 200, outcome_type=rejected, the
|
|
reason, and NO percentile keys (key-absence on the parsed dict, not a substring)."""
|
|
recorder = _Recorder(result=_result(_REJECTION))
|
|
monkeypatch.setattr(hosting, "run_project", recorder)
|
|
|
|
status, body = await _post(served, "/invocations", _PAYLOAD)
|
|
|
|
assert status == 200
|
|
assert body["outcome_type"] == "rejected"
|
|
assert body["reason"] == "stage 0: unknown cost code"
|
|
assert "p90" not in body
|
|
|
|
|
|
async def test_contract_violation_is_400_and_run_failure_is_500(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""ValueError (pydantic contract violations subclass it) is the CALLER's error → 400; any
|
|
other failure is an honest 500 carrying {error_type, error} (mirrors RunFailure's shape).
|
|
|
|
The 500 witness is a NON-budget RuntimeError on purpose. It used to be ``BudgetExceeded``,
|
|
which is what made this test the one that pinned exhaustion to the crash channel; the two
|
|
now have separate arms, and this half is what keeps the budget arm NARROW — RED if it is
|
|
widened to catch ``Exception`` and route every failure to 429."""
|
|
monkeypatch.setattr(
|
|
hosting, "run_project", _Recorder(error=ValueError("docs_dir does not exist"))
|
|
)
|
|
status, body = await _post(served, "/invocations", _PAYLOAD)
|
|
assert status == 400
|
|
assert "docs_dir does not exist" in body["error"]
|
|
|
|
monkeypatch.setattr(
|
|
hosting, "run_project", _Recorder(error=RuntimeError("chat client fell over"))
|
|
)
|
|
status, body = await _post(served, "/invocations", _PAYLOAD)
|
|
assert status == 500
|
|
assert body["error_type"] == "RuntimeError"
|
|
assert "chat client fell over" in body["error"]
|
|
assert "budget_exhausted" not in body
|
|
|
|
|
|
async def test_budget_exhaustion_is_not_the_failure_channel(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Budget exhaustion is a DESIGNED terminal state — the cap firing IS the feature working
|
|
(``Budget``: fail-fast, never an unbounded loop) — so it must not share a channel with a
|
|
crash. The first live run died exactly here (``rounds limit=12 observed=13``) and the hosted
|
|
surface answered 500, i.e. the same thing it says when the model endpoint falls over.
|
|
|
|
* NOT 500: nothing broke.
|
|
* NOT 200: unlike a ``Rejection`` — which is a run that CONCLUDED, and therefore belongs in
|
|
the payload — an exhausted budget produced no proposal at all. A 2xx would let an automated
|
|
caller record "analysed" for a run that analysed nothing.
|
|
* 429: the condition arises from an ALLOWANCE (``max_rounds``/``max_tokens`` are whitelisted
|
|
request fields, and the raise is the caller's own remedy), never from a server fault.
|
|
* ``error_type`` is ABSENT: that key belongs to the failure channel, and a caller switching
|
|
on its presence must not find it on a run that did not fail.
|
|
|
|
RED when the arm is detached (falls through to 500) or relabelled to any other status."""
|
|
monkeypatch.setattr(hosting, "run_project", _Recorder(error=BudgetExceeded("rounds", 12, 13)))
|
|
|
|
status, body = await _post(served, "/invocations", _PAYLOAD)
|
|
|
|
assert status == 429
|
|
assert "error_type" not in body
|
|
|
|
|
|
async def test_budget_stop_triple_survives_as_structure(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""kø-(y): ``kind``/``limit``/``observed`` describe ONE ledger and are ONE structured stop
|
|
event. ``str(exc)`` flattens them into prose the caller has to re-parse to learn WHICH cap
|
|
bound and how far past it the run got — which is the whole operational question (raise
|
|
``max_rounds``? raise ``max_tokens``? give up?).
|
|
|
|
Built with ``observed != limit`` deliberately: at an exactly-exhausted cap the two coincide,
|
|
and a test written there cannot tell a faithful implementation from one that echoes the limit
|
|
back as the observed value. RED when the payload carries only the message string."""
|
|
monkeypatch.setattr(hosting, "run_project", _Recorder(error=BudgetExceeded("rounds", 12, 13)))
|
|
|
|
status, body = await _post(served, "/invocations", _PAYLOAD)
|
|
|
|
assert status == 429
|
|
assert body["budget_exhausted"] == {"kind": "rounds", "limit": 12, "observed": 13}
|
|
# The human-readable line stays alongside the structure — an operator reading a log needs it.
|
|
assert body["error"] == "budget exceeded: rounds limit=12 observed=13"
|
|
|
|
|
|
async def test_readiness_answers_while_an_invocation_is_in_flight(
|
|
served: str, monkeypatch: pytest.MonkeyPatch
|
|
) -> None:
|
|
"""Liveness on ONE loop: while an invocation awaits (model I/O), /readiness must still
|
|
answer — the platform health-checks during long runs, and a server that serialized the
|
|
whole process on one in-flight request would be killed as unready."""
|
|
release = asyncio.Event()
|
|
|
|
class _Blocking(_Recorder):
|
|
async def __call__(self, *args: Any, **kwargs: Any) -> RunResult:
|
|
await release.wait()
|
|
return await super().__call__(*args, **kwargs)
|
|
|
|
monkeypatch.setattr(hosting, "run_project", _Blocking(result=_result(_VALIDATED)))
|
|
in_flight = asyncio.ensure_future(_post(served, "/invocations", _PAYLOAD))
|
|
try:
|
|
status, _ = await asyncio.wait_for(_get(served, "/readiness"), timeout=10)
|
|
assert status == 200
|
|
finally:
|
|
release.set()
|
|
status, _body = await asyncio.wait_for(in_flight, timeout=10)
|
|
assert status == 200
|
|
|
|
|
|
def test_port_resolution_is_truthiness_not_presence(monkeypatch: pytest.MonkeyPatch) -> None:
|
|
"""PORT on truthiness (the 4b rule): an exported-empty PORT is a shell accident, not a bind
|
|
instruction. Default 8088 is the hosted-agent contract's port."""
|
|
monkeypatch.delenv("PORT", raising=False)
|
|
assert hosting.resolve_port() == 8088
|
|
monkeypatch.setenv("PORT", "9001")
|
|
assert hosting.resolve_port() == 9001
|
|
monkeypatch.setenv("PORT", "")
|
|
assert hosting.resolve_port() == 8088
|
|
|
|
|
|
def _blocking_get(url: str) -> tuple[int, str]:
|
|
try:
|
|
with urllib.request.urlopen(url, timeout=10) as resp:
|
|
return resp.status, resp.read().decode("utf-8")
|
|
except urllib.error.HTTPError as err:
|
|
return err.code, err.read().decode("utf-8")
|
|
|
|
|
|
def test_main_entrypoint_serves_and_stops_on_sigterm() -> None:
|
|
"""Root main.py is the ONE process entry (``python main.py``, the command DEPLOY.md prints):
|
|
started as a subprocess it must serve /readiness and exit 0 on SIGTERM. This is the only test that
|
|
catches a shim that stops calling hosting.main() or a detached SIGTERM handler."""
|
|
with socket.socket() as probe:
|
|
probe.bind(("127.0.0.1", 0))
|
|
port = probe.getsockname()[1]
|
|
repo_root = Path(__file__).resolve().parents[1]
|
|
proc = subprocess.Popen(
|
|
[sys.executable, str(repo_root / "main.py")],
|
|
env={**os.environ, "PORT": str(port)},
|
|
cwd=repo_root,
|
|
stdout=subprocess.DEVNULL,
|
|
stderr=subprocess.DEVNULL,
|
|
)
|
|
try:
|
|
deadline = time.monotonic() + 60
|
|
up = False
|
|
while time.monotonic() < deadline:
|
|
try:
|
|
status, _ = _blocking_get(f"http://127.0.0.1:{port}/readiness")
|
|
if status == 200:
|
|
up = True
|
|
break
|
|
except (urllib.error.URLError, OSError):
|
|
time.sleep(0.2)
|
|
assert up, "main.py never served /readiness"
|
|
proc.send_signal(signal.SIGTERM)
|
|
assert proc.wait(timeout=15) == 0
|
|
finally:
|
|
if proc.poll() is None:
|
|
proc.kill()
|
|
proc.wait()
|
|
|
|
|
|
# --- Fase 4e: the whitelist against the REAL run_project -----------------------------------------
|
|
|
|
|
|
@pytest.fixture()
|
|
def _scripted_backend(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
|
"""Inject a network-free scripted backend into the seam a hosted invocation ACTUALLY resolves
|
|
through.
|
|
|
|
``client_factory`` is refused by the invocations whitelist on purpose — the caller of a hosted
|
|
agent must never choose the server's model client — so an end-to-end test cannot inject through
|
|
the payload. ``run_project`` falls back to the module-level ``_default_factory`` when no factory
|
|
is passed, which makes patching it the only injection point this surface leaves (the argument
|
|
``test_run_cli_loadbearing`` makes for ``main()``, one layer up).
|
|
|
|
Returns the shared prompt sink: non-empty is the proof that the run went through the real
|
|
machinery rather than short-circuiting somewhere before the debate."""
|
|
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
|
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
|
sink: list[str] = []
|
|
monkeypatch.setattr(
|
|
run_module,
|
|
"_default_factory",
|
|
lambda _profile: scripted_factory(
|
|
{"proposer": _BUNDLE_PROPOSER_REPLY, "checker": "VERDICT: APPROVE"}, sink
|
|
),
|
|
)
|
|
return sink
|
|
|
|
|
|
async def test_invocations_answers_through_the_real_run_project(
|
|
served: str, _scripted_backend: list[str]
|
|
) -> None:
|
|
"""END-TO-END: a bundle payload posted to ``/invocations`` drives a WHOLE real run and comes
|
|
back as a valid response — no stand-in for ``run_project`` anywhere in the path.
|
|
|
|
Every other invocations test hands ``invoke`` a recorder that swallows ``**kwargs``, so none of
|
|
them can see whether the whitelist actually composes with ``run_project``'s signature. Two ways
|
|
to break that are invisible to a recorder and fatal in a container: naming a field
|
|
``run_project`` does not take, and passing it an argument twice. Both surface here as a 500.
|
|
|
|
The payload names every FORWARDED field, and the coverage assertion below is what keeps that
|
|
true: a field added to the whitelist later cannot slip past this test unexercised.
|
|
|
|
The whitelist became a THREE-way partition when U4's ``explore_prompt`` arrived: those fields
|
|
are accepted by the surface and CONSUMED by it (the exploration runs first and hands
|
|
``run_project`` a mandate), so they can never satisfy a "reaches ``run_project``" assertion.
|
|
The consumed half has its own, negative proof in
|
|
``tests/test_explore_callsites_loadbearing.py`` — without it a field sliding from consumed to
|
|
forwarded is exactly the drift 4e exists to catch."""
|
|
payload = {
|
|
"project_id": _BUNDLE_PID,
|
|
"docs_dir": str(_BUNDLE_DIR),
|
|
"bundle_dir": str(_BUNDLE_DIR),
|
|
"verdict_input": {"decision": "approved", "rationale": "expert reviewed (4e)"},
|
|
# LOCAL, not the hosted default: the AZURE arm resolves a Foundry deployment name from the
|
|
# model map BEFORE any client is built (``run.py`` stamps provenance with it), so it cannot
|
|
# complete offline. The hosted default itself is pinned by the recorder test above.
|
|
"profile": "local",
|
|
"max_rounds": 2,
|
|
"max_tokens": 100_000,
|
|
"top_k": 3,
|
|
}
|
|
assert set(payload) == set(hosting._REQUIRED_FIELDS) | set(hosting._OPTIONAL_FIELDS), (
|
|
"the payload must exercise every FORWARDED field — a field the whitelist forwards but "
|
|
"this test never sends is a field no test proves ``run_project`` accepts"
|
|
)
|
|
assert hosting._ALLOWED_FIELDS == set(payload) | set(hosting._CONSUMED_FIELDS), (
|
|
"the whitelist is a three-way partition and this arm covers the forwarded half; a field "
|
|
"in neither half would be accepted by the surface with nothing proving what it does"
|
|
)
|
|
|
|
status, body = await _post(served, "/invocations", payload)
|
|
|
|
assert status == 200, f"a real run did not survive the hosted surface: {body}"
|
|
assert body["outcome_type"] == "validated"
|
|
# The validator's arithmetic, not a scripted string: P90 is computed from the proposal's own
|
|
# cost line, and the claim (30 000) clears it.
|
|
assert body["p90"] == _BUNDLE_P90
|
|
assert body["proposal"]["claimed_saving_nok"] == 30000.0
|
|
assert body["proposal"]["project_id"] == _BUNDLE_PID
|
|
# Provenance carries a first-class citation from the NAVIGATED bundle — the run really read the
|
|
# knowledge base rather than answering from the payload.
|
|
assert body["provenance"]["citations"], "the run produced no citation — the bundle was not read"
|
|
assert body["checker_verdict"] == "approve"
|
|
assert _scripted_backend, "the scripted backend was never called — no real run happened"
|
|
|
|
|
|
# The 4e raw-text gate on ``Dockerfile``/``azure.yaml`` lived here until 14.08. It pinned
|
|
# ``--platform linux/amd64`` and the one-copy rule for the image's ``CMD``. Both files were removed
|
|
# from the tree that day (operator directive: the delivery is runnable Python), and a gate that pins
|
|
# a surface we no longer ship is deleted with it rather than weakened into something that can only
|
|
# be green. What replaced it lives in ``tests/test_handover_package_loadbearing.py``: the package
|
|
# must carry NO container/azd wrapper and must document the Python start command.
|