test(4e): whitelisten mot EKTE run_project + rå-tekst-gate på deploy-artefaktene

De to gjenstående 4e-radene, begge målt mot hele suiten (837 passed / 4 skipped).

(1) /invocations svarer gyldig mot en SKRIPTET backend gjennom EKTE run_project.
Alle 4d-testene ga invoke en stand-in som sluker **kwargs, så whitelisten kunne
navngi et felt run_project ikke tar — eller sende samme argument to ganger — uten
at én test merket det, mens en levende container svarte 500. Sømmen er
run._default_factory, ikke payloaden: client_factory nektes av whitelisten med
vilje, så factory-defaulten er eneste injeksjonspunkt flaten etterlater.
Payloaden sender HVERT whitelistet felt, med en dekningsassert mot
_ALLOWED_FIELDS. Profilen er LOCAL fordi AZURE-armen slår opp et
Foundry-deployment-navn i modell-mappet FØR noen klient bygges (målt).

(2) Rå-tekst-gate: Dockerfile + azure.yaml kjøres av ingen test (docker build og
azd deploy er operatør-gatet). Gaten pinner --platform linux/amd64 (målt påkrevd)
og ÉN kopi av startkommandoen (imagets CMD; azure.yaml har ingen startupCommand).
Nøkkel-sjekkene er linjeforankret, ikke delstreng — azure.yaml sin egen kommentar
navngir begge nøklene for å begrunne fraværet.

Fem mutasjoner, alle røde på riktig test og på INGEN annen (836 øvrige grønne
hver gang): send project_id to ganger · whitelist et felt run_project ikke tar ·
fjern bundle_dir fra whitelisten · fjern --platform linux/amd64 · gi azure.yaml en
startupCommand-nøkkel. Kontroll: pristine tre 837/4.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018GfbDLY7YVLKVqpUHnbwVW
This commit is contained in:
Kjell Tore Guttormsen 2026-08-14 00:03:20 +02:00
commit 18dbd7d4d6
2 changed files with 185 additions and 0 deletions

View file

@ -191,6 +191,28 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
siste fanges KUN av subprosess-testen — P4-presedensen). Deploy er IKKE utført (azd-steget er
operatørens); chunked request-bodies støttes ikke, og under CPU-bundne strekk (CBC-solven)
står readiness — uttalt, ikke skjult.
- **Whitelisten må komponere med den EKTE `run_project`, og artefaktene gates som RÅ TEKST
(Fase 4e):** alle 4d-testene ga `invoke` en stand-in som sluker `**kwargs`, så whitelisten kunne
navngi et felt `run_project` ikke tar — eller sende samme argument to ganger — uten at én test
merket det, mens en levende container svarte 500. **Sømmen er `run._default_factory`, ikke
payloaden:** `client_factory` NEKTES av whitelisten med vilje (en kaller av en hostet agent skal
aldri velge serverens modellklient), så å patche factory-defaulten er eneste injeksjonspunkt
flaten etterlater (samme argument `test_run_cli_loadbearing` gjør for `main()`). Testen sender
HVERT whitelistet felt og asserterer dekningen mot `_ALLOWED_FIELDS`, så et felt lagt til senere
ikke kan gli forbi uøvet. Profilen er **LOCAL, ikke den hostede defaulten**: AZURE-armen slår opp
et Foundry-deployment-navn i modell-mappet FØR noen klient bygges (`run.py` stempler provenance
med det), så den kan ikke fullføre offline — **containeren trenger altså `PORTFOLIO_MODEL_MAP`
eller et utfylt `data/model_map.json`, ikke bare et endepunkt** (målt her, ikke antatt).
`Dockerfile`/`azure.yaml` KJØRES av ingen test (`docker build`/`azd deploy` er operatør-gatet), så
rå-tekst er eneste tilgjengelige gate: `--platform linux/amd64` (målt påkrevd, spike §1.4 — uten
det arver imaget byggerens arkitektur og bygger grønt lokalt mens det ikke kan starte i skyen) +
ÉN kopi av startkommandoen (imagets `CMD` navngir `main.py`, `azure.yaml` har ingen
`startupCommand`). **Nøkkel-sjekkene er LINJEFORANKRET, ikke delstreng:** `azure.yaml`s egen
kommentar NAVNGIR `startupCommand` og `env` for å begrunne fraværet, så en substring-gate ville
vært rød på prosaen den beskytter. Load-bearing MÅLT (`tests/test_hosting_loadbearing.py`), fem
mutasjoner alle røde på riktig test og på INGEN annen (836 øvrige grønne hver gang): send
`project_id` to ganger · whitelist et felt `run_project` ikke tar · fjern `bundle_dir` fra
whitelisten · fjern `--platform linux/amd64` · gi `azure.yaml` en `startupCommand`-nøkkel.
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (fail-fast, aldri ubegrenset loop).
- **Group Chat maker-checker** som debatt-default (IKKE Magentic, som er eksperimentell).
- **To falsifiserere, samme kandidat (Steg 3/4, målbilde §2/§6):** den deterministiske validatoren

View file

@ -21,6 +21,26 @@ Load-bearing surface pinned here:
* Root ``main.py`` is the ONE process entry (Dockerfile CMD + azure.yaml point at it): the
subprocess test is the ONLY test that catches a detached shim or a detached SIGTERM handler
(P4-presedensen: entry-point-mutasjoner fanges aldri av in-process-tester).
Fase 4e closes two gaps the above leaves open, and both are about things a stand-in cannot see:
* **The whitelist composes with the REAL ``run_project``.** Every test above hands ``invoke`` a
stand-in that swallows ``**kwargs``, so the whitelist could name a field ``run_project`` does not
take or hand it the same argument twice and every one of them would stay green while a live
container answered 500. The end-to-end test drives a whole run (bundle navigation debate
deterministic validator checker gate verdict) against a SCRIPTED backend. The seam is
``run._default_factory``, not the payload: ``client_factory`` is refused by the whitelist ON
PURPOSE (a caller must never choose the server's model client), so patching the factory the run
falls back to is the only injection point this surface leaves the same argument
``test_run_cli_loadbearing`` makes for ``main()``.
* **The deployment artifacts are raw-text-gated.** ``Dockerfile`` and ``azure.yaml`` are the two
files that decide whether the image the platform pulls can run at all, and NO test executes
them here (``docker build``/``azd deploy`` are operator-gated). A raw-text gate is therefore the
only mechanism available: it pins ``--platform linux/amd64`` (measured required spike §1.4;
an arm64 image built on this Intel-free-of-charge assumption would fail only in the cloud) and
the ONE-copy rule for the start command (the image's ``CMD``; ``azure.yaml`` carries no
``startupCommand`` to drift from it). Guard-tester leser kildefiler som TEKST reformulate
the prose around them, never the strings they pin.
"""
from __future__ import annotations
@ -41,14 +61,31 @@ from typing import Any
import pytest
from portfolio_optimiser import hosting
from portfolio_optimiser import run as run_module
from portfolio_optimiser.budget import BudgetExceeded
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
from portfolio_optimiser.provenance import Citation, ProvenanceStamp
from portfolio_optimiser.retrieval import TextSpan
from portfolio_optimiser.run import RunResult
from portfolio_optimiser.simulation import scripted_factory
from portfolio_optimiser.validator import Rejection, ValidatedProposal
from portfolio_optimiser.verdicts import ProposalFeatures, Verdict, VerdictStore
_REPO_ROOT = Path(__file__).resolve().parents[1]
_BUNDLE_DIR = _REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro"
_BUNDLE_PID = "BYGG-KONTOR-NORD"
# The proposer's scripted reply for the end-to-end run. Its cost line IS the bundle's own code, and
# the claim (30 000) sits under the degenerate Monte Carlo P90 of 90 000 that the DETERMINISTIC
# validator computes from it — so the validated outcome is the validator's arithmetic, not a
# scripted string. bygg-energi-mikro ships no ``cost-baseline.json``, so the S4.0 stage-0 anchoring
# is legitimately absent here (a commons-owned golden predates the amendment).
_BUNDLE_PROPOSER_REPLY = (
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
)
_BUNDLE_P90 = 90000.0
_PROPOSAL = SavingsProposal(
project_id="P1",
measure="LED-retrofit av kontorbelysning",
@ -376,3 +413,129 @@ def test_main_entrypoint_serves_and_stops_on_sigterm() -> None:
if proc.poll() is None:
proc.kill()
proc.wait()
# --- Fase 4e: the whitelist against the REAL run_project, and the deployment artifacts ----------
@pytest.fixture()
def _scripted_backend(monkeypatch: pytest.MonkeyPatch) -> list[str]:
"""Inject a network-free scripted backend into the seam a hosted invocation ACTUALLY resolves
through.
``client_factory`` is refused by the invocations whitelist on purpose the caller of a hosted
agent must never choose the server's model client — so an end-to-end test cannot inject through
the payload. ``run_project`` falls back to the module-level ``_default_factory`` when no factory
is passed, which makes patching it the only injection point this surface leaves (the argument
``test_run_cli_loadbearing`` makes for ``main()``, one layer up).
Returns the shared prompt sink: non-empty is the proof that the run went through the real
machinery rather than short-circuiting somewhere before the debate."""
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
sink: list[str] = []
monkeypatch.setattr(
run_module,
"_default_factory",
lambda _profile: scripted_factory(
{"proposer": _BUNDLE_PROPOSER_REPLY, "checker": "VERDICT: APPROVE"}, sink
),
)
return sink
async def test_invocations_answers_through_the_real_run_project(
served: str, _scripted_backend: list[str]
) -> None:
"""END-TO-END: a bundle payload posted to ``/invocations`` drives a WHOLE real run and comes
back as a valid response no stand-in for ``run_project`` anywhere in the path.
Every other invocations test hands ``invoke`` a recorder that swallows ``**kwargs``, so none of
them can see whether the whitelist actually composes with ``run_project``'s signature. Two ways
to break that are invisible to a recorder and fatal in a container: naming a field
``run_project`` does not take, and passing it an argument twice. Both surface here as a 500.
The payload names EVERY whitelisted field, and the coverage assertion below is what keeps that
true: a field added to the whitelist later cannot slip past this test unexercised."""
payload = {
"project_id": _BUNDLE_PID,
"docs_dir": str(_BUNDLE_DIR),
"bundle_dir": str(_BUNDLE_DIR),
"verdict_input": {"decision": "approved", "rationale": "expert reviewed (4e)"},
# LOCAL, not the hosted default: the AZURE arm resolves a Foundry deployment name from the
# model map BEFORE any client is built (``run.py`` stamps provenance with it), so it cannot
# complete offline. The hosted default itself is pinned by the recorder test above.
"profile": "local",
"max_rounds": 2,
"max_tokens": 100_000,
"top_k": 3,
}
assert set(payload) == hosting._ALLOWED_FIELDS, (
"the payload must exercise every whitelisted field — a field the whitelist accepts but "
"this test never sends is a field no test proves ``run_project`` accepts"
)
status, body = await _post(served, "/invocations", payload)
assert status == 200, f"a real run did not survive the hosted surface: {body}"
assert body["outcome_type"] == "validated"
# The validator's arithmetic, not a scripted string: P90 is computed from the proposal's own
# cost line, and the claim (30 000) clears it.
assert body["p90"] == _BUNDLE_P90
assert body["proposal"]["claimed_saving_nok"] == 30000.0
assert body["proposal"]["project_id"] == _BUNDLE_PID
# Provenance carries a first-class citation from the NAVIGATED bundle — the run really read the
# knowledge base rather than answering from the payload.
assert body["provenance"]["citations"], "the run produced no citation — the bundle was not read"
assert body["checker_verdict"] == "approve"
assert _scripted_backend, "the scripted backend was never called — no real run happened"
def test_deployment_artifacts_pin_the_measured_platform_and_one_start_command() -> None:
"""RAW-TEXT GATE on the two files that decide whether the hosted image runs at all.
Neither is executed by any test: ``docker build`` and ``azd deploy`` are operator-gated, so a
regression in them is invisible to the whole suite until it fails in the cloud. What the gate
pins is exactly what was MEASURED, and nothing about the prose around it:
* ``--platform linux/amd64`` the platform requires x86_64 (spike §1.4). Dropping it makes the
image inherit the builder's architecture, which on an arm64 laptop yields an image that
builds green locally and cannot start in the cloud. The flag lives in the documented build
command (a Dockerfile cannot set the build platform for its own invocation), so pinning the
documented string is the only gate available and it is worth having precisely because
nothing else re-derives it.
* ONE copy of the start command: the image's ``CMD`` names ``main.py``, and ``azure.yaml``
declares NO ``startupCommand``. Two copies drift (-(p)-regelen); this is the pair that
keeps there being one.
``env:`` is checked for the same reason: ``FOUNDRY_PROJECT_ENDPOINT`` is injected by the
platform, and redeclaring it here could shadow the injected value which is the failure mode
Fase 4b's endpoint precedence exists to avoid, undone from the config side."""
dockerfile = _REPO_ROOT / "Dockerfile"
azure_yaml = _REPO_ROOT / "azure.yaml"
assert dockerfile.is_file(), "the hosted image has no build definition"
assert azure_yaml.is_file(), "azd has no project definition to deploy"
docker_text = dockerfile.read_text(encoding="utf-8")
azure_text = azure_yaml.read_text(encoding="utf-8")
assert "--platform linux/amd64" in docker_text, (
"the Dockerfile no longer names the measured build platform; an image built without it "
"inherits the builder's architecture and cannot start on the hosting platform"
)
assert "CMD" in docker_text and "main.py" in docker_text, (
"the image's CMD is the ONE copy of the start command and must name main.py"
)
# Line-anchored: a mention inside a comment is prose, a top-level key is a declaration.
azure_keys = [line.split(":")[0].strip() for line in azure_text.splitlines()]
assert "startupCommand" not in azure_keys, (
"azure.yaml declares a startupCommand — a SECOND copy of the start command, free to drift "
"from the image's CMD"
)
assert "env" not in azure_keys, (
"azure.yaml declares an env block — FOUNDRY_PROJECT_ENDPOINT is injected by the platform "
"and must never be redeclared here"
)
assert "host: azure.ai.agent" in azure_text and "kind: hosted" in azure_text, (
"azure.yaml no longer declares the hosted-agent host this whole entrypoint targets"
)