test(4e): whitelisten mot EKTE run_project + rå-tekst-gate på deploy-artefaktene
De to gjenstående 4e-radene, begge målt mot hele suiten (837 passed / 4 skipped). (1) /invocations svarer gyldig mot en SKRIPTET backend gjennom EKTE run_project. Alle 4d-testene ga invoke en stand-in som sluker **kwargs, så whitelisten kunne navngi et felt run_project ikke tar — eller sende samme argument to ganger — uten at én test merket det, mens en levende container svarte 500. Sømmen er run._default_factory, ikke payloaden: client_factory nektes av whitelisten med vilje, så factory-defaulten er eneste injeksjonspunkt flaten etterlater. Payloaden sender HVERT whitelistet felt, med en dekningsassert mot _ALLOWED_FIELDS. Profilen er LOCAL fordi AZURE-armen slår opp et Foundry-deployment-navn i modell-mappet FØR noen klient bygges (målt). (2) Rå-tekst-gate: Dockerfile + azure.yaml kjøres av ingen test (docker build og azd deploy er operatør-gatet). Gaten pinner --platform linux/amd64 (målt påkrevd) og ÉN kopi av startkommandoen (imagets CMD; azure.yaml har ingen startupCommand). Nøkkel-sjekkene er linjeforankret, ikke delstreng — azure.yaml sin egen kommentar navngir begge nøklene for å begrunne fraværet. Fem mutasjoner, alle røde på riktig test og på INGEN annen (836 øvrige grønne hver gang): send project_id to ganger · whitelist et felt run_project ikke tar · fjern bundle_dir fra whitelisten · fjern --platform linux/amd64 · gi azure.yaml en startupCommand-nøkkel. Kontroll: pristine tre 837/4. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_018GfbDLY7YVLKVqpUHnbwVW
This commit is contained in:
parent
426ccb0ad6
commit
18dbd7d4d6
2 changed files with 185 additions and 0 deletions
22
CLAUDE.md
22
CLAUDE.md
|
|
@ -191,6 +191,28 @@ Python ≥3.10. MAF (`agent-framework-core` 1.9.0). Pakkehåndtering: `uv`. To b
|
|||
siste fanges KUN av subprosess-testen — P4-presedensen). Deploy er IKKE utført (azd-steget er
|
||||
operatørens); chunked request-bodies støttes ikke, og under CPU-bundne strekk (CBC-solven)
|
||||
står readiness — uttalt, ikke skjult.
|
||||
- **Whitelisten må komponere med den EKTE `run_project`, og artefaktene gates som RÅ TEKST
|
||||
(Fase 4e):** alle 4d-testene ga `invoke` en stand-in som sluker `**kwargs`, så whitelisten kunne
|
||||
navngi et felt `run_project` ikke tar — eller sende samme argument to ganger — uten at én test
|
||||
merket det, mens en levende container svarte 500. **Sømmen er `run._default_factory`, ikke
|
||||
payloaden:** `client_factory` NEKTES av whitelisten med vilje (en kaller av en hostet agent skal
|
||||
aldri velge serverens modellklient), så å patche factory-defaulten er eneste injeksjonspunkt
|
||||
flaten etterlater (samme argument `test_run_cli_loadbearing` gjør for `main()`). Testen sender
|
||||
HVERT whitelistet felt og asserterer dekningen mot `_ALLOWED_FIELDS`, så et felt lagt til senere
|
||||
ikke kan gli forbi uøvet. Profilen er **LOCAL, ikke den hostede defaulten**: AZURE-armen slår opp
|
||||
et Foundry-deployment-navn i modell-mappet FØR noen klient bygges (`run.py` stempler provenance
|
||||
med det), så den kan ikke fullføre offline — **containeren trenger altså `PORTFOLIO_MODEL_MAP`
|
||||
eller et utfylt `data/model_map.json`, ikke bare et endepunkt** (målt her, ikke antatt).
|
||||
`Dockerfile`/`azure.yaml` KJØRES av ingen test (`docker build`/`azd deploy` er operatør-gatet), så
|
||||
rå-tekst er eneste tilgjengelige gate: `--platform linux/amd64` (målt påkrevd, spike §1.4 — uten
|
||||
det arver imaget byggerens arkitektur og bygger grønt lokalt mens det ikke kan starte i skyen) +
|
||||
ÉN kopi av startkommandoen (imagets `CMD` navngir `main.py`, `azure.yaml` har ingen
|
||||
`startupCommand`). **Nøkkel-sjekkene er LINJEFORANKRET, ikke delstreng:** `azure.yaml`s egen
|
||||
kommentar NAVNGIR `startupCommand` og `env` for å begrunne fraværet, så en substring-gate ville
|
||||
vært rød på prosaen den beskytter. Load-bearing MÅLT (`tests/test_hosting_loadbearing.py`), fem
|
||||
mutasjoner alle røde på riktig test og på INGEN annen (836 øvrige grønne hver gang): send
|
||||
`project_id` to ganger · whitelist et felt `run_project` ikke tar · fjern `bundle_dir` fra
|
||||
whitelisten · fjern `--platform linux/amd64` · gi `azure.yaml` en `startupCommand`-nøkkel.
|
||||
- **Stoppkriterier + budsjett-tak påkrevd ved oppstart** (fail-fast, aldri ubegrenset loop).
|
||||
- **Group Chat maker-checker** som debatt-default (IKKE Magentic, som er eksperimentell).
|
||||
- **To falsifiserere, samme kandidat (Steg 3/4, målbilde §2/§6):** den deterministiske validatoren
|
||||
|
|
|
|||
|
|
@ -21,6 +21,26 @@ Load-bearing surface pinned here:
|
|||
* Root ``main.py`` is the ONE process entry (Dockerfile CMD + azure.yaml point at it): the
|
||||
subprocess test is the ONLY test that catches a detached shim or a detached SIGTERM handler
|
||||
(P4-presedensen: entry-point-mutasjoner fanges aldri av in-process-tester).
|
||||
|
||||
Fase 4e closes two gaps the above leaves open, and both are about things a stand-in cannot see:
|
||||
|
||||
* **The whitelist composes with the REAL ``run_project``.** Every test above hands ``invoke`` a
|
||||
stand-in that swallows ``**kwargs``, so the whitelist could name a field ``run_project`` does not
|
||||
take — or hand it the same argument twice — and every one of them would stay green while a live
|
||||
container answered 500. The end-to-end test drives a whole run (bundle navigation → debate →
|
||||
deterministic validator → checker gate → verdict) against a SCRIPTED backend. The seam is
|
||||
``run._default_factory``, not the payload: ``client_factory`` is refused by the whitelist ON
|
||||
PURPOSE (a caller must never choose the server's model client), so patching the factory the run
|
||||
falls back to is the only injection point this surface leaves — the same argument
|
||||
``test_run_cli_loadbearing`` makes for ``main()``.
|
||||
* **The deployment artifacts are raw-text-gated.** ``Dockerfile`` and ``azure.yaml`` are the two
|
||||
files that decide whether the image the platform pulls can run at all, and NO test executes
|
||||
them here (``docker build``/``azd deploy`` are operator-gated). A raw-text gate is therefore the
|
||||
only mechanism available: it pins ``--platform linux/amd64`` (measured required — spike §1.4;
|
||||
an arm64 image built on this Intel-free-of-charge assumption would fail only in the cloud) and
|
||||
the ONE-copy rule for the start command (the image's ``CMD``; ``azure.yaml`` carries no
|
||||
``startupCommand`` to drift from it). Guard-tester leser kildefiler som RÅ TEKST — reformulate
|
||||
the prose around them, never the strings they pin.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
|
@ -41,14 +61,31 @@ from typing import Any
|
|||
import pytest
|
||||
|
||||
from portfolio_optimiser import hosting
|
||||
from portfolio_optimiser import run as run_module
|
||||
from portfolio_optimiser.budget import BudgetExceeded
|
||||
from portfolio_optimiser.ir import AffectedItem, SavingsProposal
|
||||
from portfolio_optimiser.provenance import Citation, ProvenanceStamp
|
||||
from portfolio_optimiser.retrieval import TextSpan
|
||||
from portfolio_optimiser.run import RunResult
|
||||
from portfolio_optimiser.simulation import scripted_factory
|
||||
from portfolio_optimiser.validator import Rejection, ValidatedProposal
|
||||
from portfolio_optimiser.verdicts import ProposalFeatures, Verdict, VerdictStore
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||
_BUNDLE_DIR = _REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro"
|
||||
_BUNDLE_PID = "BYGG-KONTOR-NORD"
|
||||
|
||||
# The proposer's scripted reply for the end-to-end run. Its cost line IS the bundle's own code, and
|
||||
# the claim (30 000) sits under the degenerate Monte Carlo P90 of 90 000 that the DETERMINISTIC
|
||||
# validator computes from it — so the validated outcome is the validator's arithmetic, not a
|
||||
# scripted string. bygg-energi-mikro ships no ``cost-baseline.json``, so the S4.0 stage-0 anchoring
|
||||
# is legitimately absent here (a commons-owned golden predates the amendment).
|
||||
_BUNDLE_PROPOSER_REPLY = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
_BUNDLE_P90 = 90000.0
|
||||
|
||||
_PROPOSAL = SavingsProposal(
|
||||
project_id="P1",
|
||||
measure="LED-retrofit av kontorbelysning",
|
||||
|
|
@ -376,3 +413,129 @@ def test_main_entrypoint_serves_and_stops_on_sigterm() -> None:
|
|||
if proc.poll() is None:
|
||||
proc.kill()
|
||||
proc.wait()
|
||||
|
||||
|
||||
# --- Fase 4e: the whitelist against the REAL run_project, and the deployment artifacts ----------
|
||||
|
||||
|
||||
@pytest.fixture()
|
||||
def _scripted_backend(monkeypatch: pytest.MonkeyPatch) -> list[str]:
|
||||
"""Inject a network-free scripted backend into the seam a hosted invocation ACTUALLY resolves
|
||||
through.
|
||||
|
||||
``client_factory`` is refused by the invocations whitelist on purpose — the caller of a hosted
|
||||
agent must never choose the server's model client — so an end-to-end test cannot inject through
|
||||
the payload. ``run_project`` falls back to the module-level ``_default_factory`` when no factory
|
||||
is passed, which makes patching it the only injection point this surface leaves (the argument
|
||||
``test_run_cli_loadbearing`` makes for ``main()``, one layer up).
|
||||
|
||||
Returns the shared prompt sink: non-empty is the proof that the run went through the real
|
||||
machinery rather than short-circuiting somewhere before the debate."""
|
||||
monkeypatch.delenv("PORTFOLIO_MODEL_MAP", raising=False)
|
||||
monkeypatch.delenv("PORTFOLIO_FOUNDRY_PROJECT_ENDPOINT", raising=False)
|
||||
sink: list[str] = []
|
||||
monkeypatch.setattr(
|
||||
run_module,
|
||||
"_default_factory",
|
||||
lambda _profile: scripted_factory(
|
||||
{"proposer": _BUNDLE_PROPOSER_REPLY, "checker": "VERDICT: APPROVE"}, sink
|
||||
),
|
||||
)
|
||||
return sink
|
||||
|
||||
|
||||
async def test_invocations_answers_through_the_real_run_project(
|
||||
served: str, _scripted_backend: list[str]
|
||||
) -> None:
|
||||
"""END-TO-END: a bundle payload posted to ``/invocations`` drives a WHOLE real run and comes
|
||||
back as a valid response — no stand-in for ``run_project`` anywhere in the path.
|
||||
|
||||
Every other invocations test hands ``invoke`` a recorder that swallows ``**kwargs``, so none of
|
||||
them can see whether the whitelist actually composes with ``run_project``'s signature. Two ways
|
||||
to break that are invisible to a recorder and fatal in a container: naming a field
|
||||
``run_project`` does not take, and passing it an argument twice. Both surface here as a 500.
|
||||
|
||||
The payload names EVERY whitelisted field, and the coverage assertion below is what keeps that
|
||||
true: a field added to the whitelist later cannot slip past this test unexercised."""
|
||||
payload = {
|
||||
"project_id": _BUNDLE_PID,
|
||||
"docs_dir": str(_BUNDLE_DIR),
|
||||
"bundle_dir": str(_BUNDLE_DIR),
|
||||
"verdict_input": {"decision": "approved", "rationale": "expert reviewed (4e)"},
|
||||
# LOCAL, not the hosted default: the AZURE arm resolves a Foundry deployment name from the
|
||||
# model map BEFORE any client is built (``run.py`` stamps provenance with it), so it cannot
|
||||
# complete offline. The hosted default itself is pinned by the recorder test above.
|
||||
"profile": "local",
|
||||
"max_rounds": 2,
|
||||
"max_tokens": 100_000,
|
||||
"top_k": 3,
|
||||
}
|
||||
assert set(payload) == hosting._ALLOWED_FIELDS, (
|
||||
"the payload must exercise every whitelisted field — a field the whitelist accepts but "
|
||||
"this test never sends is a field no test proves ``run_project`` accepts"
|
||||
)
|
||||
|
||||
status, body = await _post(served, "/invocations", payload)
|
||||
|
||||
assert status == 200, f"a real run did not survive the hosted surface: {body}"
|
||||
assert body["outcome_type"] == "validated"
|
||||
# The validator's arithmetic, not a scripted string: P90 is computed from the proposal's own
|
||||
# cost line, and the claim (30 000) clears it.
|
||||
assert body["p90"] == _BUNDLE_P90
|
||||
assert body["proposal"]["claimed_saving_nok"] == 30000.0
|
||||
assert body["proposal"]["project_id"] == _BUNDLE_PID
|
||||
# Provenance carries a first-class citation from the NAVIGATED bundle — the run really read the
|
||||
# knowledge base rather than answering from the payload.
|
||||
assert body["provenance"]["citations"], "the run produced no citation — the bundle was not read"
|
||||
assert body["checker_verdict"] == "approve"
|
||||
assert _scripted_backend, "the scripted backend was never called — no real run happened"
|
||||
|
||||
|
||||
def test_deployment_artifacts_pin_the_measured_platform_and_one_start_command() -> None:
|
||||
"""RAW-TEXT GATE on the two files that decide whether the hosted image runs at all.
|
||||
|
||||
Neither is executed by any test: ``docker build`` and ``azd deploy`` are operator-gated, so a
|
||||
regression in them is invisible to the whole suite until it fails in the cloud. What the gate
|
||||
pins is exactly what was MEASURED, and nothing about the prose around it:
|
||||
|
||||
* ``--platform linux/amd64`` — the platform requires x86_64 (spike §1.4). Dropping it makes the
|
||||
image inherit the builder's architecture, which on an arm64 laptop yields an image that
|
||||
builds green locally and cannot start in the cloud. The flag lives in the documented build
|
||||
command (a Dockerfile cannot set the build platform for its own invocation), so pinning the
|
||||
documented string is the only gate available — and it is worth having precisely because
|
||||
nothing else re-derives it.
|
||||
* ONE copy of the start command: the image's ``CMD`` names ``main.py``, and ``azure.yaml``
|
||||
declares NO ``startupCommand``. Two copies drift (kø-(p)-regelen); this is the pair that
|
||||
keeps there being one.
|
||||
|
||||
``env:`` is checked for the same reason: ``FOUNDRY_PROJECT_ENDPOINT`` is injected by the
|
||||
platform, and redeclaring it here could shadow the injected value — which is the failure mode
|
||||
Fase 4b's endpoint precedence exists to avoid, undone from the config side."""
|
||||
dockerfile = _REPO_ROOT / "Dockerfile"
|
||||
azure_yaml = _REPO_ROOT / "azure.yaml"
|
||||
assert dockerfile.is_file(), "the hosted image has no build definition"
|
||||
assert azure_yaml.is_file(), "azd has no project definition to deploy"
|
||||
|
||||
docker_text = dockerfile.read_text(encoding="utf-8")
|
||||
azure_text = azure_yaml.read_text(encoding="utf-8")
|
||||
|
||||
assert "--platform linux/amd64" in docker_text, (
|
||||
"the Dockerfile no longer names the measured build platform; an image built without it "
|
||||
"inherits the builder's architecture and cannot start on the hosting platform"
|
||||
)
|
||||
assert "CMD" in docker_text and "main.py" in docker_text, (
|
||||
"the image's CMD is the ONE copy of the start command and must name main.py"
|
||||
)
|
||||
# Line-anchored: a mention inside a comment is prose, a top-level key is a declaration.
|
||||
azure_keys = [line.split(":")[0].strip() for line in azure_text.splitlines()]
|
||||
assert "startupCommand" not in azure_keys, (
|
||||
"azure.yaml declares a startupCommand — a SECOND copy of the start command, free to drift "
|
||||
"from the image's CMD"
|
||||
)
|
||||
assert "env" not in azure_keys, (
|
||||
"azure.yaml declares an env block — FOUNDRY_PROJECT_ENDPOINT is injected by the platform "
|
||||
"and must never be redeclared here"
|
||||
)
|
||||
assert "host: azure.ai.agent" in azure_text and "kind: hosted" in azure_text, (
|
||||
"azure.yaml no longer declares the hosted-agent host this whole entrypoint targets"
|
||||
)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue