feat(toolbox): the seven outbox writers get their own door -- row 1 moves 8 -> 15 of 17

Every one of the run path's seven `outbox.write_*` steps was reachable only through
`run.main`, and every path through that builds a chat client. The steps need no model:
they take already-rendered data and put it on disk in a byte-deterministic form.

Seven subcommands, seven thin adapters. The outbox directory is always the caller's to
name -- never a default, never the repository's own, because a step that wrote into a
folder the framework also reads as an inbox would bypass the Step-8 promotion gate.

`write-outbox` is the one that is not purely mechanical: `outbox.write_outbox` branches
on the outcome TYPE, so a door that took the outcome as an argument would let anyone
author an outbox of claims and hand it to Step 8 as results. The door DERIVES it through
`validate_proposal` -- the run path's own composition -- and a blocked proposal exits 3
with the artefacts still written, since that is where the rejection is recorded.
`verdict_id` stays an argument: `verdict-key` already owns that minting.

`--stop-reason` is required rather than defaulted to the empty string, inheriting the
core writer's measured reason: "the run finished" and "we never found out" must not be
the same value.

Eight probes, each a subprocess with the subcommand in argv, each asserting on the FILE
the command wrote. The ground truth is composed in the test -- the payload it wrote and
counted itself, and the byte form the contract requires -- never `outbox._dump`, which
would have measured the module against itself. The refusal arm carries its rc-0 control.

Measured, own run of the B gate: row 1 8 of 17 -> 15 of 17, exit 1 unchanged, no other
row moved. 0 chat-client names reachable from the toolbox (known-positive control: 24 in
run.py).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 10:32:12 +02:00
commit 6375ce5af3
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
5 changed files with 691 additions and 30 deletions

View file

@ -35,7 +35,9 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
navn frysen må bære. Verktøykassen er der fordi den er det ENE de to andre ikke kan brukes til: navn frysen må bære. Verktøykassen er der fordi den er det ENE de to andre ikke kan brukes til:
hver vei gjennom `run` bygger en chatklient, og stegene den bygger på trenger ingen modell. hver vei gjennom `run` bygger en chatklient, og stegene den bygger på trenger ingen modell.
Underkommandoene er stegenes egne navn (`navigate-bundle`, `cost-baseline`, `retrieve-chunks`, Underkommandoene er stegenes egne navn (`navigate-bundle`, `cost-baseline`, `retrieve-chunks`,
`prepass-admit`, `validate-proposal`, `verdict-key`, `capture-verdict`) og hver er en TYNN `prepass-admit`, `validate-proposal`, `verdict-key`, `capture-verdict`, og utboks-skriverne
`write-run-config`, `write-coverage`, `write-outbox`, `write-prepass`, `write-parse-failures`,
`write-proposal-reviews`, `write-debate-tools`) og hver er en TYNN
adapter over den funksjonen kjørestien kaller — aldri en andre implementasjon. Exit 0 kjørte, adapter over den funksjonen kjørestien kaller — aldri en andre implementasjon. Exit 0 kjørte,
2 feil kall, 3 NEKTET med grunnen navngitt; et forslag den deterministiske validatoren BLOKKERER 2 feil kall, 3 NEKTET med grunnen navngitt; et forslag den deterministiske validatoren BLOKKERER
er også 3, og bærer dommen (ordrett grunn + stadiet) framfor en unntaks-konvolutt. er også 3, og bærer dommen (ordrett grunn + stadiet) framfor en unntaks-konvolutt.

View file

@ -113,8 +113,30 @@ uv run portfolio-optimiser-toolbox verdict-key --proposal <proposal.json>
# Mint an expert's judgement into the Verdict the store holds (stdout is the on-disk form) # Mint an expert's judgement into the Verdict the store holds (stdout is the on-disk form)
uv run portfolio-optimiser-toolbox capture-verdict --proposal <proposal.json> \ uv run portfolio-optimiser-toolbox capture-verdict --proposal <proposal.json> \
--decision approved --rationale "why" --decision approved --rationale "why"
# Write the run's artefacts. Seven writers, one subcommand each — the outbox directory is always
# yours to name, never a default: a run must not write into a folder it also reads as an inbox.
uv run portfolio-optimiser-toolbox write-run-config --out-dir <dir> --run-id <id> \
--profile local --resolved-models <roles.json> \
--max-rounds 5 --max-tokens 600000 --top-k 3
uv run portfolio-optimiser-toolbox write-outbox --outbox-dir <dir> --run-id <id> \
--proposal <proposal.json> --provenance <stamp.json> --verdict-id <key>
uv run portfolio-optimiser-toolbox write-coverage --outbox-dir <dir> --run-id <id> \
--rows <rows.json> --stop-reason ""
uv run portfolio-optimiser-toolbox write-prepass --outbox-dir <dir> --run-id <id> \
--declaration <cut.json>
uv run portfolio-optimiser-toolbox write-parse-failures --outbox-dir <dir> --run-id <id> \
--failures <failures.json>
uv run portfolio-optimiser-toolbox write-proposal-reviews --outbox-dir <dir> --run-id <id> \
--payload <reviews.json>
uv run portfolio-optimiser-toolbox write-debate-tools --outbox-dir <dir> --run-id <id> \
--tool-calls <calls.json> [--requirements <requirements.json>]
``` ```
`write-outbox` does not take the outcome — it DERIVES it, by running the proposal through the
same blocking gate the run path runs it through, and exits `3` when that gate says no. An outbox
whose verdicts the caller could declare would be a collection of claims, not of results.
Each subcommand calls the same function the run path calls — not a copy of it. That is what makes Each subcommand calls the same function the run path calls — not a copy of it. That is what makes
the answers you get here the answers the debate gets: a proposal a human wrote meets the same the answers you get here the answers the debate gets: a proposal a human wrote meets the same
stages, in the same order, with the same sentence, as one an agent produced. stages, in the same order, with the same sentence, as one an agent produced.

View file

@ -263,11 +263,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-run-config"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_run_config_from_outside_is_the_byte_deterministic_artefact"
]
}, },
{ {
"id": "coverage", "id": "coverage",
@ -279,11 +282,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-coverage"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_coverage_from_outside_keeps_every_row_and_the_stop_reason"
]
}, },
{ {
"id": "utboks", "id": "utboks",
@ -295,11 +301,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-outbox"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_outbox_from_outside_writes_the_pair_the_run_path_writes"
]
}, },
{ {
"id": "rundebinding", "id": "rundebinding",
@ -365,11 +374,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-prepass"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_prepass_from_outside_records_the_cut_the_run_was_given"
]
}, },
{ {
"id": "parse-feil", "id": "parse-feil",
@ -381,11 +393,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-parse-failures"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_parse_failures_from_outside_keeps_every_reply_that_did_not_parse"
]
}, },
{ {
"id": "forslagsvurderinger", "id": "forslagsvurderinger",
@ -397,11 +412,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-proposal-reviews"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_proposal_reviews_from_outside_states_an_empty_review_list"
]
}, },
{ {
"id": "debatt-verktøy", "id": "debatt-verktøy",
@ -413,11 +431,14 @@
"scope": "run_project" "scope": "run_project"
}, },
"entry": { "entry": {
"kind": "console-script", "kind": "subcommand",
"module": "run.py", "module": "toolbox.py",
"scope": "main" "scope": "main",
"command": "write-debate-tools"
}, },
"probe": [] "probe": [
"tests/test_toolbox_outbox_doors.py::test_write_debate_tools_from_outside_writes_an_empty_trace_as_a_statement"
]
} }
], ],
"roles": { "roles": {

View file

@ -37,10 +37,11 @@ from dataclasses import dataclass
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from portfolio_optimiser import okf, prepass from portfolio_optimiser import okf, outbox, prepass
from portfolio_optimiser.contracts import FeedbackContract from portfolio_optimiser.contracts import FeedbackContract
from portfolio_optimiser.datasource import retrieve_chunks from portfolio_optimiser.datasource import retrieve_chunks
from portfolio_optimiser.ir import SavingsProposal from portfolio_optimiser.ir import SavingsProposal
from portfolio_optimiser.provenance import ProvenanceStamp
from portfolio_optimiser.validator import ( from portfolio_optimiser.validator import (
ValidatedProposal, ValidatedProposal,
rejection_stage, rejection_stage,
@ -255,6 +256,151 @@ def capture_verdict_command(args: argparse.Namespace) -> Mapping[str, Any]:
return verdict_to_dict(verdict) return verdict_to_dict(verdict)
def _payload_file(path: str, expected: type) -> Any:
"""A JSON argument, read as the shape the core writer already takes.
The type is checked HERE rather than left to the writer, because the writers take
``Mapping``/``Sequence`` and would serialize whatever they were handed: a list where an
object belongs becomes a valid file with the wrong shape, and the run that reads it later is
the one that fails. Exit 2 is the wrong code for it (the call parsed), so it is a refusal."""
value = json.loads(Path(path).read_text(encoding="utf-8"))
if not isinstance(value, expected):
raise ValueError(f"{path}: expected a JSON {expected.__name__}, got {type(value).__name__}")
return value
def write_run_config_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-run-config`` — everything that describes a run WITHOUT a model call.
The first artefact a caller weighing whether to pay for a debate has use for: the resolved
deployment per role, the profile and the caps, in the byte-deterministic form the comparison
protocol reads. ``resolved_models`` comes in as data because the run path resolves it before
this step too (``resolve_model`` is held out of the toolbox for exactly that reason it
looks up the chat client's deployment)."""
models = _payload_file(args.resolved_models, dict)
path = outbox.write_run_config(
args.out_dir,
args.run_id,
profile=args.profile,
resolved_models={str(role): str(model) for role, model in models.items()},
max_rounds=args.max_rounds,
max_tokens=args.max_tokens,
top_k=args.top_k,
)
return {"path": str(path), "run_id": args.run_id}
def write_coverage_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-coverage`` — WHY each commissioned approach ended as it did.
``--stop-reason`` is required, never defaulted to the empty string, because the core writer
requires it for a measured reason: "the run finished" and "we never found out" must not be
the same value. A door that supplied the empty string on the caller's behalf would turn
every unfinished run into a finished one."""
rows = _payload_file(args.rows, list)
path = outbox.write_coverage(
args.outbox_dir, args.run_id, rows=rows, stop_reason=args.stop_reason
)
return {"path": str(path), "run_id": args.run_id, "rows": len(rows)}
def write_outbox_command(args: argparse.Namespace) -> Mapping[str, Any] | Refused:
"""``write-outbox`` — the proposal/outcome pair, with the outcome DERIVED, not declared.
The outcome is not an argument, and that is the load-bearing decision here. ``write_outbox``
branches on the outcome TYPE a ``ValidatedProposal`` writes percentiles, a ``Rejection``
writes its reason so a door that let the caller hand in either would let anyone author an
outbox of claims and hand it to Step 8 as results. The door runs the proposal through the
same blocking gate the run path runs it through, and writes what came back.
``verdict_id`` IS an argument: the run path mints it with ``verdict_key`` before this step,
and that name has its own door (``verdict-key``). Minting it a second time here would give
the learning key two producers.
A blocked proposal is ``REFUSED``, and the artefacts are still written they are where the
rejection is recorded. The exit code answers the caller's question, not the writer's: an
agent that only reads rc must never take a blocked proposal for a cleared one."""
proposal = _proposal_from(args.proposal)
stamp = ProvenanceStamp.model_validate_json(Path(args.provenance).read_text(encoding="utf-8"))
baseline = (
None if args.cost_baseline is None else okf.load_cost_baseline_file(args.cost_baseline)
)
outcome = validate_proposal(proposal, baseline=baseline)
proposal_path, outcome_path = outbox.write_outbox(
args.outbox_dir,
args.run_id,
outcome=outcome,
provenance=stamp,
checker_verdict=args.checker_verdict,
verdict_id=args.verdict_id,
approach_id=args.approach_id,
)
validated = isinstance(outcome, ValidatedProposal)
payload = {
"decision": "validated" if validated else "rejected",
"proposal_path": str(proposal_path),
"outcome_path": str(outcome_path),
"run_id": args.run_id,
"verdict_id": args.verdict_id,
}
return payload if validated else Refused(payload)
def write_prepass_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-prepass`` — the CUT this run was given, as a file.
Its PRESENCE is what tells "withdrawn by design" apart from the S2c regression, so the door
writes whenever it is called, exactly as the run path writes whenever a payload was given."""
declaration = _payload_file(args.declaration, dict)
path = outbox.write_prepass(args.outbox_dir, args.run_id, declaration=declaration)
return {"path": str(path), "run_id": args.run_id}
def write_parse_failures_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-parse-failures`` — the replies that did NOT become typed IR.
Plain mappings in, exactly as the run path flattens ``generate.ParseFailure`` before this
step: the RAW output layer stays free of the module that imports MAF, and so does this door."""
failures = _payload_file(args.failures, list)
path = outbox.write_parse_failures(
args.outbox_dir,
args.run_id,
failures=[{str(k): str(v) for k, v in f.items()} for f in failures],
)
return {"path": str(path), "run_id": args.run_id, "failures": len(failures)}
def write_proposal_reviews_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-proposal-reviews`` — what a human answered about the proposals on the table.
Already-rendered payload in, for the run path's reason: the renderer lives in the module that
owns the type. The write rule iff a reviewer was given, INCLUDING an empty list is the
caller's here, because calling this door IS giving one."""
payload = _payload_file(args.payload, dict)
path = outbox.write_proposal_reviews(args.outbox_dir, args.run_id, payload=payload)
return {"path": str(path), "run_id": args.run_id}
def write_debate_tools_command(args: argparse.Namespace) -> Mapping[str, Any]:
"""``write-debate-tools`` — WHICH documents were opened, and WHICH requirement was binding.
``--requirements`` defaults to empty for the reason the core writer's parameter does: "this
debate declared nothing" is an honest positive statement, and an empty trace is the S2c
regression itself it has to be readable off the artefact, never inferred from a file that
is not there."""
tool_calls = _payload_file(args.tool_calls, list)
requirements = [] if args.requirements is None else _payload_file(args.requirements, list)
path = outbox.write_debate_tools(
args.outbox_dir, args.run_id, tool_calls=tool_calls, requirements=requirements
)
return {
"path": str(path),
"run_id": args.run_id,
"tool_calls": len(tool_calls),
"requirements": len(requirements),
}
def build_parser() -> argparse.ArgumentParser: def build_parser() -> argparse.ArgumentParser:
"""The doors, each registered by name. """The doors, each registered by name.
@ -303,6 +449,69 @@ def build_parser() -> argparse.ArgumentParser:
fang.add_argument("--proposal", required=True, help="the proposal IR as JSON") fang.add_argument("--proposal", required=True, help="the proposal IR as JSON")
fang.add_argument("--decision", required=True, help="the expert's decision") fang.add_argument("--decision", required=True, help="the expert's decision")
fang.add_argument("--rationale", required=True, help="why — carried into the store verbatim") fang.add_argument("--rationale", required=True, help="why — carried into the store verbatim")
#: The outbox doors. Each takes an outbox directory FROM THE CALLER — never a default, and
#: never the repository's own: a step that wrote into a folder the framework also reads as an
#: inbox would bypass the Step-8 promotion gate, and a probe that did it would leave files a
#: later run counts as its own.
konfig = sub.add_parser("write-run-config", help="write the run-config artefact")
konfig.add_argument("--out-dir", required=True, help="where the artefact is written")
konfig.add_argument("--run-id", required=True, help="the run the artefact belongs to")
konfig.add_argument("--profile", required=True, help="the backend profile the run used")
konfig.add_argument(
"--resolved-models", required=True, help="JSON object: role -> resolved deployment"
)
konfig.add_argument("--max-rounds", type=int, required=True, help="the round cap")
konfig.add_argument("--max-tokens", type=int, required=True, help="the token cap")
konfig.add_argument("--top-k", type=int, required=True, help="retrieval depth")
dekning = sub.add_parser("write-coverage", help="write the approach-coverage artefact")
dekning.add_argument("--outbox-dir", required=True, help="where the artefact is written")
dekning.add_argument("--run-id", required=True, help="the run the artefact belongs to")
dekning.add_argument("--rows", required=True, help="JSON array of coverage rows")
dekning.add_argument(
"--stop-reason",
required=True,
help="what cut the run short, or the empty string when nothing did — required, because "
"'the run finished' and 'we never found out' must not be the same value",
)
utboks = sub.add_parser("write-outbox", help="write the proposal/outcome artefact pair")
utboks.add_argument("--outbox-dir", required=True, help="where the artefacts are written")
utboks.add_argument("--run-id", required=True, help="the run the artefacts belong to")
utboks.add_argument("--proposal", required=True, help="the proposal IR as JSON")
utboks.add_argument("--provenance", required=True, help="the provenance stamp as JSON")
utboks.add_argument("--verdict-id", required=True, help="the key from verdict-key")
utboks.add_argument("--checker-verdict", default=None, help="the checker's decision, if any")
utboks.add_argument("--approach-id", default=None, help="which commissioned approach this is")
utboks.add_argument(
"--cost-baseline",
default=None,
help="the project's own priced lines; without it stage 0 never runs",
)
kutt = sub.add_parser("write-prepass", help="write the pre-pass declaration artefact")
kutt.add_argument("--outbox-dir", required=True, help="where the artefact is written")
kutt.add_argument("--run-id", required=True, help="the run the artefact belongs to")
kutt.add_argument("--declaration", required=True, help="the producer's declaration JSON")
parse = sub.add_parser("write-parse-failures", help="write the unparsed-replies artefact")
parse.add_argument("--outbox-dir", required=True, help="where the artefact is written")
parse.add_argument("--run-id", required=True, help="the run the artefact belongs to")
parse.add_argument("--failures", required=True, help="JSON array of {text, error} objects")
vurdering = sub.add_parser("write-proposal-reviews", help="write the expert-review artefact")
vurdering.add_argument("--outbox-dir", required=True, help="where the artefact is written")
vurdering.add_argument("--run-id", required=True, help="the run the artefact belongs to")
vurdering.add_argument("--payload", required=True, help="the rendered review payload JSON")
debatt = sub.add_parser("write-debate-tools", help="write the debate navigation artefact")
debatt.add_argument("--outbox-dir", required=True, help="where the artefact is written")
debatt.add_argument("--run-id", required=True, help="the run the artefact belongs to")
debatt.add_argument("--tool-calls", required=True, help="JSON array of tool calls, in order")
debatt.add_argument(
"--requirements", default=None, help="JSON array of declared binding requirements"
)
return parser return parser
@ -327,6 +536,20 @@ def dispatch(args: argparse.Namespace) -> Any:
return verdict_key_command(args) return verdict_key_command(args)
if args.command == "capture-verdict": if args.command == "capture-verdict":
return capture_verdict_command(args) return capture_verdict_command(args)
if args.command == "write-run-config":
return write_run_config_command(args)
if args.command == "write-coverage":
return write_coverage_command(args)
if args.command == "write-outbox":
return write_outbox_command(args)
if args.command == "write-prepass":
return write_prepass_command(args)
if args.command == "write-parse-failures":
return write_parse_failures_command(args)
if args.command == "write-proposal-reviews":
return write_proposal_reviews_command(args)
if args.command == "write-debate-tools":
return write_debate_tools_command(args)
raise RuntimeError(f"unregistered command {args.command!r}") # pragma: no cover - argparse raise RuntimeError(f"unregistered command {args.command!r}") # pragma: no cover - argparse

View file

@ -0,0 +1,393 @@
"""B-gatens rad 1, steg 4: de SJU utboks-skriverne, drevet UTENFRA som kommandoer.
Kjørestien skriver sju artefakter (`outbox.write_*`) som ingen utenfor po kunne be om: veien til
dem gikk gjennom `run.main`, og hver vei gjennom den bygger en chatklient. Stegene selv trenger
ingen modell de tar ferdig rendret data og legger den disk i en byte-deterministisk form.
Denne fila måler at de har en dør, og at det som kom ut av døren er artefaktet kjørestien
ville skrevet.
Hver arm er en **atferdsprobe** i gatens forstand: `portfolio_optimiser.toolbox` startes som en
subprosess med underkommandoens navn i argv, og asserten leser FILA kommandoen skrev. Ingen arm
importerer skriveren og kaller den in-prosess.
Fasiten er hentet UTENFOR døren i hver arm: nyttelasten testen selv skrev (og teller selv), og
den byte-formen kontrakten krever `json.dumps(sort_keys=True, indent=2)` pluss avsluttende
linjeskift komponert her, ikke lest fra `outbox._dump`. En arm som sammenlignet døren med
dørens egen hjelper ville vært grønn uansett hva den skrev.
Artefaktene skrives ALLTID til `tmp_path`. En probe som skrev i repoets egen utboks ville lagt
igjen filer en senere kjøring leser som sine egne.
"""
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
from typing import Any
_REPO = Path(__file__).resolve().parents[1]
_MIKRO = _REPO / "shared" / "examples" / "bygg-energi-mikro"
_IR = _MIKRO / "validator-input.json"
_GOLDEN = json.loads((_MIKRO / "golden.json").read_text(encoding="utf-8"))["validator"]
_RUN = "probe-utboks-01"
def _toolbox(*args: str) -> subprocess.CompletedProcess[str]:
"""Døren, som en subprosess. ``-m``-formen med vilje: den virker i en ren klone uten sync."""
return subprocess.run(
[sys.executable, "-m", "portfolio_optimiser.toolbox", *args],
cwd=_REPO,
capture_output=True,
text=True,
)
def _json_arg(path: Path, payload: Any) -> str:
path.write_text(json.dumps(payload), encoding="utf-8")
return str(path)
def _deterministic(payload: Any) -> str:
"""Kontraktens byte-form, komponert HER. Ikke `outbox._dump` — da målte armen en likhet
modulen har med seg selv."""
return json.dumps(payload, sort_keys=True, indent=2) + "\n"
# --- write-run-config -------------------------------------------------------------------------
def test_write_run_config_from_outside_is_the_byte_deterministic_artefact(tmp_path: Path) -> None:
"""Kjørekonfigurasjonen beskriver en kjøring UTEN et modellkall — og er derfor nøyaktig det
steget en kaller utenfor po har bruk for før han bestemmer seg for å betale for en."""
modeller = _json_arg(tmp_path / "modeller.json", {"proposer": "gpt-x", "checker": "gpt-y"})
ut = tmp_path / "ut"
proc = _toolbox(
"write-run-config",
"--out-dir",
str(ut),
"--run-id",
_RUN,
"--profile",
"local",
"--resolved-models",
modeller,
"--max-rounds",
"5",
"--max-tokens",
"600000",
"--top-k",
"3",
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-runconfig.json"
assert skrevet.read_text(encoding="utf-8") == _deterministic(
{
"run_id": _RUN,
"profile": "local",
"resolved_models": {"proposer": "gpt-x", "checker": "gpt-y"},
"max_rounds": 5,
"max_tokens": 600000,
"top_k": 3,
}
)
# --- write-coverage ---------------------------------------------------------------------------
def test_write_coverage_from_outside_keeps_every_row_and_the_stop_reason(tmp_path: Path) -> None:
"""Uavhengig telling: raden-lista skrives HER, og armen teller den her. `stop_reason` er
påkrevd i kjernefunksjonen fordi «kjøringen ble ferdig» og «vi fikk aldri vite» ikke være
samme verdi døren arver det kravet framfor å defaulte til tom streng."""
rader = [
{
"id": "a1",
"label": "Etterisolering",
"status": "validated",
"detail": "",
"saving_nok": 30000.0,
},
{
"id": "a2",
"label": "Styring",
"status": "not_evaluated",
"detail": "budsjett",
"saving_nok": None,
},
]
kilde = _json_arg(tmp_path / "rader.json", rader)
ut = tmp_path / "ut"
proc = _toolbox(
"write-coverage",
"--outbox-dir",
str(ut),
"--run-id",
_RUN,
"--rows",
kilde,
"--stop-reason",
"tokens",
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-coverage.json"
paa_disk = json.loads(skrevet.read_text(encoding="utf-8"))
assert paa_disk["stop_reason"] == "tokens"
assert len(paa_disk["rows"]) == len(rader)
assert paa_disk["rows"] == rader
assert [r["id"] for r in paa_disk["rows"]] == ["a1", "a2"]
# --- write-outbox -----------------------------------------------------------------------------
def _stamp(path: Path, *, decision: str, anchored: bool) -> str:
"""En provenance-stempel-JSON på modellens egen form (`ProvenanceStamp`)."""
return _json_arg(
path,
{
"citations": [
{
"file": "index.md",
"locator": {"start_index": 0, "end_index": 12},
"snippet": "energiprofil",
}
],
"model": "ingen-modell",
"role": "proposer",
"validator_decision": decision,
"token_usage": 0,
"cost_baseline_anchored": anchored,
"bundle_id_source": None,
"code_forms": {"ENERGI-TOTAL-EL": "identifier"},
"external_calls": [],
},
)
def _baseline(path: Path, codes: dict[str, tuple[float, float]]) -> str:
return _json_arg(
path,
{
"project_id": "BYGG-KONTOR-NORD",
"items": {c: {"quantity": q, "unit_cost": u} for c, (q, u) in codes.items()},
},
)
def test_write_outbox_from_outside_writes_the_pair_the_run_path_writes(tmp_path: Path) -> None:
"""Fasiten er basens INNSJEKKEDE golden-suite — produsentens frosne utfall av den samme
deterministiske porten, skrevet før verktøykassen fantes. Utfallet er ikke noe kalleren kan
OPPGI: døren utleder det gjennom `validate_proposal`, ellers ville utboksen vært en samling
påstander framfor resultater."""
ut = tmp_path / "ut"
proc = _toolbox(
"write-outbox",
"--outbox-dir",
str(ut),
"--run-id",
_RUN,
"--proposal",
str(_IR),
"--provenance",
_stamp(tmp_path / "stempel.json", decision="validated", anchored=False),
"--verdict-id",
"v-probe",
"--checker-verdict",
"approve",
)
assert proc.returncode == 0, proc.stderr
skrevet = json.loads(proc.stdout)
forslag = Path(skrevet["proposal_path"])
utfall = Path(skrevet["outcome_path"])
assert forslag == ut / f"{_RUN}-proposal.json"
assert utfall == ut / f"{_RUN}-outcome.json"
ir = json.loads(_IR.read_text(encoding="utf-8"))
paa_disk = json.loads(forslag.read_text(encoding="utf-8"))
assert paa_disk["run_id"] == _RUN
assert paa_disk["proposal"]["project_id"] == ir["project_id"]
assert paa_disk["proposal"]["measure"] == ir["measure"]
assert paa_disk["provenance"]["validator_decision"] == "validated"
dom = json.loads(utfall.read_text(encoding="utf-8"))
assert dom["p50"] == _GOLDEN["p50"]
assert dom["p10"] == _GOLDEN["p10"]
assert dom["p90"] == _GOLDEN["p90"]
assert dom["verdict_id"] == "v-probe"
assert dom["checker_verdict"] == "approve"
def test_write_outbox_carries_a_blocked_proposal_out_as_a_refusal(tmp_path: Path) -> None:
"""Et forslag den deterministiske porten BLOKKERER er exit 3 gjennom denne døren også —
artefaktene skrives (det er dem dommen bor i), men en kaller som bare leser exit-koden skal
aldri se et blokkert forslag som et klarert."""
stempel = _stamp(tmp_path / "stempel.json", decision="rejected", anchored=True)
ukjent = _baseline(tmp_path / "ukjent.json", {"VARME-TOTAL": (1.0, 2.0)})
ut = tmp_path / "nekt"
proc = _toolbox(
"write-outbox",
"--outbox-dir",
str(ut),
"--run-id",
_RUN,
"--proposal",
str(_IR),
"--provenance",
stempel,
"--verdict-id",
"v-nekt",
"--cost-baseline",
ukjent,
)
assert proc.returncode == 3, proc.stdout
skrevet = json.loads(proc.stdout)
assert skrevet["decision"] == "rejected"
dom = json.loads(Path(skrevet["outcome_path"]).read_text(encoding="utf-8"))
assert dom["reason"].startswith("unknown cost code 'ENERGI-TOTAL-EL'")
assert "p50" not in dom
# rc-0-kontroll: SAMME forslag og SAMME stempel mot et grunnlag som priser koden. Uten den
# målte armen over ingenting — den ville vært grønn mot en dør som avviste alt.
priset = _baseline(tmp_path / "priset.json", {"ENERGI-TOTAL-EL": (300000.0, 1.0)})
ok = _toolbox(
"write-outbox",
"--outbox-dir",
str(tmp_path / "ok"),
"--run-id",
_RUN,
"--proposal",
str(_IR),
"--provenance",
stempel,
"--verdict-id",
"v-ok",
"--cost-baseline",
priset,
)
assert ok.returncode == 0, ok.stdout
assert json.loads(ok.stdout)["decision"] == "validated"
# --- write-prepass ----------------------------------------------------------------------------
def test_write_prepass_from_outside_records_the_cut_the_run_was_given(tmp_path: Path) -> None:
"""Artefaktets TILSTEDEVÆRELSE er det som skiller «tilbaketrukket med vilje» fra «regrederte»
når debatt-sporet er tomt derfor måler armen både at fila finnes og hva den sier."""
erklaering = {"bundle_id": "bygg-energi-mikro-fixture", "considered": 12, "delivered": 4}
kilde = _json_arg(tmp_path / "kutt.json", erklaering)
ut = tmp_path / "ut"
proc = _toolbox(
"write-prepass", "--outbox-dir", str(ut), "--run-id", _RUN, "--declaration", kilde
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-prepass.json"
assert skrevet.read_text(encoding="utf-8") == _deterministic(
{"run_id": _RUN, "prepass": erklaering}
)
# --- write-parse-failures ---------------------------------------------------------------------
def test_write_parse_failures_from_outside_keeps_every_reply_that_did_not_parse(
tmp_path: Path,
) -> None:
"""Uavhengig telling: to feil inn, to feil ut — og rekkefølgen er kjøringens, ikke sortert."""
feil = [
{"text": "{ nesten json", "error": "Expecting property name"},
{"text": "prosa uten IR", "error": "Expecting value"},
]
kilde = _json_arg(tmp_path / "feil.json", feil)
ut = tmp_path / "ut"
proc = _toolbox(
"write-parse-failures", "--outbox-dir", str(ut), "--run-id", _RUN, "--failures", kilde
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-parse-failures.json"
paa_disk = json.loads(skrevet.read_text(encoding="utf-8"))
assert len(paa_disk["parse_failures"]) == len(feil)
assert paa_disk["parse_failures"] == feil
# --- write-proposal-reviews -------------------------------------------------------------------
def test_write_proposal_reviews_from_outside_states_an_empty_review_list(tmp_path: Path) -> None:
"""Regelen er «hvis en vurderer ble gitt, OGSÅ når lista er tom» (D4): en vurderer som ble
tilbudt og aldri konsultert er et faktum artefaktet skal SI, ikke noe en operatør slutte
seg til av en fil som ikke er der. Døren arver regelen ved å skrive det den får."""
ut = tmp_path / "ut"
tom = _json_arg(tmp_path / "tom.json", {"reviews": []})
proc = _toolbox(
"write-proposal-reviews", "--outbox-dir", str(ut), "--run-id", _RUN, "--payload", tom
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-proposal-reviews.json"
assert skrevet.read_text(encoding="utf-8") == _deterministic({"run_id": _RUN, "reviews": []})
# Kontroll mot en arm som ville vært grønn for enhver nyttelast: en ikke-tom liste må komme
# ut som den gikk inn, med samme telling.
en = _json_arg(tmp_path / "en.json", {"reviews": [{"verdict_id": "v1", "decision": "accept"}]})
andre = _toolbox(
"write-proposal-reviews",
"--outbox-dir",
str(ut),
"--run-id",
"probe-utboks-02",
"--payload",
en,
)
assert andre.returncode == 0, andre.stderr
paa_disk = json.loads(Path(json.loads(andre.stdout)["path"]).read_text(encoding="utf-8"))
assert len(paa_disk["reviews"]) == 1
assert paa_disk["reviews"][0]["verdict_id"] == "v1"
# --- write-debate-tools -----------------------------------------------------------------------
def test_write_debate_tools_from_outside_writes_an_empty_trace_as_a_statement(
tmp_path: Path,
) -> None:
"""Skrives på HVER kjøring, også en debatt som åpnet ingenting — det tomme sporet ER S2c-
regresjonen, og den kunne leses av artefaktet framfor sluttes av en fil som mangler.
`requirements` defaulter til tom av samme grunn: «denne debatten erklærte ingenting» er en
ærlig positiv påstand."""
ut = tmp_path / "ut"
tomt = _json_arg(tmp_path / "ingen.json", [])
proc = _toolbox(
"write-debate-tools", "--outbox-dir", str(ut), "--run-id", _RUN, "--tool-calls", tomt
)
assert proc.returncode == 0, proc.stderr
skrevet = Path(json.loads(proc.stdout)["path"])
assert skrevet == ut / f"{_RUN}-debate.json"
assert skrevet.read_text(encoding="utf-8") == _deterministic(
{"run_id": _RUN, "tool_calls": [], "requirements": []}
)
# Kontroll: en debatt som FAKTISK åpnet noe skal se annerledes ut — ellers målte armen over
# bare at fila finnes.
kall = [
{"tool": "retrieve", "arguments": {"query": "energi"}},
{"tool": "retrieve", "arguments": {"query": "varme"}},
]
krav = [{"id": "R1", "text": "U-verdi"}]
andre = _toolbox(
"write-debate-tools",
"--outbox-dir",
str(ut),
"--run-id",
"probe-utboks-02",
"--tool-calls",
_json_arg(tmp_path / "kall.json", kall),
"--requirements",
_json_arg(tmp_path / "krav.json", krav),
)
assert andre.returncode == 0, andre.stderr
paa_disk = json.loads(Path(json.loads(andre.stdout)["path"]).read_text(encoding="utf-8"))
assert len(paa_disk["tool_calls"]) == len(kall)
assert len(paa_disk["requirements"]) == len(krav)
assert paa_disk["tool_calls"][1]["arguments"]["query"] == "varme"