feat(toolbox): the seven outbox writers get their own door -- row 1 moves 8 -> 15 of 17
Every one of the run path's seven `outbox.write_*` steps was reachable only through `run.main`, and every path through that builds a chat client. The steps need no model: they take already-rendered data and put it on disk in a byte-deterministic form. Seven subcommands, seven thin adapters. The outbox directory is always the caller's to name -- never a default, never the repository's own, because a step that wrote into a folder the framework also reads as an inbox would bypass the Step-8 promotion gate. `write-outbox` is the one that is not purely mechanical: `outbox.write_outbox` branches on the outcome TYPE, so a door that took the outcome as an argument would let anyone author an outbox of claims and hand it to Step 8 as results. The door DERIVES it through `validate_proposal` -- the run path's own composition -- and a blocked proposal exits 3 with the artefacts still written, since that is where the rejection is recorded. `verdict_id` stays an argument: `verdict-key` already owns that minting. `--stop-reason` is required rather than defaulted to the empty string, inheriting the core writer's measured reason: "the run finished" and "we never found out" must not be the same value. Eight probes, each a subprocess with the subcommand in argv, each asserting on the FILE the command wrote. The ground truth is composed in the test -- the payload it wrote and counted itself, and the byte form the contract requires -- never `outbox._dump`, which would have measured the module against itself. The refusal arm carries its rc-0 control. Measured, own run of the B gate: row 1 8 of 17 -> 15 of 17, exit 1 unchanged, no other row moved. 0 chat-client names reachable from the toolbox (known-positive control: 24 in run.py). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
1111924000
commit
6375ce5af3
5 changed files with 691 additions and 30 deletions
|
|
@ -35,7 +35,9 @@ Python ≥3.10. MAF (`agent-framework-core` 1.16.0, `-orchestrations` 1.1.1 —
|
||||||
navn frysen må bære. Verktøykassen er der fordi den er det ENE de to andre ikke kan brukes til:
|
navn frysen må bære. Verktøykassen er der fordi den er det ENE de to andre ikke kan brukes til:
|
||||||
hver vei gjennom `run` bygger en chatklient, og stegene den bygger på trenger ingen modell.
|
hver vei gjennom `run` bygger en chatklient, og stegene den bygger på trenger ingen modell.
|
||||||
Underkommandoene er stegenes egne navn (`navigate-bundle`, `cost-baseline`, `retrieve-chunks`,
|
Underkommandoene er stegenes egne navn (`navigate-bundle`, `cost-baseline`, `retrieve-chunks`,
|
||||||
`prepass-admit`, `validate-proposal`, `verdict-key`, `capture-verdict`) og hver er en TYNN
|
`prepass-admit`, `validate-proposal`, `verdict-key`, `capture-verdict`, og utboks-skriverne
|
||||||
|
`write-run-config`, `write-coverage`, `write-outbox`, `write-prepass`, `write-parse-failures`,
|
||||||
|
`write-proposal-reviews`, `write-debate-tools`) og hver er en TYNN
|
||||||
adapter over den funksjonen kjørestien kaller — aldri en andre implementasjon. Exit 0 kjørte,
|
adapter over den funksjonen kjørestien kaller — aldri en andre implementasjon. Exit 0 kjørte,
|
||||||
2 feil kall, 3 NEKTET med grunnen navngitt; et forslag den deterministiske validatoren BLOKKERER
|
2 feil kall, 3 NEKTET med grunnen navngitt; et forslag den deterministiske validatoren BLOKKERER
|
||||||
er også 3, og bærer dommen (ordrett grunn + stadiet) framfor en unntaks-konvolutt.
|
er også 3, og bærer dommen (ordrett grunn + stadiet) framfor en unntaks-konvolutt.
|
||||||
|
|
|
||||||
22
README.md
22
README.md
|
|
@ -113,8 +113,30 @@ uv run portfolio-optimiser-toolbox verdict-key --proposal <proposal.json>
|
||||||
# Mint an expert's judgement into the Verdict the store holds (stdout is the on-disk form)
|
# Mint an expert's judgement into the Verdict the store holds (stdout is the on-disk form)
|
||||||
uv run portfolio-optimiser-toolbox capture-verdict --proposal <proposal.json> \
|
uv run portfolio-optimiser-toolbox capture-verdict --proposal <proposal.json> \
|
||||||
--decision approved --rationale "why"
|
--decision approved --rationale "why"
|
||||||
|
|
||||||
|
# Write the run's artefacts. Seven writers, one subcommand each — the outbox directory is always
|
||||||
|
# yours to name, never a default: a run must not write into a folder it also reads as an inbox.
|
||||||
|
uv run portfolio-optimiser-toolbox write-run-config --out-dir <dir> --run-id <id> \
|
||||||
|
--profile local --resolved-models <roles.json> \
|
||||||
|
--max-rounds 5 --max-tokens 600000 --top-k 3
|
||||||
|
uv run portfolio-optimiser-toolbox write-outbox --outbox-dir <dir> --run-id <id> \
|
||||||
|
--proposal <proposal.json> --provenance <stamp.json> --verdict-id <key>
|
||||||
|
uv run portfolio-optimiser-toolbox write-coverage --outbox-dir <dir> --run-id <id> \
|
||||||
|
--rows <rows.json> --stop-reason ""
|
||||||
|
uv run portfolio-optimiser-toolbox write-prepass --outbox-dir <dir> --run-id <id> \
|
||||||
|
--declaration <cut.json>
|
||||||
|
uv run portfolio-optimiser-toolbox write-parse-failures --outbox-dir <dir> --run-id <id> \
|
||||||
|
--failures <failures.json>
|
||||||
|
uv run portfolio-optimiser-toolbox write-proposal-reviews --outbox-dir <dir> --run-id <id> \
|
||||||
|
--payload <reviews.json>
|
||||||
|
uv run portfolio-optimiser-toolbox write-debate-tools --outbox-dir <dir> --run-id <id> \
|
||||||
|
--tool-calls <calls.json> [--requirements <requirements.json>]
|
||||||
```
|
```
|
||||||
|
|
||||||
|
`write-outbox` does not take the outcome — it DERIVES it, by running the proposal through the
|
||||||
|
same blocking gate the run path runs it through, and exits `3` when that gate says no. An outbox
|
||||||
|
whose verdicts the caller could declare would be a collection of claims, not of results.
|
||||||
|
|
||||||
Each subcommand calls the same function the run path calls — not a copy of it. That is what makes
|
Each subcommand calls the same function the run path calls — not a copy of it. That is what makes
|
||||||
the answers you get here the answers the debate gets: a proposal a human wrote meets the same
|
the answers you get here the answers the debate gets: a proposal a human wrote meets the same
|
||||||
stages, in the same order, with the same sentence, as one an agent produced.
|
stages, in the same order, with the same sentence, as one an agent produced.
|
||||||
|
|
|
||||||
|
|
@ -263,11 +263,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-run-config"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_run_config_from_outside_is_the_byte_deterministic_artefact"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "coverage",
|
"id": "coverage",
|
||||||
|
|
@ -279,11 +282,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-coverage"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_coverage_from_outside_keeps_every_row_and_the_stop_reason"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "utboks",
|
"id": "utboks",
|
||||||
|
|
@ -295,11 +301,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-outbox"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_outbox_from_outside_writes_the_pair_the_run_path_writes"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "rundebinding",
|
"id": "rundebinding",
|
||||||
|
|
@ -365,11 +374,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-prepass"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_prepass_from_outside_records_the_cut_the_run_was_given"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "parse-feil",
|
"id": "parse-feil",
|
||||||
|
|
@ -381,11 +393,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-parse-failures"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_parse_failures_from_outside_keeps_every_reply_that_did_not_parse"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "forslagsvurderinger",
|
"id": "forslagsvurderinger",
|
||||||
|
|
@ -397,11 +412,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-proposal-reviews"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_proposal_reviews_from_outside_states_an_empty_review_list"
|
||||||
|
]
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"id": "debatt-verktøy",
|
"id": "debatt-verktøy",
|
||||||
|
|
@ -413,11 +431,14 @@
|
||||||
"scope": "run_project"
|
"scope": "run_project"
|
||||||
},
|
},
|
||||||
"entry": {
|
"entry": {
|
||||||
"kind": "console-script",
|
"kind": "subcommand",
|
||||||
"module": "run.py",
|
"module": "toolbox.py",
|
||||||
"scope": "main"
|
"scope": "main",
|
||||||
|
"command": "write-debate-tools"
|
||||||
},
|
},
|
||||||
"probe": []
|
"probe": [
|
||||||
|
"tests/test_toolbox_outbox_doors.py::test_write_debate_tools_from_outside_writes_an_empty_trace_as_a_statement"
|
||||||
|
]
|
||||||
}
|
}
|
||||||
],
|
],
|
||||||
"roles": {
|
"roles": {
|
||||||
|
|
|
||||||
|
|
@ -37,10 +37,11 @@ from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any
|
from typing import Any
|
||||||
|
|
||||||
from portfolio_optimiser import okf, prepass
|
from portfolio_optimiser import okf, outbox, prepass
|
||||||
from portfolio_optimiser.contracts import FeedbackContract
|
from portfolio_optimiser.contracts import FeedbackContract
|
||||||
from portfolio_optimiser.datasource import retrieve_chunks
|
from portfolio_optimiser.datasource import retrieve_chunks
|
||||||
from portfolio_optimiser.ir import SavingsProposal
|
from portfolio_optimiser.ir import SavingsProposal
|
||||||
|
from portfolio_optimiser.provenance import ProvenanceStamp
|
||||||
from portfolio_optimiser.validator import (
|
from portfolio_optimiser.validator import (
|
||||||
ValidatedProposal,
|
ValidatedProposal,
|
||||||
rejection_stage,
|
rejection_stage,
|
||||||
|
|
@ -255,6 +256,151 @@ def capture_verdict_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
return verdict_to_dict(verdict)
|
return verdict_to_dict(verdict)
|
||||||
|
|
||||||
|
|
||||||
|
def _payload_file(path: str, expected: type) -> Any:
|
||||||
|
"""A JSON argument, read as the shape the core writer already takes.
|
||||||
|
|
||||||
|
The type is checked HERE rather than left to the writer, because the writers take
|
||||||
|
``Mapping``/``Sequence`` and would serialize whatever they were handed: a list where an
|
||||||
|
object belongs becomes a valid file with the wrong shape, and the run that reads it later is
|
||||||
|
the one that fails. Exit 2 is the wrong code for it (the call parsed), so it is a refusal."""
|
||||||
|
value = json.loads(Path(path).read_text(encoding="utf-8"))
|
||||||
|
if not isinstance(value, expected):
|
||||||
|
raise ValueError(f"{path}: expected a JSON {expected.__name__}, got {type(value).__name__}")
|
||||||
|
return value
|
||||||
|
|
||||||
|
|
||||||
|
def write_run_config_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-run-config`` — everything that describes a run WITHOUT a model call.
|
||||||
|
|
||||||
|
The first artefact a caller weighing whether to pay for a debate has use for: the resolved
|
||||||
|
deployment per role, the profile and the caps, in the byte-deterministic form the comparison
|
||||||
|
protocol reads. ``resolved_models`` comes in as data because the run path resolves it before
|
||||||
|
this step too (``resolve_model`` is held out of the toolbox for exactly that reason — it
|
||||||
|
looks up the chat client's deployment)."""
|
||||||
|
models = _payload_file(args.resolved_models, dict)
|
||||||
|
path = outbox.write_run_config(
|
||||||
|
args.out_dir,
|
||||||
|
args.run_id,
|
||||||
|
profile=args.profile,
|
||||||
|
resolved_models={str(role): str(model) for role, model in models.items()},
|
||||||
|
max_rounds=args.max_rounds,
|
||||||
|
max_tokens=args.max_tokens,
|
||||||
|
top_k=args.top_k,
|
||||||
|
)
|
||||||
|
return {"path": str(path), "run_id": args.run_id}
|
||||||
|
|
||||||
|
|
||||||
|
def write_coverage_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-coverage`` — WHY each commissioned approach ended as it did.
|
||||||
|
|
||||||
|
``--stop-reason`` is required, never defaulted to the empty string, because the core writer
|
||||||
|
requires it for a measured reason: "the run finished" and "we never found out" must not be
|
||||||
|
the same value. A door that supplied the empty string on the caller's behalf would turn
|
||||||
|
every unfinished run into a finished one."""
|
||||||
|
rows = _payload_file(args.rows, list)
|
||||||
|
path = outbox.write_coverage(
|
||||||
|
args.outbox_dir, args.run_id, rows=rows, stop_reason=args.stop_reason
|
||||||
|
)
|
||||||
|
return {"path": str(path), "run_id": args.run_id, "rows": len(rows)}
|
||||||
|
|
||||||
|
|
||||||
|
def write_outbox_command(args: argparse.Namespace) -> Mapping[str, Any] | Refused:
|
||||||
|
"""``write-outbox`` — the proposal/outcome pair, with the outcome DERIVED, not declared.
|
||||||
|
|
||||||
|
The outcome is not an argument, and that is the load-bearing decision here. ``write_outbox``
|
||||||
|
branches on the outcome TYPE — a ``ValidatedProposal`` writes percentiles, a ``Rejection``
|
||||||
|
writes its reason — so a door that let the caller hand in either would let anyone author an
|
||||||
|
outbox of claims and hand it to Step 8 as results. The door runs the proposal through the
|
||||||
|
same blocking gate the run path runs it through, and writes what came back.
|
||||||
|
|
||||||
|
``verdict_id`` IS an argument: the run path mints it with ``verdict_key`` before this step,
|
||||||
|
and that name has its own door (``verdict-key``). Minting it a second time here would give
|
||||||
|
the learning key two producers.
|
||||||
|
|
||||||
|
A blocked proposal is ``REFUSED``, and the artefacts are still written — they are where the
|
||||||
|
rejection is recorded. The exit code answers the caller's question, not the writer's: an
|
||||||
|
agent that only reads rc must never take a blocked proposal for a cleared one."""
|
||||||
|
proposal = _proposal_from(args.proposal)
|
||||||
|
stamp = ProvenanceStamp.model_validate_json(Path(args.provenance).read_text(encoding="utf-8"))
|
||||||
|
baseline = (
|
||||||
|
None if args.cost_baseline is None else okf.load_cost_baseline_file(args.cost_baseline)
|
||||||
|
)
|
||||||
|
outcome = validate_proposal(proposal, baseline=baseline)
|
||||||
|
proposal_path, outcome_path = outbox.write_outbox(
|
||||||
|
args.outbox_dir,
|
||||||
|
args.run_id,
|
||||||
|
outcome=outcome,
|
||||||
|
provenance=stamp,
|
||||||
|
checker_verdict=args.checker_verdict,
|
||||||
|
verdict_id=args.verdict_id,
|
||||||
|
approach_id=args.approach_id,
|
||||||
|
)
|
||||||
|
validated = isinstance(outcome, ValidatedProposal)
|
||||||
|
payload = {
|
||||||
|
"decision": "validated" if validated else "rejected",
|
||||||
|
"proposal_path": str(proposal_path),
|
||||||
|
"outcome_path": str(outcome_path),
|
||||||
|
"run_id": args.run_id,
|
||||||
|
"verdict_id": args.verdict_id,
|
||||||
|
}
|
||||||
|
return payload if validated else Refused(payload)
|
||||||
|
|
||||||
|
|
||||||
|
def write_prepass_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-prepass`` — the CUT this run was given, as a file.
|
||||||
|
|
||||||
|
Its PRESENCE is what tells "withdrawn by design" apart from the S2c regression, so the door
|
||||||
|
writes whenever it is called, exactly as the run path writes whenever a payload was given."""
|
||||||
|
declaration = _payload_file(args.declaration, dict)
|
||||||
|
path = outbox.write_prepass(args.outbox_dir, args.run_id, declaration=declaration)
|
||||||
|
return {"path": str(path), "run_id": args.run_id}
|
||||||
|
|
||||||
|
|
||||||
|
def write_parse_failures_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-parse-failures`` — the replies that did NOT become typed IR.
|
||||||
|
|
||||||
|
Plain mappings in, exactly as the run path flattens ``generate.ParseFailure`` before this
|
||||||
|
step: the RAW output layer stays free of the module that imports MAF, and so does this door."""
|
||||||
|
failures = _payload_file(args.failures, list)
|
||||||
|
path = outbox.write_parse_failures(
|
||||||
|
args.outbox_dir,
|
||||||
|
args.run_id,
|
||||||
|
failures=[{str(k): str(v) for k, v in f.items()} for f in failures],
|
||||||
|
)
|
||||||
|
return {"path": str(path), "run_id": args.run_id, "failures": len(failures)}
|
||||||
|
|
||||||
|
|
||||||
|
def write_proposal_reviews_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-proposal-reviews`` — what a human answered about the proposals on the table.
|
||||||
|
|
||||||
|
Already-rendered payload in, for the run path's reason: the renderer lives in the module that
|
||||||
|
owns the type. The write rule — iff a reviewer was given, INCLUDING an empty list — is the
|
||||||
|
caller's here, because calling this door IS giving one."""
|
||||||
|
payload = _payload_file(args.payload, dict)
|
||||||
|
path = outbox.write_proposal_reviews(args.outbox_dir, args.run_id, payload=payload)
|
||||||
|
return {"path": str(path), "run_id": args.run_id}
|
||||||
|
|
||||||
|
|
||||||
|
def write_debate_tools_command(args: argparse.Namespace) -> Mapping[str, Any]:
|
||||||
|
"""``write-debate-tools`` — WHICH documents were opened, and WHICH requirement was binding.
|
||||||
|
|
||||||
|
``--requirements`` defaults to empty for the reason the core writer's parameter does: "this
|
||||||
|
debate declared nothing" is an honest positive statement, and an empty trace is the S2c
|
||||||
|
regression itself — it has to be readable off the artefact, never inferred from a file that
|
||||||
|
is not there."""
|
||||||
|
tool_calls = _payload_file(args.tool_calls, list)
|
||||||
|
requirements = [] if args.requirements is None else _payload_file(args.requirements, list)
|
||||||
|
path = outbox.write_debate_tools(
|
||||||
|
args.outbox_dir, args.run_id, tool_calls=tool_calls, requirements=requirements
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"path": str(path),
|
||||||
|
"run_id": args.run_id,
|
||||||
|
"tool_calls": len(tool_calls),
|
||||||
|
"requirements": len(requirements),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
def build_parser() -> argparse.ArgumentParser:
|
def build_parser() -> argparse.ArgumentParser:
|
||||||
"""The doors, each registered by name.
|
"""The doors, each registered by name.
|
||||||
|
|
||||||
|
|
@ -303,6 +449,69 @@ def build_parser() -> argparse.ArgumentParser:
|
||||||
fang.add_argument("--proposal", required=True, help="the proposal IR as JSON")
|
fang.add_argument("--proposal", required=True, help="the proposal IR as JSON")
|
||||||
fang.add_argument("--decision", required=True, help="the expert's decision")
|
fang.add_argument("--decision", required=True, help="the expert's decision")
|
||||||
fang.add_argument("--rationale", required=True, help="why — carried into the store verbatim")
|
fang.add_argument("--rationale", required=True, help="why — carried into the store verbatim")
|
||||||
|
|
||||||
|
#: The outbox doors. Each takes an outbox directory FROM THE CALLER — never a default, and
|
||||||
|
#: never the repository's own: a step that wrote into a folder the framework also reads as an
|
||||||
|
#: inbox would bypass the Step-8 promotion gate, and a probe that did it would leave files a
|
||||||
|
#: later run counts as its own.
|
||||||
|
konfig = sub.add_parser("write-run-config", help="write the run-config artefact")
|
||||||
|
konfig.add_argument("--out-dir", required=True, help="where the artefact is written")
|
||||||
|
konfig.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
konfig.add_argument("--profile", required=True, help="the backend profile the run used")
|
||||||
|
konfig.add_argument(
|
||||||
|
"--resolved-models", required=True, help="JSON object: role -> resolved deployment"
|
||||||
|
)
|
||||||
|
konfig.add_argument("--max-rounds", type=int, required=True, help="the round cap")
|
||||||
|
konfig.add_argument("--max-tokens", type=int, required=True, help="the token cap")
|
||||||
|
konfig.add_argument("--top-k", type=int, required=True, help="retrieval depth")
|
||||||
|
|
||||||
|
dekning = sub.add_parser("write-coverage", help="write the approach-coverage artefact")
|
||||||
|
dekning.add_argument("--outbox-dir", required=True, help="where the artefact is written")
|
||||||
|
dekning.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
dekning.add_argument("--rows", required=True, help="JSON array of coverage rows")
|
||||||
|
dekning.add_argument(
|
||||||
|
"--stop-reason",
|
||||||
|
required=True,
|
||||||
|
help="what cut the run short, or the empty string when nothing did — required, because "
|
||||||
|
"'the run finished' and 'we never found out' must not be the same value",
|
||||||
|
)
|
||||||
|
|
||||||
|
utboks = sub.add_parser("write-outbox", help="write the proposal/outcome artefact pair")
|
||||||
|
utboks.add_argument("--outbox-dir", required=True, help="where the artefacts are written")
|
||||||
|
utboks.add_argument("--run-id", required=True, help="the run the artefacts belong to")
|
||||||
|
utboks.add_argument("--proposal", required=True, help="the proposal IR as JSON")
|
||||||
|
utboks.add_argument("--provenance", required=True, help="the provenance stamp as JSON")
|
||||||
|
utboks.add_argument("--verdict-id", required=True, help="the key from verdict-key")
|
||||||
|
utboks.add_argument("--checker-verdict", default=None, help="the checker's decision, if any")
|
||||||
|
utboks.add_argument("--approach-id", default=None, help="which commissioned approach this is")
|
||||||
|
utboks.add_argument(
|
||||||
|
"--cost-baseline",
|
||||||
|
default=None,
|
||||||
|
help="the project's own priced lines; without it stage 0 never runs",
|
||||||
|
)
|
||||||
|
|
||||||
|
kutt = sub.add_parser("write-prepass", help="write the pre-pass declaration artefact")
|
||||||
|
kutt.add_argument("--outbox-dir", required=True, help="where the artefact is written")
|
||||||
|
kutt.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
kutt.add_argument("--declaration", required=True, help="the producer's declaration JSON")
|
||||||
|
|
||||||
|
parse = sub.add_parser("write-parse-failures", help="write the unparsed-replies artefact")
|
||||||
|
parse.add_argument("--outbox-dir", required=True, help="where the artefact is written")
|
||||||
|
parse.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
parse.add_argument("--failures", required=True, help="JSON array of {text, error} objects")
|
||||||
|
|
||||||
|
vurdering = sub.add_parser("write-proposal-reviews", help="write the expert-review artefact")
|
||||||
|
vurdering.add_argument("--outbox-dir", required=True, help="where the artefact is written")
|
||||||
|
vurdering.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
vurdering.add_argument("--payload", required=True, help="the rendered review payload JSON")
|
||||||
|
|
||||||
|
debatt = sub.add_parser("write-debate-tools", help="write the debate navigation artefact")
|
||||||
|
debatt.add_argument("--outbox-dir", required=True, help="where the artefact is written")
|
||||||
|
debatt.add_argument("--run-id", required=True, help="the run the artefact belongs to")
|
||||||
|
debatt.add_argument("--tool-calls", required=True, help="JSON array of tool calls, in order")
|
||||||
|
debatt.add_argument(
|
||||||
|
"--requirements", default=None, help="JSON array of declared binding requirements"
|
||||||
|
)
|
||||||
return parser
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -327,6 +536,20 @@ def dispatch(args: argparse.Namespace) -> Any:
|
||||||
return verdict_key_command(args)
|
return verdict_key_command(args)
|
||||||
if args.command == "capture-verdict":
|
if args.command == "capture-verdict":
|
||||||
return capture_verdict_command(args)
|
return capture_verdict_command(args)
|
||||||
|
if args.command == "write-run-config":
|
||||||
|
return write_run_config_command(args)
|
||||||
|
if args.command == "write-coverage":
|
||||||
|
return write_coverage_command(args)
|
||||||
|
if args.command == "write-outbox":
|
||||||
|
return write_outbox_command(args)
|
||||||
|
if args.command == "write-prepass":
|
||||||
|
return write_prepass_command(args)
|
||||||
|
if args.command == "write-parse-failures":
|
||||||
|
return write_parse_failures_command(args)
|
||||||
|
if args.command == "write-proposal-reviews":
|
||||||
|
return write_proposal_reviews_command(args)
|
||||||
|
if args.command == "write-debate-tools":
|
||||||
|
return write_debate_tools_command(args)
|
||||||
raise RuntimeError(f"unregistered command {args.command!r}") # pragma: no cover - argparse
|
raise RuntimeError(f"unregistered command {args.command!r}") # pragma: no cover - argparse
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
393
tests/test_toolbox_outbox_doors.py
Normal file
393
tests/test_toolbox_outbox_doors.py
Normal file
|
|
@ -0,0 +1,393 @@
|
||||||
|
"""B-gatens rad 1, steg 4: de SJU utboks-skriverne, drevet UTENFRA som kommandoer.
|
||||||
|
|
||||||
|
Kjørestien skriver sju artefakter (`outbox.write_*`) som ingen utenfor po kunne be om: veien til
|
||||||
|
dem gikk gjennom `run.main`, og hver vei gjennom den bygger en chatklient. Stegene selv trenger
|
||||||
|
ingen modell — de tar ferdig rendret data og legger den på disk i en byte-deterministisk form.
|
||||||
|
Denne fila måler at de nå har en dør, og at det som kom ut av døren er artefaktet kjørestien
|
||||||
|
ville skrevet.
|
||||||
|
|
||||||
|
Hver arm er en **atferdsprobe** i gatens forstand: `portfolio_optimiser.toolbox` startes som en
|
||||||
|
subprosess med underkommandoens navn i argv, og asserten leser FILA kommandoen skrev. Ingen arm
|
||||||
|
importerer skriveren og kaller den in-prosess.
|
||||||
|
|
||||||
|
Fasiten er hentet UTENFOR døren i hver arm: nyttelasten testen selv skrev (og teller selv), og
|
||||||
|
den byte-formen kontrakten krever — `json.dumps(sort_keys=True, indent=2)` pluss avsluttende
|
||||||
|
linjeskift — komponert her, ikke lest fra `outbox._dump`. En arm som sammenlignet døren med
|
||||||
|
dørens egen hjelper ville vært grønn uansett hva den skrev.
|
||||||
|
|
||||||
|
Artefaktene skrives ALLTID til `tmp_path`. En probe som skrev i repoets egen utboks ville lagt
|
||||||
|
igjen filer en senere kjøring leser som sine egne.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import subprocess
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
_REPO = Path(__file__).resolve().parents[1]
|
||||||
|
_MIKRO = _REPO / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
_IR = _MIKRO / "validator-input.json"
|
||||||
|
_GOLDEN = json.loads((_MIKRO / "golden.json").read_text(encoding="utf-8"))["validator"]
|
||||||
|
_RUN = "probe-utboks-01"
|
||||||
|
|
||||||
|
|
||||||
|
def _toolbox(*args: str) -> subprocess.CompletedProcess[str]:
|
||||||
|
"""Døren, som en subprosess. ``-m``-formen med vilje: den virker i en ren klone uten sync."""
|
||||||
|
return subprocess.run(
|
||||||
|
[sys.executable, "-m", "portfolio_optimiser.toolbox", *args],
|
||||||
|
cwd=_REPO,
|
||||||
|
capture_output=True,
|
||||||
|
text=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _json_arg(path: Path, payload: Any) -> str:
|
||||||
|
path.write_text(json.dumps(payload), encoding="utf-8")
|
||||||
|
return str(path)
|
||||||
|
|
||||||
|
|
||||||
|
def _deterministic(payload: Any) -> str:
|
||||||
|
"""Kontraktens byte-form, komponert HER. Ikke `outbox._dump` — da målte armen en likhet
|
||||||
|
modulen har med seg selv."""
|
||||||
|
return json.dumps(payload, sort_keys=True, indent=2) + "\n"
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-run-config -------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_run_config_from_outside_is_the_byte_deterministic_artefact(tmp_path: Path) -> None:
|
||||||
|
"""Kjørekonfigurasjonen beskriver en kjøring UTEN et modellkall — og er derfor nøyaktig det
|
||||||
|
steget en kaller utenfor po har bruk for før han bestemmer seg for å betale for en."""
|
||||||
|
modeller = _json_arg(tmp_path / "modeller.json", {"proposer": "gpt-x", "checker": "gpt-y"})
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-run-config",
|
||||||
|
"--out-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
_RUN,
|
||||||
|
"--profile",
|
||||||
|
"local",
|
||||||
|
"--resolved-models",
|
||||||
|
modeller,
|
||||||
|
"--max-rounds",
|
||||||
|
"5",
|
||||||
|
"--max-tokens",
|
||||||
|
"600000",
|
||||||
|
"--top-k",
|
||||||
|
"3",
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-runconfig.json"
|
||||||
|
assert skrevet.read_text(encoding="utf-8") == _deterministic(
|
||||||
|
{
|
||||||
|
"run_id": _RUN,
|
||||||
|
"profile": "local",
|
||||||
|
"resolved_models": {"proposer": "gpt-x", "checker": "gpt-y"},
|
||||||
|
"max_rounds": 5,
|
||||||
|
"max_tokens": 600000,
|
||||||
|
"top_k": 3,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-coverage ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_coverage_from_outside_keeps_every_row_and_the_stop_reason(tmp_path: Path) -> None:
|
||||||
|
"""Uavhengig telling: raden-lista skrives HER, og armen teller den her. `stop_reason` er
|
||||||
|
påkrevd i kjernefunksjonen fordi «kjøringen ble ferdig» og «vi fikk aldri vite» ikke må være
|
||||||
|
samme verdi — døren arver det kravet framfor å defaulte til tom streng."""
|
||||||
|
rader = [
|
||||||
|
{
|
||||||
|
"id": "a1",
|
||||||
|
"label": "Etterisolering",
|
||||||
|
"status": "validated",
|
||||||
|
"detail": "",
|
||||||
|
"saving_nok": 30000.0,
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "a2",
|
||||||
|
"label": "Styring",
|
||||||
|
"status": "not_evaluated",
|
||||||
|
"detail": "budsjett",
|
||||||
|
"saving_nok": None,
|
||||||
|
},
|
||||||
|
]
|
||||||
|
kilde = _json_arg(tmp_path / "rader.json", rader)
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-coverage",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
_RUN,
|
||||||
|
"--rows",
|
||||||
|
kilde,
|
||||||
|
"--stop-reason",
|
||||||
|
"tokens",
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-coverage.json"
|
||||||
|
paa_disk = json.loads(skrevet.read_text(encoding="utf-8"))
|
||||||
|
assert paa_disk["stop_reason"] == "tokens"
|
||||||
|
assert len(paa_disk["rows"]) == len(rader)
|
||||||
|
assert paa_disk["rows"] == rader
|
||||||
|
assert [r["id"] for r in paa_disk["rows"]] == ["a1", "a2"]
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-outbox -----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _stamp(path: Path, *, decision: str, anchored: bool) -> str:
|
||||||
|
"""En provenance-stempel-JSON på modellens egen form (`ProvenanceStamp`)."""
|
||||||
|
return _json_arg(
|
||||||
|
path,
|
||||||
|
{
|
||||||
|
"citations": [
|
||||||
|
{
|
||||||
|
"file": "index.md",
|
||||||
|
"locator": {"start_index": 0, "end_index": 12},
|
||||||
|
"snippet": "energiprofil",
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"model": "ingen-modell",
|
||||||
|
"role": "proposer",
|
||||||
|
"validator_decision": decision,
|
||||||
|
"token_usage": 0,
|
||||||
|
"cost_baseline_anchored": anchored,
|
||||||
|
"bundle_id_source": None,
|
||||||
|
"code_forms": {"ENERGI-TOTAL-EL": "identifier"},
|
||||||
|
"external_calls": [],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _baseline(path: Path, codes: dict[str, tuple[float, float]]) -> str:
|
||||||
|
return _json_arg(
|
||||||
|
path,
|
||||||
|
{
|
||||||
|
"project_id": "BYGG-KONTOR-NORD",
|
||||||
|
"items": {c: {"quantity": q, "unit_cost": u} for c, (q, u) in codes.items()},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_outbox_from_outside_writes_the_pair_the_run_path_writes(tmp_path: Path) -> None:
|
||||||
|
"""Fasiten er basens INNSJEKKEDE golden-suite — produsentens frosne utfall av den samme
|
||||||
|
deterministiske porten, skrevet før verktøykassen fantes. Utfallet er ikke noe kalleren kan
|
||||||
|
OPPGI: døren utleder det gjennom `validate_proposal`, ellers ville utboksen vært en samling
|
||||||
|
påstander framfor resultater."""
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-outbox",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
_RUN,
|
||||||
|
"--proposal",
|
||||||
|
str(_IR),
|
||||||
|
"--provenance",
|
||||||
|
_stamp(tmp_path / "stempel.json", decision="validated", anchored=False),
|
||||||
|
"--verdict-id",
|
||||||
|
"v-probe",
|
||||||
|
"--checker-verdict",
|
||||||
|
"approve",
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = json.loads(proc.stdout)
|
||||||
|
forslag = Path(skrevet["proposal_path"])
|
||||||
|
utfall = Path(skrevet["outcome_path"])
|
||||||
|
assert forslag == ut / f"{_RUN}-proposal.json"
|
||||||
|
assert utfall == ut / f"{_RUN}-outcome.json"
|
||||||
|
ir = json.loads(_IR.read_text(encoding="utf-8"))
|
||||||
|
paa_disk = json.loads(forslag.read_text(encoding="utf-8"))
|
||||||
|
assert paa_disk["run_id"] == _RUN
|
||||||
|
assert paa_disk["proposal"]["project_id"] == ir["project_id"]
|
||||||
|
assert paa_disk["proposal"]["measure"] == ir["measure"]
|
||||||
|
assert paa_disk["provenance"]["validator_decision"] == "validated"
|
||||||
|
dom = json.loads(utfall.read_text(encoding="utf-8"))
|
||||||
|
assert dom["p50"] == _GOLDEN["p50"]
|
||||||
|
assert dom["p10"] == _GOLDEN["p10"]
|
||||||
|
assert dom["p90"] == _GOLDEN["p90"]
|
||||||
|
assert dom["verdict_id"] == "v-probe"
|
||||||
|
assert dom["checker_verdict"] == "approve"
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_outbox_carries_a_blocked_proposal_out_as_a_refusal(tmp_path: Path) -> None:
|
||||||
|
"""Et forslag den deterministiske porten BLOKKERER er exit 3 gjennom denne døren også —
|
||||||
|
artefaktene skrives (det er dem dommen bor i), men en kaller som bare leser exit-koden skal
|
||||||
|
aldri se et blokkert forslag som et klarert."""
|
||||||
|
stempel = _stamp(tmp_path / "stempel.json", decision="rejected", anchored=True)
|
||||||
|
ukjent = _baseline(tmp_path / "ukjent.json", {"VARME-TOTAL": (1.0, 2.0)})
|
||||||
|
ut = tmp_path / "nekt"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-outbox",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
_RUN,
|
||||||
|
"--proposal",
|
||||||
|
str(_IR),
|
||||||
|
"--provenance",
|
||||||
|
stempel,
|
||||||
|
"--verdict-id",
|
||||||
|
"v-nekt",
|
||||||
|
"--cost-baseline",
|
||||||
|
ukjent,
|
||||||
|
)
|
||||||
|
assert proc.returncode == 3, proc.stdout
|
||||||
|
skrevet = json.loads(proc.stdout)
|
||||||
|
assert skrevet["decision"] == "rejected"
|
||||||
|
dom = json.loads(Path(skrevet["outcome_path"]).read_text(encoding="utf-8"))
|
||||||
|
assert dom["reason"].startswith("unknown cost code 'ENERGI-TOTAL-EL'")
|
||||||
|
assert "p50" not in dom
|
||||||
|
# rc-0-kontroll: SAMME forslag og SAMME stempel mot et grunnlag som priser koden. Uten den
|
||||||
|
# målte armen over ingenting — den ville vært grønn mot en dør som avviste alt.
|
||||||
|
priset = _baseline(tmp_path / "priset.json", {"ENERGI-TOTAL-EL": (300000.0, 1.0)})
|
||||||
|
ok = _toolbox(
|
||||||
|
"write-outbox",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(tmp_path / "ok"),
|
||||||
|
"--run-id",
|
||||||
|
_RUN,
|
||||||
|
"--proposal",
|
||||||
|
str(_IR),
|
||||||
|
"--provenance",
|
||||||
|
stempel,
|
||||||
|
"--verdict-id",
|
||||||
|
"v-ok",
|
||||||
|
"--cost-baseline",
|
||||||
|
priset,
|
||||||
|
)
|
||||||
|
assert ok.returncode == 0, ok.stdout
|
||||||
|
assert json.loads(ok.stdout)["decision"] == "validated"
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-prepass ----------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_prepass_from_outside_records_the_cut_the_run_was_given(tmp_path: Path) -> None:
|
||||||
|
"""Artefaktets TILSTEDEVÆRELSE er det som skiller «tilbaketrukket med vilje» fra «regrederte»
|
||||||
|
når debatt-sporet er tomt — derfor måler armen både at fila finnes og hva den sier."""
|
||||||
|
erklaering = {"bundle_id": "bygg-energi-mikro-fixture", "considered": 12, "delivered": 4}
|
||||||
|
kilde = _json_arg(tmp_path / "kutt.json", erklaering)
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-prepass", "--outbox-dir", str(ut), "--run-id", _RUN, "--declaration", kilde
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-prepass.json"
|
||||||
|
assert skrevet.read_text(encoding="utf-8") == _deterministic(
|
||||||
|
{"run_id": _RUN, "prepass": erklaering}
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-parse-failures ---------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_parse_failures_from_outside_keeps_every_reply_that_did_not_parse(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""Uavhengig telling: to feil inn, to feil ut — og rekkefølgen er kjøringens, ikke sortert."""
|
||||||
|
feil = [
|
||||||
|
{"text": "{ nesten json", "error": "Expecting property name"},
|
||||||
|
{"text": "prosa uten IR", "error": "Expecting value"},
|
||||||
|
]
|
||||||
|
kilde = _json_arg(tmp_path / "feil.json", feil)
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-parse-failures", "--outbox-dir", str(ut), "--run-id", _RUN, "--failures", kilde
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-parse-failures.json"
|
||||||
|
paa_disk = json.loads(skrevet.read_text(encoding="utf-8"))
|
||||||
|
assert len(paa_disk["parse_failures"]) == len(feil)
|
||||||
|
assert paa_disk["parse_failures"] == feil
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-proposal-reviews -------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_proposal_reviews_from_outside_states_an_empty_review_list(tmp_path: Path) -> None:
|
||||||
|
"""Regelen er «hvis en vurderer ble gitt, OGSÅ når lista er tom» (D4): en vurderer som ble
|
||||||
|
tilbudt og aldri konsultert er et faktum artefaktet skal SI, ikke noe en operatør må slutte
|
||||||
|
seg til av en fil som ikke er der. Døren arver regelen ved å skrive det den får."""
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
tom = _json_arg(tmp_path / "tom.json", {"reviews": []})
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-proposal-reviews", "--outbox-dir", str(ut), "--run-id", _RUN, "--payload", tom
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-proposal-reviews.json"
|
||||||
|
assert skrevet.read_text(encoding="utf-8") == _deterministic({"run_id": _RUN, "reviews": []})
|
||||||
|
# Kontroll mot en arm som ville vært grønn for enhver nyttelast: en ikke-tom liste må komme
|
||||||
|
# ut som den gikk inn, med samme telling.
|
||||||
|
en = _json_arg(tmp_path / "en.json", {"reviews": [{"verdict_id": "v1", "decision": "accept"}]})
|
||||||
|
andre = _toolbox(
|
||||||
|
"write-proposal-reviews",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
"probe-utboks-02",
|
||||||
|
"--payload",
|
||||||
|
en,
|
||||||
|
)
|
||||||
|
assert andre.returncode == 0, andre.stderr
|
||||||
|
paa_disk = json.loads(Path(json.loads(andre.stdout)["path"]).read_text(encoding="utf-8"))
|
||||||
|
assert len(paa_disk["reviews"]) == 1
|
||||||
|
assert paa_disk["reviews"][0]["verdict_id"] == "v1"
|
||||||
|
|
||||||
|
|
||||||
|
# --- write-debate-tools -----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def test_write_debate_tools_from_outside_writes_an_empty_trace_as_a_statement(
|
||||||
|
tmp_path: Path,
|
||||||
|
) -> None:
|
||||||
|
"""Skrives på HVER kjøring, også en debatt som åpnet ingenting — det tomme sporet ER S2c-
|
||||||
|
regresjonen, og den må kunne leses av artefaktet framfor sluttes av en fil som mangler.
|
||||||
|
`requirements` defaulter til tom av samme grunn: «denne debatten erklærte ingenting» er en
|
||||||
|
ærlig positiv påstand."""
|
||||||
|
ut = tmp_path / "ut"
|
||||||
|
tomt = _json_arg(tmp_path / "ingen.json", [])
|
||||||
|
proc = _toolbox(
|
||||||
|
"write-debate-tools", "--outbox-dir", str(ut), "--run-id", _RUN, "--tool-calls", tomt
|
||||||
|
)
|
||||||
|
assert proc.returncode == 0, proc.stderr
|
||||||
|
skrevet = Path(json.loads(proc.stdout)["path"])
|
||||||
|
assert skrevet == ut / f"{_RUN}-debate.json"
|
||||||
|
assert skrevet.read_text(encoding="utf-8") == _deterministic(
|
||||||
|
{"run_id": _RUN, "tool_calls": [], "requirements": []}
|
||||||
|
)
|
||||||
|
# Kontroll: en debatt som FAKTISK åpnet noe skal se annerledes ut — ellers målte armen over
|
||||||
|
# bare at fila finnes.
|
||||||
|
kall = [
|
||||||
|
{"tool": "retrieve", "arguments": {"query": "energi"}},
|
||||||
|
{"tool": "retrieve", "arguments": {"query": "varme"}},
|
||||||
|
]
|
||||||
|
krav = [{"id": "R1", "text": "U-verdi"}]
|
||||||
|
andre = _toolbox(
|
||||||
|
"write-debate-tools",
|
||||||
|
"--outbox-dir",
|
||||||
|
str(ut),
|
||||||
|
"--run-id",
|
||||||
|
"probe-utboks-02",
|
||||||
|
"--tool-calls",
|
||||||
|
_json_arg(tmp_path / "kall.json", kall),
|
||||||
|
"--requirements",
|
||||||
|
_json_arg(tmp_path / "krav.json", krav),
|
||||||
|
)
|
||||||
|
assert andre.returncode == 0, andre.stderr
|
||||||
|
paa_disk = json.loads(Path(json.loads(andre.stdout)["path"]).read_text(encoding="utf-8"))
|
||||||
|
assert len(paa_disk["tool_calls"]) == len(kall)
|
||||||
|
assert len(paa_disk["requirements"]) == len(krav)
|
||||||
|
assert paa_disk["tool_calls"][1]["arguments"]["query"] == "varme"
|
||||||
Loading…
Add table
Add a link
Reference in a new issue