feat(portfolio): K10 — notification/notifier seam, opt-in webhook egress (parity row 23) [skip-docs]
S5.2-analog. New notify.py: Notifier protocol + console/file/webhook sinks. The webhook (the one transport that leaves the machine) fires ONLY behind an explicit per-run opt-in flag (--allow-webhook-egress), mirroring ingest-spec §8 (the flag is a run argument, never a config field). Transport is injected — canned in the suite (NULL socket), real transport behind one seam function default_webhook_transport; an AST grep-guard proves no network path exists outside that seam. run.py (both outcomes — a budget stop notifies too) and hitl.py (read-only preserved) share the same opt-in-gated CLI seam, refusing a webhook-without-opt-in before any spend. Payload shape is stack-local (no shared notification spec; divergence documented). Two new load-bearing test files (18 tests): opt-in gate + payload structure + the grep-guard + run/hitl emit wiring + run-level opt-in threading, each detach-proven RED. 544 -> 562 green, full gate clean (ruff+format+mypy strict, 26 src files). README sync (test count x2 + notify.py module note + load-bearing omtale). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01RiTwaKLesgcwXx2mDviqpt
This commit is contained in:
parent
b9dd479865
commit
a2acfc0f98
6 changed files with 863 additions and 11 deletions
22
README.md
22
README.md
|
|
@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
|
||||||
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
> **Status:** the D7 build (S5–S10) is complete, and the deterministic **ingest layer**
|
||||||
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
> (CSV and SQL source types) has since been added in front of the loop. The deterministic
|
||||||
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
> backbone, the agentic loop, the learning loop, and the ingest connectors are wired seam by
|
||||||
> seam, each proven by load-bearing tests (544 tests, all running offline without an API
|
> seam, each proven by load-bearing tests (562 tests, all running offline without an API
|
||||||
> key). The programme's single budgeted **live model run has been executed and validated** —
|
> key). The programme's single budgeted **live model run has been executed and validated** —
|
||||||
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
> its artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
|
||||||
|
|
||||||
|
|
@ -141,6 +141,17 @@ description, never from its code)
|
||||||
now; K13 formalizes the dimension catalog) to an expert via a schema-validated table
|
now; K13 formalizes the dimension catalog) to an expert via a schema-validated table
|
||||||
(`nøkkel→ekspert`, fail-fast) with an optional default; an unmatched measure is UNROUTED.
|
(`nøkkel→ekspert`, fail-fast) with an optional default; an unmatched measure is UNROUTED.
|
||||||
`uv run python -m portfolio_optimiser_claude.hitl pending|route`.
|
`uv run python -m portfolio_optimiser_claude.hitl pending|route`.
|
||||||
|
- `notify.py` — deliverable notification sinks that never break the no-silent-egress
|
||||||
|
invariant (§8): `console` and `file` deliver locally, `webhook` is the one transport that
|
||||||
|
leaves the machine and fires ONLY behind an explicit per-run opt-in flag
|
||||||
|
(`--allow-webhook-egress`) — mirroring the ingest-spec §8 rule that *the flag is a run
|
||||||
|
argument, never a config field, so the config cannot grant itself network access*. The
|
||||||
|
webhook transport is injected: the suite passes a canned transport (no socket is ever
|
||||||
|
opened), and the real transport lives behind one seam function that the suite never calls (a
|
||||||
|
grep-guard proves no network path exists elsewhere in the module). `run.py` (on both
|
||||||
|
outcomes — a budget stop notifies too) and `hitl.py` (read-only preserved) share the same
|
||||||
|
opt-in-gated CLI seam, refusing a webhook-without-opt-in *before* any spend. The payload
|
||||||
|
shape is stack-local (no shared notification spec across the siblings).
|
||||||
|
|
||||||
### Load-bearing tests (§11)
|
### Load-bearing tests (§11)
|
||||||
|
|
||||||
|
|
@ -164,7 +175,12 @@ artifacts and stops before the first model call — a call-counting client prove
|
||||||
red the moment the stop seam is detached),
|
red the moment the stop seam is detached),
|
||||||
`test_hitl_loadbearing.py` (a proposal with no verdict is listed pending and disappears once
|
`test_hitl_loadbearing.py` (a proposal with no verdict is listed pending and disappears once
|
||||||
an inbox or promoted verdict shares its id — red the moment the id-join filter is detached —
|
an inbox or promoted verdict shares its id — red the moment the id-join filter is detached —
|
||||||
and hitl never writes any layer, proven by a before/after byte snapshot), and
|
and hitl never writes any layer, proven by a before/after byte snapshot),
|
||||||
|
`test_notify_loadbearing.py` and `test_notify_seam_loadbearing.py` (a webhook without the
|
||||||
|
per-run opt-in flag refuses fail-fast and its transport never fires — red the moment the gate
|
||||||
|
is detached — the canned transport receives the structured payload, an AST grep-guard proves
|
||||||
|
no network path lives outside the one injectable seam function, and the run/hitl entrances
|
||||||
|
emit on their outcomes while hitl stays read-only), and
|
||||||
`test_sdk_isolation.py` (local config cannot capture the checker).
|
`test_sdk_isolation.py` (local config cannot capture the checker).
|
||||||
|
|
||||||
## The ingest layer — CSV and SQL, in front of the loop
|
## The ingest layer — CSV and SQL, in front of the loop
|
||||||
|
|
@ -227,7 +243,7 @@ Python ≥3.10 · [`claude-agent-sdk`](https://pypi.org/project/claude-agent-sdk
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # install dependencies
|
uv sync # install dependencies
|
||||||
uv run pytest # 544 tests — run without any API key and without network
|
uv run pytest # 562 tests — run without any API key and without network
|
||||||
uv run ruff check . && uv run ruff format --check .
|
uv run ruff check . && uv run ruff format --check .
|
||||||
uv run mypy src # strict
|
uv run mypy src # strict
|
||||||
```
|
```
|
||||||
|
|
|
||||||
|
|
@ -41,6 +41,16 @@ from pydantic import BaseModel, Field, ValidationError, model_validator
|
||||||
|
|
||||||
from portfolio_optimiser_claude.inbox import load_inbox
|
from portfolio_optimiser_claude.inbox import load_inbox
|
||||||
from portfolio_optimiser_claude.ir import SavingsProposal
|
from portfolio_optimiser_claude.ir import SavingsProposal
|
||||||
|
from portfolio_optimiser_claude.notify import (
|
||||||
|
EgressNotPermitted,
|
||||||
|
Notification,
|
||||||
|
Notifier,
|
||||||
|
Transport,
|
||||||
|
add_notify_args,
|
||||||
|
build_notifiers,
|
||||||
|
emit,
|
||||||
|
notify_config_from_args,
|
||||||
|
)
|
||||||
from portfolio_optimiser_claude.okf import navigate_bundle
|
from portfolio_optimiser_claude.okf import navigate_bundle
|
||||||
|
|
||||||
_OUTCOME_SUFFIX = "-outcome.json"
|
_OUTCOME_SUFFIX = "-outcome.json"
|
||||||
|
|
@ -199,7 +209,7 @@ def _bundle_dirs(values: list[str] | None) -> list[Path]:
|
||||||
return [Path(value) for value in (values or [])]
|
return [Path(value) for value in (values or [])]
|
||||||
|
|
||||||
|
|
||||||
def _cmd_pending(args: argparse.Namespace) -> int:
|
def _cmd_pending(args: argparse.Namespace, notifiers: Sequence[Notifier]) -> int:
|
||||||
pending = pending_proposals(
|
pending = pending_proposals(
|
||||||
Path(args.outbox), Path(args.inbox), bundle_dirs=_bundle_dirs(args.bundle)
|
Path(args.outbox), Path(args.inbox), bundle_dirs=_bundle_dirs(args.bundle)
|
||||||
)
|
)
|
||||||
|
|
@ -209,10 +219,21 @@ def _cmd_pending(args: argparse.Namespace) -> int:
|
||||||
f" {proposal.run_id} verdict_id={proposal.verdict_id} "
|
f" {proposal.run_id} verdict_id={proposal.verdict_id} "
|
||||||
f"measure={proposal.measure!r} project={proposal.project_id}"
|
f"measure={proposal.measure!r} project={proposal.project_id}"
|
||||||
)
|
)
|
||||||
|
# K10: notification is egress, NOT writing any of the three read layers —
|
||||||
|
# the read-only invariant (§3 Step 7) holds (proven by the seam test's
|
||||||
|
# before/after byte snapshot of outbox+inbox).
|
||||||
|
emit(
|
||||||
|
notifiers,
|
||||||
|
Notification(
|
||||||
|
event="hitl.pending",
|
||||||
|
summary=f"{len(pending)} proposal(s) awaiting a verdict",
|
||||||
|
fields={"pending": len(pending)},
|
||||||
|
),
|
||||||
|
)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def _cmd_route(args: argparse.Namespace) -> int:
|
def _cmd_route(args: argparse.Namespace, notifiers: Sequence[Notifier]) -> int:
|
||||||
try:
|
try:
|
||||||
routing = load_routing(json.loads(Path(args.routing).read_text("utf-8")))
|
routing = load_routing(json.loads(Path(args.routing).read_text("utf-8")))
|
||||||
except (OSError, ValueError, ValidationError) as exc:
|
except (OSError, ValueError, ValidationError) as exc:
|
||||||
|
|
@ -229,15 +250,29 @@ def _cmd_route(args: argparse.Namespace) -> int:
|
||||||
f" {item.proposal.run_id} measure={item.proposal.measure!r} "
|
f" {item.proposal.run_id} measure={item.proposal.measure!r} "
|
||||||
f"verdict_id={item.proposal.verdict_id} -> {expert}"
|
f"verdict_id={item.proposal.verdict_id} -> {expert}"
|
||||||
)
|
)
|
||||||
|
unrouted = sum(1 for item in routed if item.expert is None)
|
||||||
|
emit(
|
||||||
|
notifiers,
|
||||||
|
Notification(
|
||||||
|
event="hitl.route",
|
||||||
|
summary=f"{len(routed)} pending proposal(s) routed, {unrouted} UNROUTED",
|
||||||
|
fields={"pending": len(routed), "unrouted": unrouted},
|
||||||
|
),
|
||||||
|
)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None) -> int:
|
def main(argv: list[str] | None = None, *, notifier_transport: Transport | None = None) -> int:
|
||||||
"""The thin CLI: ``pending`` lists outstanding proposals; ``route`` adds experts.
|
"""The thin CLI: ``pending`` lists outstanding proposals; ``route`` adds experts.
|
||||||
|
|
||||||
``pending`` is a pure report (exit 0). ``route`` loads the routing config
|
``pending`` is a pure report (exit 0). ``route`` loads the routing config
|
||||||
fail-fast — a malformed/missing config exits non-zero WITHOUT touching any
|
fail-fast — a malformed/missing config exits non-zero WITHOUT touching any
|
||||||
layer (§10). Neither subcommand writes anything.
|
layer (§10). Neither subcommand writes any of the three read layers.
|
||||||
|
|
||||||
|
An optional notify seam (K10) delivers a summary event; a ``--notify-webhook``
|
||||||
|
without ``--allow-webhook-egress`` is refused fail-fast (§8) BEFORE any read.
|
||||||
|
``notifier_transport`` is the injected webhook transport (canned in the suite,
|
||||||
|
``default_webhook_transport`` on the CLI path — no socket in tests).
|
||||||
"""
|
"""
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
description=(
|
description=(
|
||||||
|
|
@ -256,6 +291,7 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
action="append",
|
action="append",
|
||||||
help="bundle dir whose promoted verdicts also settle a proposal (repeatable)",
|
help="bundle dir whose promoted verdicts also settle a proposal (repeatable)",
|
||||||
)
|
)
|
||||||
|
add_notify_args(sub)
|
||||||
|
|
||||||
pending_parser = subparsers.add_parser("pending", help="list proposals awaiting a verdict")
|
pending_parser = subparsers.add_parser("pending", help="list proposals awaiting a verdict")
|
||||||
_add_common(pending_parser)
|
_add_common(pending_parser)
|
||||||
|
|
@ -269,7 +305,16 @@ def main(argv: list[str] | None = None) -> int:
|
||||||
route_parser.set_defaults(func=_cmd_route)
|
route_parser.set_defaults(func=_cmd_route)
|
||||||
|
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
result: int = args.func(args)
|
# K10 (§8): build the notify sinks BEFORE any read — a --notify-webhook
|
||||||
|
# without --allow-webhook-egress is refused fail-fast here (the opt-in is a
|
||||||
|
# run argument, never a config field).
|
||||||
|
try:
|
||||||
|
notifiers = build_notifiers(
|
||||||
|
notify_config_from_args(args), webhook_transport=notifier_transport
|
||||||
|
)
|
||||||
|
except EgressNotPermitted as exc:
|
||||||
|
parser.error(str(exc))
|
||||||
|
result: int = args.func(args, notifiers)
|
||||||
return result
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
229
src/portfolio_optimiser_claude/notify.py
Normal file
229
src/portfolio_optimiser_claude/notify.py
Normal file
|
|
@ -0,0 +1,229 @@
|
||||||
|
"""Notification delivery (method-spec §5; S5.2-analog; paritetsrad 23; K10).
|
||||||
|
|
||||||
|
Deliverable notification implementations that never break the no-silent-egress
|
||||||
|
invariant (§8). A run or a HITL pass can DELIVER a small structured event to one
|
||||||
|
or more sinks:
|
||||||
|
|
||||||
|
* ``ConsoleNotifier`` — prints a one-line summary (local, no socket).
|
||||||
|
* ``FileNotifier`` — writes the event as deterministic house JSON (local).
|
||||||
|
* ``WebhookNotifier`` — POSTs the event to a URL. This is the ONE transport that
|
||||||
|
leaves the machine, so it is gated behind an EXPLICIT per-run opt-in flag,
|
||||||
|
mirroring ingest-spec §8: *the flag is a run argument, never a config field —
|
||||||
|
the config cannot grant itself network access.* Without the flag the webhook
|
||||||
|
refuses fail-fast (``EgressNotPermitted``) before any transport exists.
|
||||||
|
|
||||||
|
The webhook transport is INJECTED (``Transport`` = ``(url, payload) -> None``).
|
||||||
|
The suite passes a canned transport, so no socket is ever opened; the real
|
||||||
|
transport lives behind the ONE seam function ``default_webhook_transport`` (the
|
||||||
|
only network path in this module — a grep-guard in
|
||||||
|
``tests/test_notify_loadbearing.py`` proves nothing outside it touches the
|
||||||
|
network). This mirrors the run entrance's client seam: ``run.py`` constructs the
|
||||||
|
real SDK client only on the CLI path, never in the suite.
|
||||||
|
|
||||||
|
Payload shape is STACK-LOCAL. There is no shared notification spec across the
|
||||||
|
MAF/SDK siblings; the ``event`` / ``summary`` / ``fields`` shape here is this
|
||||||
|
stack's own, and the divergence is accepted and documented (pinned by the
|
||||||
|
payload-structure test). ``fields`` must be JSON-serializable.
|
||||||
|
|
||||||
|
Run wiring: ``add_notify_args``/``notify_config_from_args`` give ``run.py`` and
|
||||||
|
``hitl.py`` the same opt-in-gated CLI seam; ``build_notifiers`` fails fast on a
|
||||||
|
webhook without opt-in; ``emit`` fans one event out to every configured sink.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Callable, Mapping, Protocol, Sequence, runtime_checkable
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.artifacts import _dump_json
|
||||||
|
|
||||||
|
# The injectable webhook seam: (url, payload bytes) -> None. The suite injects a
|
||||||
|
# canned callable; the run path injects ``default_webhook_transport``.
|
||||||
|
Transport = Callable[[str, bytes], None]
|
||||||
|
|
||||||
|
|
||||||
|
class EgressNotPermitted(RuntimeError):
|
||||||
|
"""A webhook was configured WITHOUT the explicit per-run egress opt-in (§8)."""
|
||||||
|
|
||||||
|
def __init__(self, url: str) -> None:
|
||||||
|
super().__init__(
|
||||||
|
f"webhook egress to {url!r} refused: it requires the explicit per-run "
|
||||||
|
"opt-in flag (a run argument, never a config field — the config cannot "
|
||||||
|
"grant itself network access, ingest-spec §8)"
|
||||||
|
)
|
||||||
|
self.url = url
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class Notification:
|
||||||
|
"""One structured event to deliver — the stack-local payload shape (§1).
|
||||||
|
|
||||||
|
``event`` is a stable machine key (e.g. ``"run.completed"``); ``summary`` is
|
||||||
|
one human line; ``fields`` carries the structured, JSON-serializable detail.
|
||||||
|
"""
|
||||||
|
|
||||||
|
event: str
|
||||||
|
summary: str
|
||||||
|
fields: Mapping[str, object] = field(default_factory=dict)
|
||||||
|
|
||||||
|
def as_payload(self) -> dict[str, object]:
|
||||||
|
"""The canonical dict — the SAME shape every sink serializes."""
|
||||||
|
return {"event": self.event, "summary": self.summary, "fields": dict(self.fields)}
|
||||||
|
|
||||||
|
|
||||||
|
def _encode(event: Notification) -> bytes:
|
||||||
|
"""Deterministic compact JSON bytes (sorted keys) — no run-to-run drift."""
|
||||||
|
return json.dumps(event.as_payload(), sort_keys=True).encode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class Notifier(Protocol):
|
||||||
|
"""A delivery sink — one event in, delivered to one destination."""
|
||||||
|
|
||||||
|
def notify(self, event: Notification) -> None: ...
|
||||||
|
|
||||||
|
|
||||||
|
class ConsoleNotifier:
|
||||||
|
"""Print a one-line summary — local, never a socket."""
|
||||||
|
|
||||||
|
def notify(self, event: Notification) -> None:
|
||||||
|
print(f"NOTIFY [{event.event}] {event.summary}")
|
||||||
|
|
||||||
|
|
||||||
|
class FileNotifier:
|
||||||
|
"""Write the event as deterministic house JSON to ``{out_dir}/{event}.json``."""
|
||||||
|
|
||||||
|
def __init__(self, out_dir: Path) -> None:
|
||||||
|
self._out_dir = out_dir
|
||||||
|
|
||||||
|
def notify(self, event: Notification) -> None:
|
||||||
|
self._out_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
_dump_json(self._out_dir / f"{event.event}.json", event.as_payload())
|
||||||
|
|
||||||
|
|
||||||
|
class WebhookNotifier:
|
||||||
|
"""POST the event to ``url`` via the injected ``transport`` — egress, opt-in only.
|
||||||
|
|
||||||
|
The opt-in gate lives HERE, at construction (fail-fast, §8): an un-opted-in
|
||||||
|
webhook never comes into existence, so its transport can never fire. The only
|
||||||
|
construction path is ``build_notifiers``, which threads the run-argument flag
|
||||||
|
into ``egress_opt_in``.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, url: str, transport: Transport, *, egress_opt_in: bool) -> None:
|
||||||
|
if not egress_opt_in:
|
||||||
|
raise EgressNotPermitted(url)
|
||||||
|
self._url = url
|
||||||
|
self._transport = transport
|
||||||
|
|
||||||
|
def notify(self, event: Notification) -> None:
|
||||||
|
self._transport(self._url, _encode(event))
|
||||||
|
|
||||||
|
|
||||||
|
def default_webhook_transport() -> Transport:
|
||||||
|
"""THE injectable network seam — the ONLY socket path in this module.
|
||||||
|
|
||||||
|
``urllib`` is imported LAZILY here (like ``run.py``'s ``default_client_factory``
|
||||||
|
imports the SDK client) so importing ``notify.py`` stays network-free and the
|
||||||
|
grep-guard can assert every network import lives INSIDE this one function.
|
||||||
|
Never called by the suite — tests inject a canned transport, so no socket is
|
||||||
|
opened. Returning the closure opens nothing; only invoking it (a real,
|
||||||
|
opted-in run) reaches the network.
|
||||||
|
"""
|
||||||
|
import urllib.request
|
||||||
|
|
||||||
|
def _send(url: str, payload: bytes) -> None:
|
||||||
|
request = urllib.request.Request(
|
||||||
|
url,
|
||||||
|
data=payload,
|
||||||
|
headers={"Content-Type": "application/json"},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
with urllib.request.urlopen(request): # noqa: S310 — opt-in egress, URL is operator-set
|
||||||
|
pass
|
||||||
|
|
||||||
|
return _send
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class NotifyConfig:
|
||||||
|
"""The notify seam's run arguments — the webhook opt-in is one of them (§8).
|
||||||
|
|
||||||
|
``webhook_egress_opt_in`` is a RUN ARGUMENT (the CLI flag), never a config
|
||||||
|
field: a config cannot grant itself egress. A ``webhook_url`` without the flag
|
||||||
|
is refused by ``build_notifiers`` before any transport exists.
|
||||||
|
"""
|
||||||
|
|
||||||
|
console: bool = False
|
||||||
|
file_dir: Path | None = None
|
||||||
|
webhook_url: str | None = None
|
||||||
|
webhook_egress_opt_in: bool = False
|
||||||
|
|
||||||
|
|
||||||
|
def build_notifiers(
|
||||||
|
cfg: NotifyConfig, *, webhook_transport: Transport | None = None
|
||||||
|
) -> list[Notifier]:
|
||||||
|
"""Build the configured sinks; fail-fast on a webhook without the opt-in (§8).
|
||||||
|
|
||||||
|
The transport is injected: the suite passes a canned transport (no socket);
|
||||||
|
the run path leaves it ``None`` so ``default_webhook_transport`` is used. An
|
||||||
|
empty config builds nothing.
|
||||||
|
"""
|
||||||
|
notifiers: list[Notifier] = []
|
||||||
|
if cfg.console:
|
||||||
|
notifiers.append(ConsoleNotifier())
|
||||||
|
if cfg.file_dir is not None:
|
||||||
|
notifiers.append(FileNotifier(cfg.file_dir))
|
||||||
|
if cfg.webhook_url is not None:
|
||||||
|
transport = (
|
||||||
|
webhook_transport if webhook_transport is not None else default_webhook_transport()
|
||||||
|
)
|
||||||
|
notifiers.append(
|
||||||
|
WebhookNotifier(cfg.webhook_url, transport, egress_opt_in=cfg.webhook_egress_opt_in)
|
||||||
|
)
|
||||||
|
return notifiers
|
||||||
|
|
||||||
|
|
||||||
|
def emit(notifiers: Sequence[Notifier], event: Notification) -> None:
|
||||||
|
"""Fan one event out to every configured sink (no-op when there are none)."""
|
||||||
|
for notifier in notifiers:
|
||||||
|
notifier.notify(event)
|
||||||
|
|
||||||
|
|
||||||
|
def add_notify_args(parser: argparse.ArgumentParser) -> None:
|
||||||
|
"""Add the shared notify CLI seam to a parser (used by run.py and hitl.py)."""
|
||||||
|
parser.add_argument(
|
||||||
|
"--notify-console", action="store_true", help="print a one-line notification summary"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--notify-file",
|
||||||
|
type=Path,
|
||||||
|
default=None,
|
||||||
|
metavar="DIR",
|
||||||
|
help="write the notification as deterministic JSON to DIR",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--notify-webhook",
|
||||||
|
type=str,
|
||||||
|
default=None,
|
||||||
|
metavar="URL",
|
||||||
|
help="POST the notification to URL (egress — requires --allow-webhook-egress)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--allow-webhook-egress",
|
||||||
|
action="store_true",
|
||||||
|
help="explicit per-run opt-in for webhook egress (§8; a run argument, never config)",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def notify_config_from_args(args: argparse.Namespace) -> NotifyConfig:
|
||||||
|
"""Map the shared notify CLI flags into a ``NotifyConfig``."""
|
||||||
|
return NotifyConfig(
|
||||||
|
console=bool(args.notify_console),
|
||||||
|
file_dir=args.notify_file,
|
||||||
|
webhook_url=args.notify_webhook,
|
||||||
|
webhook_egress_opt_in=bool(args.allow_webhook_egress),
|
||||||
|
)
|
||||||
|
|
@ -24,7 +24,7 @@ from __future__ import annotations
|
||||||
import argparse
|
import argparse
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
from typing import Any, Callable
|
from typing import Any, Callable, Sequence
|
||||||
|
|
||||||
from portfolio_optimiser_claude.artifacts import (
|
from portfolio_optimiser_claude.artifacts import (
|
||||||
_dump_json,
|
_dump_json,
|
||||||
|
|
@ -47,6 +47,16 @@ from portfolio_optimiser_claude.ir import SavingsProposal, load_validator_input
|
||||||
from portfolio_optimiser_claude.okf import bundle_context, navigate_bundle
|
from portfolio_optimiser_claude.okf import bundle_context, navigate_bundle
|
||||||
from portfolio_optimiser_claude.provenance import Citation, Provenance
|
from portfolio_optimiser_claude.provenance import Citation, Provenance
|
||||||
from portfolio_optimiser_claude.loop import ModelClient, run_project
|
from portfolio_optimiser_claude.loop import ModelClient, run_project
|
||||||
|
from portfolio_optimiser_claude.notify import (
|
||||||
|
EgressNotPermitted,
|
||||||
|
Notification,
|
||||||
|
Notifier,
|
||||||
|
Transport,
|
||||||
|
add_notify_args,
|
||||||
|
build_notifiers,
|
||||||
|
emit,
|
||||||
|
notify_config_from_args,
|
||||||
|
)
|
||||||
from portfolio_optimiser_claude.validator import Rejection
|
from portfolio_optimiser_claude.validator import Rejection
|
||||||
|
|
||||||
_PROPOSER_ROLE = "proposer"
|
_PROPOSER_ROLE = "proposer"
|
||||||
|
|
@ -135,6 +145,7 @@ def execute_run(
|
||||||
max_attempts: int,
|
max_attempts: int,
|
||||||
outbox_dir: Path | None = None,
|
outbox_dir: Path | None = None,
|
||||||
run_id: str | None = None,
|
run_id: str | None = None,
|
||||||
|
notifiers: Sequence[Notifier] = (),
|
||||||
) -> int:
|
) -> int:
|
||||||
"""Drive the loop under the §8 meter; persist artifacts on BOTH outcomes.
|
"""Drive the loop under the §8 meter; persist artifacts on BOTH outcomes.
|
||||||
|
|
||||||
|
|
@ -146,7 +157,12 @@ def execute_run(
|
||||||
proposal/outcome pair to the outbox (S2.1) — the system's own output layer,
|
proposal/outcome pair to the outbox (S2.1) — the system's own output layer,
|
||||||
read by K8 (live capture) and K9 (pending tracking). A budget stop has no
|
read by K8 (live capture) and K9 (pending tracking). A budget stop has no
|
||||||
proposal, so it writes no outbox pair.
|
proposal, so it writes no outbox pair.
|
||||||
|
|
||||||
|
``notifiers`` (K10) receive a structured event on BOTH outcomes — a budget
|
||||||
|
stop is a run outcome, not an absence of one, so it notifies too. The list
|
||||||
|
is empty unless the operator configured a sink; delivery never gates the run.
|
||||||
"""
|
"""
|
||||||
|
run_label = run_id or out_dir.name
|
||||||
meter = BudgetMeter(contracts.termination)
|
meter = BudgetMeter(contracts.termination)
|
||||||
try:
|
try:
|
||||||
result = run_project(
|
result = run_project(
|
||||||
|
|
@ -169,6 +185,14 @@ def execute_run(
|
||||||
)
|
)
|
||||||
for name, path in sorted(stop_paths.items()):
|
for name, path in sorted(stop_paths.items()):
|
||||||
print(f"artifact: {name} -> {path}")
|
print(f"artifact: {name} -> {path}")
|
||||||
|
emit(
|
||||||
|
notifiers,
|
||||||
|
Notification(
|
||||||
|
event="run.stopped",
|
||||||
|
summary=f"run {run_label} stopped by budget: {stop.kind}",
|
||||||
|
fields={"kind": stop.kind, "observed": stop.observed, "limit": stop.limit},
|
||||||
|
),
|
||||||
|
)
|
||||||
return 3
|
return 3
|
||||||
|
|
||||||
provenance = Provenance(
|
provenance = Provenance(
|
||||||
|
|
@ -200,6 +224,19 @@ def execute_run(
|
||||||
)
|
)
|
||||||
for name, path in sorted(outbox_paths.items()):
|
for name, path in sorted(outbox_paths.items()):
|
||||||
print(f"outbox: {name} -> {path}")
|
print(f"outbox: {name} -> {path}")
|
||||||
|
emit(
|
||||||
|
notifiers,
|
||||||
|
Notification(
|
||||||
|
event="run.completed",
|
||||||
|
summary=f"run {run_label}: {outcome_kind}",
|
||||||
|
fields={
|
||||||
|
"validator_decision": result.validator_decision,
|
||||||
|
"checker_decision": result.checker_decision,
|
||||||
|
"attempts": result.attempts,
|
||||||
|
"outcome": outcome_kind,
|
||||||
|
},
|
||||||
|
),
|
||||||
|
)
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -294,8 +331,18 @@ def execute_dry_run(
|
||||||
return 0
|
return 0
|
||||||
|
|
||||||
|
|
||||||
def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None = None) -> int:
|
def main(
|
||||||
"""The thin CLI: contracts fail-fast (§10) → compose (§5) → execute (§3, §8)."""
|
argv: list[str] | None = None,
|
||||||
|
*,
|
||||||
|
client_factory: ClientFactory | None = None,
|
||||||
|
notifier_transport: Transport | None = None,
|
||||||
|
) -> int:
|
||||||
|
"""The thin CLI: contracts fail-fast (§10) → compose (§5) → execute (§3, §8).
|
||||||
|
|
||||||
|
``notifier_transport`` is the injected webhook seam (K10): ``None`` uses the
|
||||||
|
real ``default_webhook_transport`` on the CLI path; the suite injects a
|
||||||
|
canned transport so no socket is opened.
|
||||||
|
"""
|
||||||
parser = argparse.ArgumentParser(
|
parser = argparse.ArgumentParser(
|
||||||
description="Run one project through the loop (merge inbox → seed → fold → run)."
|
description="Run one project through the loop (merge inbox → seed → fold → run)."
|
||||||
)
|
)
|
||||||
|
|
@ -316,6 +363,7 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
|
||||||
help="build all, capture run-config + preflight to the outbox, STOP before "
|
help="build all, capture run-config + preflight to the outbox, STOP before "
|
||||||
"the first model call (K8 live-run drill; no spend, no model call).",
|
"the first model call (K8 live-run drill; no spend, no model call).",
|
||||||
)
|
)
|
||||||
|
add_notify_args(parser)
|
||||||
args = parser.parse_args(argv)
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
# fail-fast (§10 spirit): a run persisted to the outbox MUST carry an explicit
|
# fail-fast (§10 spirit): a run persisted to the outbox MUST carry an explicit
|
||||||
|
|
@ -331,6 +379,17 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
|
||||||
elif args.outbox is not None and not (args.run_id or "").strip():
|
elif args.outbox is not None and not (args.run_id or "").strip():
|
||||||
parser.error("--outbox requires --run-id (no wall-clock default)")
|
parser.error("--outbox requires --run-id (no wall-clock default)")
|
||||||
|
|
||||||
|
# K10 (§8): build the notify sinks BEFORE any spend — a --notify-webhook
|
||||||
|
# without --allow-webhook-egress is refused fail-fast here, so no run rides
|
||||||
|
# on a misconfigured egress. The opt-in is a run argument, never a config
|
||||||
|
# field (the config cannot grant itself network access).
|
||||||
|
try:
|
||||||
|
notifiers = build_notifiers(
|
||||||
|
notify_config_from_args(args), webhook_transport=notifier_transport
|
||||||
|
)
|
||||||
|
except EgressNotPermitted as exc:
|
||||||
|
parser.error(str(exc))
|
||||||
|
|
||||||
# §10: ALL startup contracts schema-validated BEFORE any model client exists.
|
# §10: ALL startup contracts schema-validated BEFORE any model client exists.
|
||||||
contracts = load_contracts(
|
contracts = load_contracts(
|
||||||
data_source={"docs_dir": str(args.bundle), "top_k": args.top_k},
|
data_source={"docs_dir": str(args.bundle), "top_k": args.top_k},
|
||||||
|
|
@ -392,6 +451,7 @@ def main(argv: list[str] | None = None, *, client_factory: ClientFactory | None
|
||||||
max_attempts=args.max_attempts,
|
max_attempts=args.max_attempts,
|
||||||
outbox_dir=args.outbox,
|
outbox_dir=args.outbox,
|
||||||
run_id=args.run_id,
|
run_id=args.run_id,
|
||||||
|
notifiers=notifiers,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
257
tests/test_notify_loadbearing.py
Normal file
257
tests/test_notify_loadbearing.py
Normal file
|
|
@ -0,0 +1,257 @@
|
||||||
|
"""Notification delivery — LOAD-BEARING (S5.2-analog; §8; §11; K10).
|
||||||
|
|
||||||
|
The seam this file keeps alive: a run/HITL pass can DELIVER a notification
|
||||||
|
(console, file, or webhook) WITHOUT breaking the no-silent-egress invariant. A
|
||||||
|
webhook — the one transport that leaves the machine — fires ONLY behind an
|
||||||
|
explicit per-run opt-in flag (mirroring ingest-spec §8: "the flag is a run
|
||||||
|
argument, never a config field — the config cannot grant itself network
|
||||||
|
access"). The transport is INJECTED: the suite passes a canned transport, so no
|
||||||
|
socket is ever opened; the real transport lives behind one clearly-marked seam
|
||||||
|
function that the suite never calls.
|
||||||
|
|
||||||
|
Three detached seams proven RED here:
|
||||||
|
|
||||||
|
* Opt-in gate: build a webhook notifier WITHOUT the opt-in flag → it MUST refuse
|
||||||
|
fail-fast (``EgressNotPermitted``) and its transport MUST stay unfired. Detach
|
||||||
|
point: drop the ``if not egress_opt_in: raise`` guard → the webhook constructs
|
||||||
|
and its transport fires on ``notify`` → RED.
|
||||||
|
* Payload structure: the canned transport receives the STRUCTURED event
|
||||||
|
(``event`` / ``summary`` / ``fields`` as deterministic JSON), not a prose blob.
|
||||||
|
Detach point: change the encoded shape → RED.
|
||||||
|
* No socket outside the seam: an AST grep-guard proves ``notify.py`` carries no
|
||||||
|
network import ANYWHERE except inside ``default_webhook_transport`` (the one
|
||||||
|
injectable seam). Detach point: hoist the ``urllib`` import to module scope (or
|
||||||
|
add any ``socket``/``httpx`` path elsewhere) → RED.
|
||||||
|
|
||||||
|
Key assumption (stack-local, no shared spec): the payload shape is this stack's
|
||||||
|
own — divergence from the MAF sibling is accepted and documented in the module
|
||||||
|
docstring. Pinned here by asserting the exact structured payload.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import ast
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.notify import (
|
||||||
|
ConsoleNotifier,
|
||||||
|
EgressNotPermitted,
|
||||||
|
FileNotifier,
|
||||||
|
Notification,
|
||||||
|
NotifyConfig,
|
||||||
|
WebhookNotifier,
|
||||||
|
build_notifiers,
|
||||||
|
emit,
|
||||||
|
notify_config_from_args,
|
||||||
|
)
|
||||||
|
|
||||||
|
SRC_PKG = Path(__file__).resolve().parents[1] / "src" / "portfolio_optimiser_claude"
|
||||||
|
_NETWORK_MODULES = {"socket", "urllib", "http", "https", "requests", "httpx", "anthropic", "ssl"}
|
||||||
|
_SEAM_FUNC = "default_webhook_transport"
|
||||||
|
|
||||||
|
|
||||||
|
class _CannedTransport:
|
||||||
|
"""A canned webhook transport — records calls, opens NO socket (§11 suite)."""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.calls: list[tuple[str, bytes]] = []
|
||||||
|
|
||||||
|
def send(self, url: str, payload: bytes) -> None:
|
||||||
|
self.calls.append((url, payload))
|
||||||
|
|
||||||
|
|
||||||
|
def _event() -> Notification:
|
||||||
|
return Notification(
|
||||||
|
event="run.completed",
|
||||||
|
summary="run r-001: validated",
|
||||||
|
fields={"run_id": "r-001", "decision": "validated", "attempts": 1},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# --- the opt-in gate (LOAD-BEARING: webhook egress is opt-in per run) ------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestWebhookOptInGate:
|
||||||
|
"""§8 mirror: the webhook refuses fail-fast WITHOUT the per-run opt-in flag."""
|
||||||
|
|
||||||
|
def test_webhook_without_optin_refuses_and_never_fires(self) -> None:
|
||||||
|
# LOAD-BEARING (the opt-in seam). Detach point: drop the guard in
|
||||||
|
# WebhookNotifier.__init__ → construction succeeds → transport fires on
|
||||||
|
# notify → this RED. Restore from the implemented copy, never git checkout.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
with pytest.raises(EgressNotPermitted):
|
||||||
|
WebhookNotifier("https://hooks.example/x", canned.send, egress_opt_in=False)
|
||||||
|
assert canned.calls == [] # nothing left the machine
|
||||||
|
|
||||||
|
def test_build_notifiers_refuses_webhook_without_optin(self) -> None:
|
||||||
|
# The run-argument seam: the CLI flag threads the opt-in into build; a
|
||||||
|
# webhook_url without the flag is refused BEFORE any transport exists.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
with pytest.raises(EgressNotPermitted):
|
||||||
|
build_notifiers(
|
||||||
|
NotifyConfig(webhook_url="https://hooks.example/x", webhook_egress_opt_in=False),
|
||||||
|
webhook_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert canned.calls == []
|
||||||
|
|
||||||
|
def test_optin_webhook_constructs_and_fires(self) -> None:
|
||||||
|
canned = _CannedTransport()
|
||||||
|
notifier = WebhookNotifier("https://hooks.example/x", canned.send, egress_opt_in=True)
|
||||||
|
notifier.notify(_event())
|
||||||
|
assert len(canned.calls) == 1
|
||||||
|
|
||||||
|
|
||||||
|
# --- the payload structure (LOAD-BEARING: structured, deterministic) -------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestPayloadStructure:
|
||||||
|
"""The canned transport receives the structured event, not a prose blob."""
|
||||||
|
|
||||||
|
def test_canned_transport_receives_structured_payload(self) -> None:
|
||||||
|
# LOAD-BEARING (payload shape, stack-local). Detach point: change the
|
||||||
|
# encoded shape (e.g. drop ``fields`` or emit prose) → RED.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
notifiers = build_notifiers(
|
||||||
|
NotifyConfig(webhook_url="https://hooks.example/x", webhook_egress_opt_in=True),
|
||||||
|
webhook_transport=canned.send,
|
||||||
|
)
|
||||||
|
emit(notifiers, _event())
|
||||||
|
assert len(canned.calls) == 1
|
||||||
|
url, payload = canned.calls[0]
|
||||||
|
assert url == "https://hooks.example/x"
|
||||||
|
assert json.loads(payload) == {
|
||||||
|
"event": "run.completed",
|
||||||
|
"summary": "run r-001: validated",
|
||||||
|
"fields": {"run_id": "r-001", "decision": "validated", "attempts": 1},
|
||||||
|
}
|
||||||
|
|
||||||
|
def test_payload_is_byte_deterministic(self) -> None:
|
||||||
|
# Same event → identical bytes (sorted keys) — a notification is not a
|
||||||
|
# source of run-to-run drift.
|
||||||
|
a, b = _CannedTransport(), _CannedTransport()
|
||||||
|
WebhookNotifier("https://x", a.send, egress_opt_in=True).notify(_event())
|
||||||
|
WebhookNotifier("https://x", b.send, egress_opt_in=True).notify(_event())
|
||||||
|
assert a.calls[0][1] == b.calls[0][1]
|
||||||
|
|
||||||
|
|
||||||
|
# --- no socket outside the injectable seam (LOAD-BEARING: AST grep-guard) --------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _import_nodes(tree: ast.AST) -> list[ast.stmt]:
|
||||||
|
return [n for n in ast.walk(tree) if isinstance(n, (ast.Import, ast.ImportFrom))]
|
||||||
|
|
||||||
|
|
||||||
|
def _modules_of(node: ast.stmt) -> set[str]:
|
||||||
|
if isinstance(node, ast.Import):
|
||||||
|
return {alias.name.split(".")[0] for alias in node.names}
|
||||||
|
if isinstance(node, ast.ImportFrom) and node.module:
|
||||||
|
return {node.module.split(".")[0]}
|
||||||
|
return set()
|
||||||
|
|
||||||
|
|
||||||
|
class TestNoSocketOutsideSeam:
|
||||||
|
"""LOAD-BEARING (§11): the ONLY network path in notify.py is the seam function."""
|
||||||
|
|
||||||
|
def test_no_network_import_outside_the_seam(self) -> None:
|
||||||
|
# Detach point: hoist the urllib import out of default_webhook_transport
|
||||||
|
# to module scope, or add any socket/httpx path elsewhere → an import
|
||||||
|
# node OUTSIDE the seam references a network module → RED.
|
||||||
|
tree = ast.parse((SRC_PKG / "notify.py").read_text("utf-8"))
|
||||||
|
seam = next(
|
||||||
|
n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == _SEAM_FUNC
|
||||||
|
)
|
||||||
|
seam_node_ids = {id(n) for n in _import_nodes(seam)}
|
||||||
|
outside: set[str] = set()
|
||||||
|
for node in _import_nodes(tree):
|
||||||
|
if id(node) not in seam_node_ids:
|
||||||
|
outside |= _modules_of(node) & _NETWORK_MODULES
|
||||||
|
assert outside == set()
|
||||||
|
|
||||||
|
def test_the_seam_is_real_not_hollow(self) -> None:
|
||||||
|
# Green-but-dead guard: the seam MUST actually carry the network import
|
||||||
|
# it isolates — deleting it (so the guard passes vacuously) is RED here.
|
||||||
|
tree = ast.parse((SRC_PKG / "notify.py").read_text("utf-8"))
|
||||||
|
seam = next(
|
||||||
|
n for n in ast.walk(tree) if isinstance(n, ast.FunctionDef) and n.name == _SEAM_FUNC
|
||||||
|
)
|
||||||
|
seam_modules: set[str] = set()
|
||||||
|
for node in _import_nodes(seam):
|
||||||
|
seam_modules |= _modules_of(node)
|
||||||
|
assert "urllib" in seam_modules
|
||||||
|
|
||||||
|
|
||||||
|
# --- the local notifiers (console, file — no network) ----------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestLocalNotifiers:
|
||||||
|
"""console and file notifiers deliver locally — never a socket."""
|
||||||
|
|
||||||
|
def test_console_notifier_prints_event_and_summary(
|
||||||
|
self, capsys: pytest.CaptureFixture[str]
|
||||||
|
) -> None:
|
||||||
|
ConsoleNotifier().notify(_event())
|
||||||
|
out = capsys.readouterr().out
|
||||||
|
assert "run.completed" in out
|
||||||
|
assert "run r-001: validated" in out
|
||||||
|
|
||||||
|
def test_file_notifier_writes_deterministic_json(self, tmp_path: Path) -> None:
|
||||||
|
FileNotifier(tmp_path).notify(_event())
|
||||||
|
written = tmp_path / "run.completed.json"
|
||||||
|
assert written.is_file()
|
||||||
|
body = json.loads(written.read_text("utf-8"))
|
||||||
|
assert body["event"] == "run.completed"
|
||||||
|
assert body["fields"]["run_id"] == "r-001"
|
||||||
|
# deterministic house JSON: sorted keys, trailing newline
|
||||||
|
assert written.read_text("utf-8").endswith("}\n")
|
||||||
|
|
||||||
|
def test_build_notifiers_composes_the_configured_set(self, tmp_path: Path) -> None:
|
||||||
|
notifiers = build_notifiers(NotifyConfig(console=True, file_dir=tmp_path))
|
||||||
|
assert [type(n).__name__ for n in notifiers] == ["ConsoleNotifier", "FileNotifier"]
|
||||||
|
|
||||||
|
def test_no_config_builds_nothing(self) -> None:
|
||||||
|
assert build_notifiers(NotifyConfig()) == []
|
||||||
|
|
||||||
|
|
||||||
|
# --- the CLI-arg mapping (shared by run.py and hitl.py) --------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestNotifyConfigFromArgs:
|
||||||
|
"""notify_config_from_args maps the shared CLI flags into a NotifyConfig."""
|
||||||
|
|
||||||
|
def test_flags_map_into_config(self, tmp_path: Path) -> None:
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.notify import add_notify_args
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
add_notify_args(parser)
|
||||||
|
args = parser.parse_args(
|
||||||
|
[
|
||||||
|
"--notify-console",
|
||||||
|
"--notify-file",
|
||||||
|
str(tmp_path),
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://x",
|
||||||
|
"--allow-webhook-egress",
|
||||||
|
]
|
||||||
|
)
|
||||||
|
cfg = notify_config_from_args(args)
|
||||||
|
assert cfg == NotifyConfig(
|
||||||
|
console=True,
|
||||||
|
file_dir=tmp_path,
|
||||||
|
webhook_url="https://x",
|
||||||
|
webhook_egress_opt_in=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
def test_defaults_are_all_off(self) -> None:
|
||||||
|
import argparse
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.notify import add_notify_args
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
add_notify_args(parser)
|
||||||
|
cfg = notify_config_from_args(parser.parse_args([]))
|
||||||
|
assert cfg == NotifyConfig()
|
||||||
245
tests/test_notify_seam_loadbearing.py
Normal file
245
tests/test_notify_seam_loadbearing.py
Normal file
|
|
@ -0,0 +1,245 @@
|
||||||
|
"""Notify seam in run.py / hitl.py — LOAD-BEARING (S5.2-analog; §8; §11; K10).
|
||||||
|
|
||||||
|
The seam this file keeps alive: BOTH deliverable entrances (``run.py`` and the
|
||||||
|
read-only ``hitl.py``) can emit a notification behind the SAME opt-in-gated CLI
|
||||||
|
seam, and a webhook without the explicit per-run opt-in flag refuses fail-fast
|
||||||
|
BEFORE any spend (run) / BEFORE any transport fires (hitl). No socket is opened
|
||||||
|
in the suite — the transport is injected (canned).
|
||||||
|
|
||||||
|
Detach proofs:
|
||||||
|
|
||||||
|
* Opt-in threaded, not hardcoded: ``--notify-webhook URL`` WITHOUT
|
||||||
|
``--allow-webhook-egress`` → refuse fail-fast; the scripted client is never
|
||||||
|
constructed (run: no spend) and the canned transport stays empty. Detach
|
||||||
|
point: hardcode the opt-in true (or drop the gate) → the webhook builds and
|
||||||
|
fires → RED.
|
||||||
|
* Emit wired: an opted-in run/hitl pass delivers the STRUCTURED event to the
|
||||||
|
injected transport. Detach point: drop the ``emit`` call → the canned
|
||||||
|
transport gets nothing → RED.
|
||||||
|
* hitl stays read-only: notification does NOT write any of the three layers —
|
||||||
|
a before/after byte snapshot of outbox+inbox is unchanged even while emitting.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from _scripted import ScriptedClient, reply
|
||||||
|
|
||||||
|
from portfolio_optimiser_claude.contracts import Contracts
|
||||||
|
from portfolio_optimiser_claude.hitl import main as hitl_main
|
||||||
|
from portfolio_optimiser_claude.ir import AffectedItem, SavingsProposal, load_validator_input
|
||||||
|
from portfolio_optimiser_claude.loop import ModelClient, RunResult
|
||||||
|
from portfolio_optimiser_claude.outbox import persist_outbox
|
||||||
|
from portfolio_optimiser_claude.provenance import Citation, Provenance
|
||||||
|
from portfolio_optimiser_claude.run import main as run_main
|
||||||
|
from portfolio_optimiser_claude.validator import ValidatedProposal
|
||||||
|
|
||||||
|
BUNDLE = Path(__file__).resolve().parents[1] / "shared" / "examples" / "bygg-energi-mikro"
|
||||||
|
|
||||||
|
|
||||||
|
class _CannedTransport:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
self.calls: list[tuple[str, bytes]] = []
|
||||||
|
|
||||||
|
def send(self, url: str, payload: bytes) -> None:
|
||||||
|
self.calls.append((url, payload))
|
||||||
|
|
||||||
|
|
||||||
|
def _scripted_factory(replies: list[object]) -> tuple[object, list[ScriptedClient]]:
|
||||||
|
created: list[ScriptedClient] = []
|
||||||
|
|
||||||
|
def factory(contracts: Contracts, max_budget_usd_per_call: float) -> ModelClient:
|
||||||
|
client = ScriptedClient(replies=list(replies)) # type: ignore[arg-type]
|
||||||
|
created.append(client)
|
||||||
|
return client
|
||||||
|
|
||||||
|
return factory, created
|
||||||
|
|
||||||
|
|
||||||
|
def _happy_replies() -> list[object]:
|
||||||
|
return [
|
||||||
|
reply("debate reasoning"),
|
||||||
|
reply("VERDICT: APPROVE"),
|
||||||
|
reply(json.dumps(load_validator_input(BUNDLE).model_dump())),
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
# --- run.py notify seam ----------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class TestRunNotifySeam:
|
||||||
|
def test_webhook_without_optin_refuses_before_any_spend(self, tmp_path: Path) -> None:
|
||||||
|
# LOAD-BEARING (opt-in threaded). Detach: hardcode opt-in true → the
|
||||||
|
# webhook builds and the run proceeds → this RED. The client is never
|
||||||
|
# constructed, so no model call (and no spend) rides on the refusal.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
factory, created = _scripted_factory(_happy_replies())
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
run_main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--out",
|
||||||
|
str(tmp_path / "out"),
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://hooks.example/x",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
notifier_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert created == [] # refused before the client was constructed → no spend
|
||||||
|
assert canned.calls == []
|
||||||
|
|
||||||
|
def test_completion_emits_structured_event(self, tmp_path: Path) -> None:
|
||||||
|
# LOAD-BEARING (emit wired). Detach: drop the completion emit → the
|
||||||
|
# canned transport gets nothing → RED.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
factory, _ = _scripted_factory(_happy_replies())
|
||||||
|
code = run_main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--out",
|
||||||
|
str(tmp_path / "out"),
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://hooks.example/x",
|
||||||
|
"--allow-webhook-egress",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
notifier_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
assert len(canned.calls) == 1
|
||||||
|
body = json.loads(canned.calls[0][1])
|
||||||
|
assert body["event"] == "run.completed"
|
||||||
|
assert body["fields"]["validator_decision"] == "validated"
|
||||||
|
|
||||||
|
def test_budget_stop_emits_stopped_event(self, tmp_path: Path) -> None:
|
||||||
|
# A budget stop is a run outcome, not an absence of one — it notifies too.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
factory, _ = _scripted_factory([reply("debate reasoning", usage_tokens=10)])
|
||||||
|
code = run_main(
|
||||||
|
[
|
||||||
|
"--bundle",
|
||||||
|
str(BUNDLE),
|
||||||
|
"--out",
|
||||||
|
str(tmp_path / "out"),
|
||||||
|
"--max-tokens",
|
||||||
|
"5",
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://hooks.example/x",
|
||||||
|
"--allow-webhook-egress",
|
||||||
|
],
|
||||||
|
client_factory=factory,
|
||||||
|
notifier_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert code == 3
|
||||||
|
assert len(canned.calls) == 1
|
||||||
|
body = json.loads(canned.calls[0][1])
|
||||||
|
assert body["event"] == "run.stopped"
|
||||||
|
assert body["fields"]["kind"] == "tokens"
|
||||||
|
|
||||||
|
|
||||||
|
# --- hitl.py notify seam (read-only preserved) -----------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _proposal() -> SavingsProposal:
|
||||||
|
return SavingsProposal(
|
||||||
|
project_id="bygg-kontor-nord",
|
||||||
|
measure="LED-retrofit",
|
||||||
|
affected_items=[AffectedItem(code="EL-01", quantity=100, unit_cost=250.0)],
|
||||||
|
claimed_saving_nok=20000.0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _run(proposal: SavingsProposal) -> RunResult:
|
||||||
|
return RunResult(
|
||||||
|
outcome=ValidatedProposal(
|
||||||
|
validates=True,
|
||||||
|
claimed_saving_nok=20000.0,
|
||||||
|
nominal_feasible=25000.0,
|
||||||
|
p10=18000.0,
|
||||||
|
p50=22000.0,
|
||||||
|
p90=27000.0,
|
||||||
|
),
|
||||||
|
validator_decision="validated",
|
||||||
|
checker_decision="approve",
|
||||||
|
attempts=1,
|
||||||
|
proposal=proposal,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _provenance() -> Provenance:
|
||||||
|
return Provenance(
|
||||||
|
citations=[Citation(file="index.md", span="chars 0-5", snippet="Bygg-")],
|
||||||
|
model="claude-haiku-4-5-20251001",
|
||||||
|
role="proposer",
|
||||||
|
validator_decision="validated",
|
||||||
|
tokens_used=1234,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _snapshot(root: Path) -> dict[str, bytes]:
|
||||||
|
if not root.exists():
|
||||||
|
return {}
|
||||||
|
return {
|
||||||
|
str(p.relative_to(root)): p.read_bytes() for p in sorted(root.rglob("*")) if p.is_file()
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _seed_pending(tmp_path: Path) -> tuple[Path, Path]:
|
||||||
|
outbox, inbox = tmp_path / "outbox", tmp_path / "inbox"
|
||||||
|
persist_outbox(outbox, run=_run(_proposal()), provenance=_provenance(), run_id="r-001")
|
||||||
|
return outbox, inbox
|
||||||
|
|
||||||
|
|
||||||
|
class TestHitlNotifySeam:
|
||||||
|
def test_pending_webhook_without_optin_refuses(self, tmp_path: Path) -> None:
|
||||||
|
canned = _CannedTransport()
|
||||||
|
outbox, inbox = _seed_pending(tmp_path)
|
||||||
|
before_out = _snapshot(outbox)
|
||||||
|
with pytest.raises(SystemExit):
|
||||||
|
hitl_main(
|
||||||
|
[
|
||||||
|
"pending",
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--inbox",
|
||||||
|
str(inbox),
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://hooks.example/x",
|
||||||
|
],
|
||||||
|
notifier_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert canned.calls == []
|
||||||
|
assert _snapshot(outbox) == before_out # read-only preserved
|
||||||
|
|
||||||
|
def test_pending_emits_count_and_stays_read_only(self, tmp_path: Path) -> None:
|
||||||
|
# LOAD-BEARING (emit wired + read-only). Detach: drop the emit → canned
|
||||||
|
# empty → RED. The three layers are never written even while notifying.
|
||||||
|
canned = _CannedTransport()
|
||||||
|
outbox, inbox = _seed_pending(tmp_path)
|
||||||
|
before_out, before_in = _snapshot(outbox), _snapshot(inbox)
|
||||||
|
code = hitl_main(
|
||||||
|
[
|
||||||
|
"pending",
|
||||||
|
"--outbox",
|
||||||
|
str(outbox),
|
||||||
|
"--inbox",
|
||||||
|
str(inbox),
|
||||||
|
"--notify-webhook",
|
||||||
|
"https://hooks.example/x",
|
||||||
|
"--allow-webhook-egress",
|
||||||
|
],
|
||||||
|
notifier_transport=canned.send,
|
||||||
|
)
|
||||||
|
assert code == 0
|
||||||
|
assert len(canned.calls) == 1
|
||||||
|
body = json.loads(canned.calls[0][1])
|
||||||
|
assert body["event"] == "hitl.pending"
|
||||||
|
assert body["fields"]["pending"] == 1
|
||||||
|
assert _snapshot(outbox) == before_out
|
||||||
|
assert _snapshot(inbox) == before_in
|
||||||
Loading…
Add table
Add a link
Reference in a new issue