feat(portfolio): stamp the producing SDK build in provenance (wiki-advisory F1) [skip-docs]

The advisory finding: provenance.py/artifacts.py stamped no SDK version, while
the SDK's total_cost_usd is a client-side ESTIMATE computed against a price
table frozen when the SDK was built. An untraceable estimate is a figure nobody
can check later, so the run now records which build produced it.

Provenance gains sdk_version: str | None. The value comes from the PRODUCING
CLIENT — getattr(client, "sdk_version", None) — exactly as model and cost_usd
already do, never from importlib.metadata at stamp time. That distinction is
the seam: a run driven by the scripted stand-in used no SDK at all, and
stamping the installed version there would attribute a build to a run that
never touched it (§1). SdkModelClient reads the installed build once from
package metadata (offline: no key, no network); every other client reports
null. A blank string is refused by the schema — null is the one way to say
"not produced by the SDK".

Scope note: this traceability covers OUR run cost only. The savings the
framework recommends are settled by the deterministic validator against the
golden suite, and no SDK estimate touches them.

Two seams, both detach-proven RED:
- make the stamp read importlib.metadata instead of the client → a scripted run
  claims a build it never used → red
- back-fill runs/s10/provenance.json → red

That second guard is the point of the change as much as the first. runs/s10/ is
the byte-frozen record of the ONE live run (2026-07-03), executed before this
field existed; the suite reads it nowhere else, so nothing would have caught a
retro-stamp. Adding a build id to it now would be a guess presented as
provenance. It stays without one, and the README says why.

run_s10.py is deliberately untouched (byte-frozen fasit script), and the field
defaults to None, so every existing caller and artifact shape is unchanged.

603 passed · ruff clean · mypy strict clean · runs/s10/ byte-identical.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MQu2xxwedckjU56byu1aUG
This commit is contained in:
Kjell Tore Guttormsen 2026-07-25 06:57:30 +02:00
commit bf87776bb3
6 changed files with 193 additions and 3 deletions

View file

@ -39,8 +39,14 @@ reverse-engineering the MAF sibling.
test. test.
- **Knowledge-base recipe** — the documented team process for building the OKF bundles the - **Knowledge-base recipe** — the documented team process for building the OKF bundles the
framework reads (`docs/oppskrift-kunnskapsbase.md`), with an honest 12 week expectation. framework reads (`docs/oppskrift-kunnskapsbase.md`), with an honest 12 week expectation.
- **Traceable run cost** — the provenance stamp records which SDK build produced the run,
read from the producing client rather than the environment, so the SDK's cost estimate can
be traced to the price table that computed it. A run not produced by the SDK reports `null`
instead of borrowing the installed version.
- **The programme's one live model run** (S10) — executed and validated at a documented - **The programme's one live model run** (S10) — executed and validated at a documented
$0.127514, its four artifacts committed as fixed reference output under `runs/s10/`. $0.127514, its four artifacts committed as fixed reference output under `runs/s10/`. That
record is never edited after the fact: it predates the `sdk_version` field and is left
without one rather than back-filled with a guess.
- **Load-bearing tests** — every seam is proven by a test that goes red when the seam is - **Load-bearing tests** — every seam is proven by a test that goes red when the seam is
detached; the whole suite runs offline, with no API key and no network. detached; the whole suite runs offline, with no API key and no network.

View file

@ -13,7 +13,7 @@ human-in-the-loop, and the system learns from the verdicts.
> **Status:** the D7 build (S5S10) is complete, and the deterministic **ingest layer** > **Status:** the D7 build (S5S10) is complete, and the deterministic **ingest layer**
> (CSV and SQL source types) has since been added in front of the loop. The deterministic > (CSV and SQL source types) has since been added in front of the loop. The deterministic
> backbone, the agentic loop, the learning loop, the value layer, and the ingest connectors > backbone, the agentic loop, the learning loop, the value layer, and the ingest connectors
> are wired seam by seam, each proven by load-bearing tests (597 at the time of writing, all > are wired seam by seam, each proven by load-bearing tests (603 at the time of writing, all
> running offline without an API key — `uv run pytest` is the source of truth). The > running offline without an API key — `uv run pytest` is the source of truth). The
> programme's single budgeted **live model run has been executed and validated** — its > programme's single budgeted **live model run has been executed and validated** — its
> artifacts are committed under [`runs/s10/`](runs/s10/) (see below). > artifacts are committed under [`runs/s10/`](runs/s10/) (see below).
@ -44,7 +44,12 @@ offline. Module by module:
- `validator.py` — the deterministic validator; blocking, and frozen by the shared golden - `validator.py` — the deterministic validator; blocking, and frozen by the shared golden
suite (§7.2), which is the only fasit it answers to. suite (§7.2), which is the only fasit it answers to.
- `provenance.py` — the first-class provenance stamp (§9); authoritative data, not - `provenance.py` — the first-class provenance stamp (§9); authoritative data, not
after-the-fact logging. after-the-fact logging. It also records **which SDK build produced the run**, taken from
the producing client and never from the environment: the SDK's reported cost is a
client-side estimate against a price table frozen at that build, so the figure is only
checkable later if the run says which build computed it. A run driven by anything else
(the scripted stand-in, an injected client) reports `null` rather than borrowing the
installed version it never used.
- `contracts.py` — fail-fast startup contracts (§10): stop criteria and budget caps are - `contracts.py` — fail-fast startup contracts (§10): stop criteria and budget caps are
required at startup, and the model map (`data/model_map.json`, role → Claude model id required at startup, and the model map (`data/model_map.json`, role → Claude model id
per backend profile) is validated before anything runs. per backend profile) is validated before anything runs.
@ -273,6 +278,9 @@ the run entrance produces the report on both outcomes while leaving the run's ve
before a single model call — red the moment the goal check is unwired — the portfolio config's before a single model call — red the moment the goal check is unwired — the portfolio config's
projects genuinely run through the CLI, and this README's documented flags are checked against projects genuinely run through the CLI, and this README's documented flags are checked against
the actual `--help` output), the actual `--help` output),
`test_provenance_sdk_version_loadbearing.py` (a run not produced by the SDK stamps no build
rather than borrowing the installed one — red the moment the stamp reads the environment
instead of the producing client — and the committed S10 record stays un-back-filled),
and `test_sdk_isolation.py` (local config cannot capture the checker). and `test_sdk_isolation.py` (local config cannot capture the checker).
## The ingest layer — CSV and SQL, in front of the loop ## The ingest layer — CSV and SQL, in front of the loop
@ -321,6 +329,10 @@ whole programme), executed 2026-07-03 against the micro bundle
bundle's p10p90 band of 68.5k121k — and validates. bundle's p10p90 band of 68.5k121k — and validates.
- All four artifacts are committed as fixed reference output in [`runs/s10/`](runs/s10/): - All four artifacts are committed as fixed reference output in [`runs/s10/`](runs/s10/):
`proposal.json`, `provenance.json` (with §9 citations), `run_result.json`, `usage.json`. `proposal.json`, `provenance.json` (with §9 citations), `run_result.json`, `usage.json`.
They are the record of that run as it happened and are never edited afterwards — the
provenance stamp's later `sdk_version` field is absent there because the run predates it,
and back-filling a build id would be a guess presented as provenance. A load-bearing test
keeps that record frozen.
Honesty rule (§1): everything else in the repo is deterministic and offline; nothing here Honesty rule (§1): everything else in the repo is deterministic and offline; nothing here
claims more live behaviour than that one documented run. claims more live behaviour than that one documented run.

View file

@ -6,6 +6,16 @@ At least one citation into the source documents; the producing ``model`` and
``validator_decision``, which mirrors the DETERMINISTIC VALIDATOR only stamped ``validator_decision``, which mirrors the DETERMINISTIC VALIDATOR only stamped
from the validator's outcome BEFORE any checker override, so a checker-gated from the validator's outcome BEFORE any checker override, so a checker-gated
proposal whose numbers passed is never mislabelled as validator-rejected (§9). proposal whose numbers passed is never mislabelled as validator-rejected (§9).
``sdk_version`` records WHICH SDK build produced the run, and comes from the
producing client never from the environment. The SDK's reported cost is a
client-side estimate against a price table frozen at SDK build time, so the
figure is only traceable if the run says which build computed it. A run driven
by anything other than the SDK client (the scripted stand-in, an injected
client) reports ``None``: attributing an installed build to a run that never
used it would be a fabricated claim (§1). Note that this reasoning covers OUR
run cost only the savings the framework recommends are settled by the
deterministic validator against the golden suite, never by an SDK estimate.
""" """
from __future__ import annotations from __future__ import annotations
@ -33,6 +43,9 @@ class Provenance(BaseModel):
role: str = Field(min_length=1) role: str = Field(min_length=1)
validator_decision: Literal["validated", "rejected"] validator_decision: Literal["validated", "rejected"]
tokens_used: int = Field(ge=0) tokens_used: int = Field(ge=0)
# None = not produced by the SDK client. A blank string is refused: it would
# read as "no SDK" while occupying the field, and null is the one way to say it.
sdk_version: str | None = Field(default=None, min_length=1)
def stamp_validator_decision( def stamp_validator_decision(

View file

@ -159,6 +159,15 @@ def _client_cost_usd(client: ModelClient) -> float | None:
return None if cost is None else round(float(cost), 6) return None if cost is None else round(float(cost), 6)
def _client_sdk_version(client: ModelClient) -> str | None:
# The build is read from the PRODUCING CLIENT, never from the environment:
# a run driven by the scripted stand-in used no SDK at all, and stamping
# the installed version there would attribute a build to a run that never
# touched it (§1). Same seam rule as the cost and the model id above.
version = getattr(client, "sdk_version", None)
return None if version is None else str(version)
def execute_run( def execute_run(
client: ModelClient, client: ModelClient,
composed: ComposedRunContext, composed: ComposedRunContext,
@ -225,6 +234,7 @@ def execute_run(
role=_PROPOSER_ROLE, role=_PROPOSER_ROLE,
validator_decision=result.validator_decision, validator_decision=result.validator_decision,
tokens_used=meter.tokens_used, tokens_used=meter.tokens_used,
sdk_version=_client_sdk_version(client),
) )
paths = persist_run_artifacts( paths = persist_run_artifacts(
out_dir, out_dir,

View file

@ -21,6 +21,7 @@ blocks + real model id) and a closing ``ResultMessage`` (provider-reported
from __future__ import annotations from __future__ import annotations
import asyncio import asyncio
import importlib.metadata
from typing import Any from typing import Any
from claude_agent_sdk import ( from claude_agent_sdk import (
@ -79,6 +80,13 @@ class SdkModelClient:
``total_cost_usd`` accumulates the provider-reported cost across calls so ``total_cost_usd`` accumulates the provider-reported cost across calls so
the run can log it (D6); ``last_model`` carries the REAL model id from the the run can log it (D6); ``last_model`` carries the REAL model id from the
latest reply for the §9 provenance stamp. latest reply for the §9 provenance stamp.
``sdk_version`` is the installed build, read once from package metadata
(offline: no key, no network). It exists so a run's provenance can say which
build produced it the reported cost is an estimate against a price table
frozen at that build, and an untraceable estimate is a figure nobody can
check later. Only THIS client carries the attribute, so a run driven by any
other client honestly reports no build at all (§1).
""" """
def __init__( def __init__(
@ -97,6 +105,7 @@ class SdkModelClient:
self._max_budget_usd_per_call = max_budget_usd_per_call self._max_budget_usd_per_call = max_budget_usd_per_call
self.total_cost_usd = 0.0 self.total_cost_usd = 0.0
self.last_model: str | None = None self.last_model: str | None = None
self.sdk_version: str = importlib.metadata.version("claude-agent-sdk")
def complete(self, prompt: str, *, role: str) -> ModelReply: def complete(self, prompt: str, *, role: str) -> ModelReply:
model_id = resolve_model(self._model_map, role, profile=self._profile) model_id = resolve_model(self._model_map, role, profile=self._profile)

View file

@ -0,0 +1,140 @@
"""SDK build in the provenance stamp — LOAD-BEARING (§1, §9, §11).
The seam this file keeps alive: a run's provenance records **which SDK build
produced it**, and it takes that from the PRODUCING CLIENT, never from the
environment. That distinction is the whole point. The SDK's reported
``total_cost_usd`` is a client-side estimate computed against a price table
frozen when the SDK was built, so a cost figure is only traceable if the run
says which build produced it. But a run driven by the scripted stand-in used no
SDK at all stamping the installed version there would attribute a build to a
run that never touched it, which is exactly the fabrication §1 forbids.
Detach proof: make the stamp read ``importlib.metadata`` instead of the client
a scripted run claims an SDK build it never used red.
Second guard: the S10 fasit predates this field and is a historical record of a
run executed 2026-07-03. Retro-stamping it would invent provenance for a run
whose SDK build we would be guessing. The fasit stays byte-frozen red the
moment someone back-fills it.
"""
from __future__ import annotations
import importlib.metadata
import json
from pathlib import Path
from typing import Callable
import pytest
from pydantic import ValidationError
from _scripted import ScriptedClient, reply
from portfolio_optimiser_claude.contracts import Contracts, ModelMapContract
from portfolio_optimiser_claude.ir import load_validator_input
from portfolio_optimiser_claude.loop import ModelClient
from portfolio_optimiser_claude.provenance import Citation, Provenance
from portfolio_optimiser_claude.run import main
from portfolio_optimiser_claude.sdk_client import SdkModelClient
REPO_ROOT = Path(__file__).resolve().parents[1]
BUNDLE = REPO_ROOT / "shared" / "examples" / "bygg-energi-mikro"
S10_PROVENANCE = REPO_ROOT / "runs" / "s10" / "provenance.json"
ClientFactory = Callable[[Contracts, float], ModelClient]
# A build id no installed package could ever report, so a test that sees it
# knows the value came from the CLIENT and from nowhere else.
FIXTURE_BUILD = "0.2.120-fixture-not-installed"
def _scripted_factory(*, sdk_version: str | None) -> tuple[ClientFactory, list[ScriptedClient]]:
created: list[ScriptedClient] = []
def factory(contracts: Contracts, max_budget_usd_per_call: float) -> ModelClient:
client = ScriptedClient(
replies=[
reply("debate reasoning"),
reply("VERDICT: APPROVE"),
reply(json.dumps(load_validator_input(BUNDLE).model_dump())),
]
)
if sdk_version is not None:
# Only a client that genuinely carries the attribute reports one.
client.sdk_version = sdk_version # type: ignore[attr-defined]
created.append(client)
return client
return factory, created
def _run_and_read_provenance(tmp_path: Path, *, sdk_version: str | None) -> dict[str, object]:
out = tmp_path / "out"
factory, _ = _scripted_factory(sdk_version=sdk_version)
code = main(["--bundle", str(BUNDLE), "--out", str(out)], client_factory=factory)
assert code == 0
payload: dict[str, object] = json.loads((out / "provenance.json").read_text(encoding="utf-8"))
return payload
class TestTheStampComesFromTheClient:
"""LOAD-BEARING (§11): the producing client reports the build, or nobody does."""
def test_a_run_not_produced_by_the_sdk_stamps_no_build(self, tmp_path: Path) -> None:
# Detach point: read importlib.metadata instead of the client → this
# scripted run claims the installed build it never used → RED.
payload = _run_and_read_provenance(tmp_path, sdk_version=None)
assert payload["sdk_version"] is None
assert payload["sdk_version"] != importlib.metadata.version("claude-agent-sdk")
def test_a_producing_client_has_its_build_stamped_verbatim(self, tmp_path: Path) -> None:
payload = _run_and_read_provenance(tmp_path, sdk_version=FIXTURE_BUILD)
assert payload["sdk_version"] == FIXTURE_BUILD
def test_the_sdk_client_reports_the_installed_build_offline(self) -> None:
# Constructing the real client needs no key and touches no network
# (the same premise test_sdk_isolation.py relies on), so the build id
# it exposes is readable in the offline suite.
client = SdkModelClient(
ModelMapContract(profiles={"anthropic": {"default": "claude-haiku-4-5-20251001"}})
)
assert client.sdk_version == importlib.metadata.version("claude-agent-sdk")
class TestTheStampStaysOptional:
"""The field is additive: every existing caller keeps validating unchanged."""
def test_provenance_validates_without_a_build(self) -> None:
stamp = Provenance(
citations=[Citation(file="index.md", span="chars 0-1", snippet="x")],
model="claude-haiku-4-5-20251001",
role="proposer",
validator_decision="validated",
tokens_used=10,
)
assert stamp.sdk_version is None
def test_a_blank_build_is_refused_rather_than_stored(self) -> None:
# An empty string would read as "no SDK" while occupying the field —
# null is the one way to say "not produced by the SDK" (§1).
with pytest.raises(ValidationError):
Provenance(
citations=[Citation(file="index.md", span="chars 0-1", snippet="x")],
model="m",
role="proposer",
validator_decision="validated",
tokens_used=10,
sdk_version="",
)
class TestTheFasitIsNotRetroStamped:
"""LOAD-BEARING (§1): the historical record is never back-filled."""
def test_the_s10_provenance_carries_no_sdk_build(self) -> None:
# runs/s10/ is the byte-frozen record of the ONE live run (2026-07-03),
# executed before this field existed. Adding a build id to it now would
# be a guess presented as provenance. RED if someone back-fills it.
payload = json.loads(S10_PROVENANCE.read_text(encoding="utf-8"))
assert "sdk_version" not in payload
assert payload["model"] == "claude-haiku-4-5-20251001"