The context sets, the packaged knowledge bases and the example bundles are
replaced by one fictitious example set about IT operations in an invented
organisation: three context sets (serverrom-2027, driftsavtale-2027 and the
two-base drift-og-avtale-2027), two synthetic knowledge bases under
src/portfolio_optimiser/data/kunnskapsbaser and two example bundles under
src/portfolio_optimiser/data/bundles. Numbers, codes and structural values in
tests and fixtures are kept; names, ids and wording change. Dated measurement
documents that only recorded runs on the replaced material are deleted.
Gate figures measured on the new set are not comparable with earlier ones.
The exclusion gate from the previous commit is green: 0 tracked files hit
outside the shared/ subtree.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The 18.09 re-measurement took row 2 to 3 of 3 GREEN on a tree this product had never
run in: four handwritten outcome.json, four handwritten <run_id>-coverage.json in an
outbox the forger named in those same files, and os.utime for the ordering. "Round 0
must be a named real run" was implemented as "a file with that name exists" — which
touch satisfies. The attack is committed as a red test in b769537.
Three bindings, chosen because each removes one of the forger's three moves:
1. The outbox is DERIVED, never declared. It is <rounds-dir>/<n>/outbox/, and an
outcome.json that names one is refused by name. A path a round file chooses is a
path it can point at a directory the same hand just wrote.
2. "The run exists" now means the run's own artefacts agree WITH EACH OTHER on content
the gate recomputes (verify_run). Every evaluated approach has the proposal/outcome
pair write_outbox actually persists; outcome_type IS the coverage status, reason IS
its detail, a validated row's figure IS the proposal's own claimed_saving_nok, the
provenance stamp follows the same branch, and verdict_id is RE-MINTED here from the
proposal's own IR with the product's one minting rule (A5) rather than read. A
not_evaluated approach wrote neither file, and an artefact naming an approach the
coverage omits belongs to another run. verdicts._features_from_ir is made public for
this: a second private copy in the gate could drift from the rule the run stamped
with, which would turn the binding into a coincidence.
3. mtime decides nothing. The run's time is the round's declared ran_at (ISO-8601 with
zone, required). An mtime is not evidence — it is a filesystem attribute one utime
call sets.
What this does NOT do is prove a run happened. Nothing in a directory can: the outbox
writers are byte-deterministic and carry no clock by contract. So row 2 states its own
limit on every run instead of leaving GREEN to imply it (Row.attests / RUN_ATTESTATION):
that a run was actually made, and when, is the operator's to confirm. The cost of a
forgery moves from touch to reproducing the product's own artefact set, minting rule
included.
Measured, in a scratch copy, never in the work tree — four new mutants in this class,
each one line, each felled by the whole suite:
M7 if "outbox" in data: -> if False: 1 failed (declares its own outbox)
M8 if verdict_id != minted: -> if False: 1 failed (key is not the IR's)
M9 if strays: -> if False: 1 failed (artefact of another run)
M10 ran_at -> coverage mtime 18 failed (incl. the utime test)
Control, same scratch copy, unmutated: 1993 passed, 10 skipped, 5 xfailed.
Work tree, re-run after git add: uv run pytest -q -> 1998 passed, 5 skipped, 5 xfailed.
Gate: uv run python -m portfolio_optimiser.evals.v1_gate -> exit 1, row 2 RED (0 of 3).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Walked from a fresh clone: `--embedder-config` was accepted in every mode
without `--semantic-retrieval` and then had no effect whatsoever. MEASURED,
not inferred — an injected embedder is consulted ZERO times with the flag
off and once with it on, because the only consumer is the HybridRanker that
flag builds; the default StructuralRetriever takes no embedder at all.
That is the silent-ignore this CLI's flag contract exists to prevent, and
the same ground on which `--semantic-retrieval` itself is already refused
when it cannot take effect.
REFUSED, not wired — the opposite call from `--scripted-replies` in
portfolio mode, and for a stated reason: there the seam already existed, so
refusing would have left a whole mode without an offline door. Here there
is nothing to wire to.
Mode-independent (both modes gate the embedder on the same flag) and placed
ABOVE the scripted door, mirroring the required-args hoist: a refused run
must not first print a banner claiming a scripted loop closed.
Five mutations against the WHOLE suite, all red, each isolating one seam:
detach the refusal (3 red) · scope it to single-project mode (portfolio arm
red) · move it below the banner (banner arm red, rc intact) · build the
ranker unconditionally (the zero-consultation measurement red) · ignore the
injected embedder (its control red).
663 -> 668 tests.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01GyAbxJoyypnLLUDcMvnKh8
S5.4 review MINOR (SC5 fail-fast hole, run.py:740). A valid-JSON but
wrong-shape savings ledger escaped the --report fail-fast refusal:
- top-level {} iterated zero keys -> entries=[] -> rc 0 "0,00 kr"
(a malformed file masquerading as a real zero-savings result)
- a bare scalar / object-with-keys / list-of-non-objects raised an
uncaught TypeError -> traceback (violates SC5 "rc 1, no traceback")
Fix at the fail-fast boundary, not the run.py except tuple: the review's
first option (add TypeError to run.py:740) leaves the {} masquerade
because {} is an empty iteration, not a TypeError. SavingsLedger.load now
raises ValueError for a non-array top-level and a non-object row, caught
by run.py:740's existing ValueError arm. Hardens both callers
(run.py:740 report + run.py:785 portfolio).
RED-first: 6 unit cases (test_ledger) + 2 CLI rc-1 cases (test_run_cli).
452 passed; ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KNNiJRk1sSwxgVLS5AobT1
SC2 second half: --verdict-dir had no main()-level test (exploration gap). New test drops one
valid verdict into a tmp inbox and drives main([pid, --docs-dir, --bundle-dir, --verdict-dir,
--live-dry-run]) -> rc 0: the inbox ingestion (load_verdicts_from_dir, run.py:287) runs before the
dry-run cut (run.py:335), so the flag's wiring is exercised offline without raising. --bundle-dir's
main()-level coverage already exists in test_live_dry_run.py and is referenced, not duplicated.
run.py untouched (never re-wired). 11 passed in test_run_cli.py.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KNNiJRk1sSwxgVLS5AobT1
After parse_args, validate mode consistency with structured refusals (rc 1, not argparse.error):
--portfolio + any single-project-only flag (--docs-dir/--bundle-dir/--verdict-dir/--outbox-dir/
--run-id/--live-dry-run) is refused naming the offending flag; --goals/--ledger outside --portfolio
is refused. --decision/--rationale are EXCLUDED (non-None defaults make explicit-vs-default
indistinguishable — Pass-2 #2; inert in portfolio mode, README says so). --dimension-config is
valid in both modes. Validation precedes the portfolio dispatch, so refusals fire before any load.
RED-first: (a)/(a')/(c) failed offline (rc 0 fall-through) before the check, green after; (b)/(b')
single-project guard + legacy backward-compat pin were already green post-Step-3. All refusal-arm
RED fall-throughs held OFFLINE (met-goal / --live-dry-run) — no socket, per brief NFR. 15 passed,
ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KNNiJRk1sSwxgVLS5AobT1
main() single-project path now parses --dimension-config (fail-fast via load_dimension),
--outbox-dir, and --run-id (deliberate 7th companion flag: determinism invariant forbids a
wall-clock run_id default). All three threaded into BOTH run_project call sites. Full-run
branch wrapped in a structured-refusal (catch ValueError/FileNotFoundError/ValidationError ->
'run refused: {exc}' on stderr, rc 1, no traceback); the EXISTING dry-run handler widened to
the same tuple (pydantic ValidationError is not a ValueError subclass; load_dimension's
FileNotFoundError would otherwise traceback — Pass-2 #1). --outbox-dir carries a loud help=
note it must differ from --verdict-dir (self-contamination footgun; documented, not enforced).
RED-first: 4 CLI tests failed on unrecognized args, green after. 7 passed (incl. live-dry-run
regression). ruff + mypy clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KNNiJRk1sSwxgVLS5AobT1
