feat(sim): the demo script is keyed on the project the prompt names, so a new project is data
The scripted proposer answered one hard-coded pair of proposals. A second project meant a second
hand-written selector, written under demo-week time pressure -- the risk the week plan names
explicitly (§4, risk 2). It is now a registry: `ScriptedCandidate` entries selected by
`scripted_proposer`, plus `project_id` as an argument to `simulate_learning_loop`.
The open decision was WHAT identifies the candidate in the prompt blob; the plan flagged it as
unverified, so it was measured. Two prompt shapes reach the selector: the debate prompt carries the
whole bundle context, the generation prompt carries `Project: {id} - {name}` plus -- as its context
-- the debate output, which is the selector's own earlier reply. So the cost code and the measure
name are present in the generation prompt only because the script put them there; keying on them
would key the script on its own output. The project id is the one identifier both shapes carry and
the framework stamps.
Validation, never repair: no match, or more than one, raises `ScriptedCandidateError`. A default
reply would answer an unregistered project with another project's numbers, which on screen is
indistinguishable from a correct run; an ambiguous blob is a data problem that must surface at the
rehearsal rather than be decided by registry order.
Load-bearing MEASURED against the whole suite, five mutations all red plus a green control: detach
the project keying - one global flip key - fall back on an unknown project - first-match on an
ambiguous prompt - detach the `project_id` argument. The flip-key test was rewritten mid-measurement
because its first form asserted on the FIRST registry entry, where "the matched candidate's key" and
"candidates[0]'s key" coincide -- it could not separate the two implementations, and proved nothing.
766 passed / 4 skipped. Simulation still exits 0, still prints eight labelled steps, still
byte-identical across two runs.
[skip-docs] README is deliberately untouched: O4 defers the README rewrite to 14-15 August, after
the demo has produced the evidence for the level-2 claim. CLAUDE.md carries the invariant.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XoHJCKBTjFKcjsfEQyGbzh
This commit is contained in:
parent
e93e921b1f
commit
688ee24973
4 changed files with 278 additions and 20 deletions
|
|
@ -52,32 +52,96 @@ def _default_bundle_dir() -> Path:
|
|||
return shared_root() / "examples" / "bygg-energi-mikro"
|
||||
|
||||
|
||||
class ScriptedCandidateError(LookupError):
|
||||
"""No single registered candidate matches the prompt (none, or more than one)."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class ScriptedCandidate:
|
||||
"""One project's scripted proposer script, expressed as DATA.
|
||||
|
||||
Adding a project to the walkthrough is a registry entry — never a second hand-written selector.
|
||||
``overclaimed`` is answered until the deterministic validator's rejection comes back in the
|
||||
prompt (recognised by ``flip_key``), after which ``corrected`` is answered. Both are raw reply
|
||||
strings, exactly what a model would have returned.
|
||||
|
||||
``flip_key`` must be ABSENT from the bundle this candidate is demoed against, or attempt 1's
|
||||
prompt already contains it and the proposer 'corrects' before anything was falsified.
|
||||
"""
|
||||
|
||||
project_id: str
|
||||
overclaimed: str
|
||||
corrected: str
|
||||
flip_key: str
|
||||
|
||||
|
||||
def scripted_proposer(
|
||||
candidates: Sequence[ScriptedCandidate],
|
||||
) -> Callable[[str, str], str]:
|
||||
"""Build the scripted proposer over a candidate registry — a ``reply_selector`` for the canonical
|
||||
client, keyed on the PROJECT the prompt names.
|
||||
|
||||
**Why the project id and not the cost code or the measure name** (measured, not assumed — the
|
||||
demo-week plan §6 flagged this as unverified): two prompt shapes reach this selector. The debate
|
||||
prompt (``run.py``) carries the whole bundle context; the generation prompt
|
||||
(``generate._build_messages``) carries ``Project: {id} - {name}`` plus, as its context, the
|
||||
*debate output* — which is this selector's own earlier reply. So the cost code and measure name
|
||||
are present in the generation prompt only because the script put them there; keying on them
|
||||
would key the script on itself. The project id is the one identifier BOTH shapes carry and the
|
||||
FRAMEWORK stamps.
|
||||
|
||||
Stateless — no per-turn counter — so the debate turns and the generation attempts share it.
|
||||
|
||||
Anything other than exactly one match raises ``ScriptedCandidateError``. Validation, never
|
||||
repair: a default reply would let an unregistered project be answered with another project's
|
||||
numbers, which on screen is indistinguishable from a correct run. An ambiguous blob (a bundle
|
||||
context that names a sibling project) is a DATA problem, and it must surface at the rehearsal
|
||||
rather than be silently decided by registry order.
|
||||
"""
|
||||
|
||||
def select(prompt: str, _role: str) -> str:
|
||||
matches = [c for c in candidates if c.project_id in prompt]
|
||||
if len(matches) != 1:
|
||||
found = ", ".join(c.project_id for c in matches) or "-"
|
||||
raise ScriptedCandidateError(
|
||||
f"no scripted candidate uniquely matches the prompt (ambiguous or unknown); "
|
||||
f"matched: {found}"
|
||||
)
|
||||
candidate = matches[0]
|
||||
return candidate.corrected if candidate.flip_key in prompt else candidate.overclaimed
|
||||
|
||||
return select
|
||||
|
||||
|
||||
# Two SavingsProposals for BYGG-KONTOR-NORD: total = 300000 x 1.0, so the degenerate Monte Carlo
|
||||
# P90 = 0.30 x 300000 = 90000 (no `assumptions`). The OVERCLAIMED one asks for 250000 — parseable,
|
||||
# and internally consistent, but above P90, so the DETERMINISTIC validator falsifies it. The
|
||||
# corrected one claims 30000 <= 90000 and validates. Together they drive Step 5 (informed
|
||||
# refinement): the proposer is scripted, but the rejection that turns proposal 1 into proposal 2 is
|
||||
# genuinely computed by the validator, not scripted.
|
||||
_OVERCLAIMED_PROPOSAL = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":250000}'
|
||||
)
|
||||
_VALID_PROPOSAL = (
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],"claimed_saving_nok":30000}'
|
||||
)
|
||||
# The flip key: the overclaimed figure, which the validator's rejection reason carries and
|
||||
# ``generate._build_messages`` appends to the NEXT attempt's prompt. Verified ABSENT from the demo
|
||||
# bundle, so it cannot pre-exist in attempt 1's prompt — the correction is caused by the
|
||||
#
|
||||
# 250000 — the overclaimed figure the validator's rejection reason carries and
|
||||
# ``generate._build_messages`` appends to the NEXT attempt's prompt — is verified ABSENT from the
|
||||
# demo bundle (``test_content_keyed_script_loadbearing``), so the correction is caused by the
|
||||
# falsification travelling back, never by the proposer simply being asked twice.
|
||||
_REJECTED_CLAIM_KEY = "250000"
|
||||
_CANDIDATES: tuple[ScriptedCandidate, ...] = (
|
||||
ScriptedCandidate(
|
||||
project_id=_PROJECT_ID,
|
||||
overclaimed=(
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||
'"claimed_saving_nok":250000}'
|
||||
),
|
||||
corrected=(
|
||||
'{"measure":"LED-retrofit av kontorbelysning","affected_items":'
|
||||
'[{"code":"ENERGI-TOTAL-EL","quantity":300000,"unit_cost":1.0}],'
|
||||
'"claimed_saving_nok":30000}'
|
||||
),
|
||||
flip_key="250000",
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def _proposer_reply(prompt: str, _role: str) -> str:
|
||||
"""The scripted proposer, keyed on PROMPT CONTENT (the canonical client's ``reply_selector``
|
||||
seam): it overclaims until the validator's rejection comes back in the prompt, then corrects.
|
||||
Stateless — no per-turn counter — so the debate turns and the generation attempts share it."""
|
||||
return _VALID_PROPOSAL if _REJECTED_CLAIM_KEY in prompt else _OVERCLAIMED_PROPOSAL
|
||||
_proposer_reply = scripted_proposer(_CANDIDATES)
|
||||
|
||||
|
||||
# The checker's debate turn ends with the gate marker the run parses (run._checker_verdict).
|
||||
|
|
@ -216,6 +280,7 @@ async def simulate_learning_loop(
|
|||
bundle_dir: str,
|
||||
work_dir: str,
|
||||
*,
|
||||
project_id: str = _PROJECT_ID,
|
||||
persona_rationale: str | None = None,
|
||||
marker: str | None = None,
|
||||
timestamp: str = "2026-06-30",
|
||||
|
|
@ -224,6 +289,11 @@ async def simulate_learning_loop(
|
|||
"""Run the loop twice on a throwaway COPY of the bundle (the shared fixture is never mutated),
|
||||
with a promotion in between, and trace whether the persona's approved knowledge crosses runs.
|
||||
|
||||
``bundle_dir`` and ``project_id`` are BOTH arguments, so pointing the walkthrough at new content
|
||||
is a call-site change plus a ``_CANDIDATES`` entry — no edit to this module's logic. The id must
|
||||
match the bundle's own IR projection (``run._project_from_bundle`` raises on a mismatch) and must
|
||||
have a registered scripted candidate.
|
||||
|
||||
The persona's verdict (decision + rationale + traced ``marker``) defaults to the shared
|
||||
expert-reviewer skill's canonical example (``load_persona_example``), read at CALL time — so the
|
||||
simulation is genuinely artifact-driven, not inlined. Callers may override ``marker`` /
|
||||
|
|
@ -256,7 +326,7 @@ async def simulate_learning_loop(
|
|||
run_a = cast(
|
||||
RunResult, # the sim only drives full runs; never dry-run (S4.2 widened run_project)
|
||||
await run_project(
|
||||
_PROJECT_ID,
|
||||
project_id,
|
||||
"local",
|
||||
docs_dir=copy_s,
|
||||
bundle_dir=copy_s,
|
||||
|
|
@ -284,7 +354,7 @@ async def simulate_learning_loop(
|
|||
run_b = cast(
|
||||
RunResult, # the sim only drives full runs; never dry-run (S4.2 widened run_project)
|
||||
await run_project(
|
||||
_PROJECT_ID,
|
||||
project_id,
|
||||
"local",
|
||||
docs_dir=copy_s,
|
||||
bundle_dir=copy_s,
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue