test(major2): hver genereringstelling paret med debattens telling (SC1/SC4)
Funn 5935d942 (MISSING_TEST, tests/...:820). Briefens SC1 gjorde paringen til vakten mot aa telle en debatt-tur som en genereringsprompt - og ingen arm i fila asserterte debattens turantall mellom treated og control (grep «debate» traff bare en kommentar og et filnavn). `_debate_entries` er KOMPLEMENTET av `_GENERATION_MARK`, ikke en positiv debatt-markoer: paastanden som gates er nettopp «ingenting som IKKE er generering flyttet seg», og en positiv markoer ville latt en tur klassifisereren ikke kjenner drive usett. - T5-run: de tre genereringstellingene paret med likhet paa debatt-oppfoeringene. - T8: fikk sink + en control-kjoering (alltid-godkjenn reviewer) - dens attempt-indekser [0,1] per kandidat ER en genereringstelling. - Begge har en VAKUITETSVAKT (debatt-lista maa vaere ikke-tom): to tomme lister er like gratis. - T13: record-indeksene er DOKUMENTERT som SC4s telle-proxy ved den doera - barnet kjoerer i egen interpreter og `--scripted-replies` har ingen sink-dump, saa prompt-nivaaet maales in-process (T1 for verbatim, T5-run/T8 for paringen). Reviewens andre alternativ. MAALT mot HELE suiten, to mutasjoner, begge roede paa NOEYAKTIG de to parede armene og paa ingen andre: MA klassifisereren returnerer konstant tom liste (2 roede - vakuitetsvakten) og MB filteret droppet, saa generering telles som debatt (2 roede). Kontroll 1364 passed / 5 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
c81a90c88a
commit
d4a0220b68
1 changed files with 54 additions and 9 deletions
|
|
@ -657,6 +657,19 @@ _VERDICT_INPUT = {"decision": "approved", "rationale": "expert reviewed (sim)"}
|
||||||
#: else — the one identifier that separates a generation call from a debate turn.
|
#: else — the one identifier that separates a generation call from a debate turn.
|
||||||
_GENERATION_MARK = "Respond with ONLY a JSON object"
|
_GENERATION_MARK = "Respond with ONLY a JSON object"
|
||||||
|
|
||||||
|
|
||||||
|
def _debate_entries(sink: list[str]) -> list[str]:
|
||||||
|
"""Everything in the sink that is NOT a generation prompt — i.e. the debate's turns.
|
||||||
|
|
||||||
|
The complement of ``_GENERATION_MARK`` rather than a positive debate marker, because the claim
|
||||||
|
being guarded is exactly "nothing that is not generation moved": a positive marker would let a
|
||||||
|
turn the classifier does not know about drift unseen. Every "N generation prompts" assertion in
|
||||||
|
this file is paired with an equality of THIS list between the treated and the control run
|
||||||
|
(SC1/SC4) — without the pairing, an arm that counted a debate turn as a generation prompt, or a
|
||||||
|
door that perturbed the debate, would read as a correct generation count."""
|
||||||
|
return [blob for blob in sink if _GENERATION_MARK not in blob]
|
||||||
|
|
||||||
|
|
||||||
# BYGG-KONTOR-NORD: affected total 300000 x 1.0 -> degenerate Monte Carlo P90 = 90000. A claim
|
# BYGG-KONTOR-NORD: affected total 300000 x 1.0 -> degenerate Monte Carlo P90 = 90000. A claim
|
||||||
# <= 90000 validates; above it the deterministic validator rejects.
|
# <= 90000 validates; above it the deterministic validator rejects.
|
||||||
_A1 = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer")
|
_A1 = Approach(id="led-retrofit", label="Behovsstyrt belysning i fellesarealer")
|
||||||
|
|
@ -755,18 +768,35 @@ async def test_t8_every_candidate_gets_its_own_keyed_answer(tmp_path: Path) -> N
|
||||||
With a commissioned mandate one expert is asked N times at one terminal, so an artefact that
|
With a commissioned mandate one expert is asked N times at one terminal, so an artefact that
|
||||||
cannot say WHICH candidate a sentence was about is an artefact nobody can act on. The run's
|
cannot say WHICH candidate a sentence was about is an artefact nobody can act on. The run's
|
||||||
own proposal is keyed ``OWN_PROPOSAL_ID`` — recording ``None`` there would make it
|
own proposal is keyed ``OWN_PROPOSAL_ID`` — recording ``None`` there would make it
|
||||||
indistinguishable from a non-mandate run's entry, which is the property keying exists for."""
|
indistinguishable from a non-mandate run's entry, which is the property keying exists for.
|
||||||
|
|
||||||
|
The per-candidate attempt indices below are a GENERATION count (two attempts each), so SC1
|
||||||
|
requires them paired with the debate's count: a control run whose reviewer always approves must
|
||||||
|
leave the non-generation entries identical."""
|
||||||
|
mandate = Mandate(
|
||||||
|
objective="Cut energy cost without rebuilding.",
|
||||||
|
approaches=(_A1, _A2),
|
||||||
|
allow_own_proposals=True,
|
||||||
|
)
|
||||||
reviewer = _RunReviewer()
|
reviewer = _RunReviewer()
|
||||||
|
treated_sink: list[str] = []
|
||||||
result = await _run_with(
|
result = await _run_with(
|
||||||
outbox_dir=tmp_path / "outbox",
|
outbox_dir=tmp_path / "outbox",
|
||||||
reviewer=reviewer,
|
reviewer=reviewer,
|
||||||
mandate=Mandate(
|
mandate=mandate,
|
||||||
objective="Cut energy cost without rebuilding.",
|
sink=treated_sink,
|
||||||
approaches=(_A1, _A2),
|
|
||||||
allow_own_proposals=True,
|
|
||||||
),
|
|
||||||
)
|
)
|
||||||
assert isinstance(result, RunResult)
|
control_sink: list[str] = []
|
||||||
|
control = await _run_with(
|
||||||
|
outbox_dir=tmp_path / "control",
|
||||||
|
run_id="run-control",
|
||||||
|
reviewer=_RunReviewer(revise_for=set()),
|
||||||
|
mandate=mandate,
|
||||||
|
sink=control_sink,
|
||||||
|
)
|
||||||
|
assert isinstance(result, RunResult) and isinstance(control, RunResult)
|
||||||
|
assert _debate_entries(treated_sink), "no debate turns reached the sink: the pairing is vacuous"
|
||||||
|
assert _debate_entries(treated_sink) == _debate_entries(control_sink)
|
||||||
revisions = {
|
revisions = {
|
||||||
(r.approach_id, r.feedback) for r in result.expert_revisions if r.decision == "revise"
|
(r.approach_id, r.feedback) for r in result.expert_revisions if r.decision == "revise"
|
||||||
}
|
}
|
||||||
|
|
@ -824,6 +854,10 @@ async def test_t5run_a_revise_on_one_approach_leaves_the_next_untouched(tmp_path
|
||||||
assert len(_gen_prompts(control_sink, _A2.label)) == 1
|
assert len(_gen_prompts(control_sink, _A2.label)) == 1
|
||||||
assert _gen_prompts(treated_sink, _A3.label) == _gen_prompts(control_sink, _A3.label)
|
assert _gen_prompts(treated_sink, _A3.label) == _gen_prompts(control_sink, _A3.label)
|
||||||
assert all(_FEEDBACK_SENTINEL not in b for b in _gen_prompts(treated_sink, _A3.label))
|
assert all(_FEEDBACK_SENTINEL not in b for b in _gen_prompts(treated_sink, _A3.label))
|
||||||
|
# SC1's pairing: the counts above mean "one more GENERATION prompt" only if the debate did not
|
||||||
|
# move. The non-empty check is the vacuity guard — two empty lists are equal for free.
|
||||||
|
assert _debate_entries(treated_sink), "no debate turns reached the sink: the pairing is vacuous"
|
||||||
|
assert _debate_entries(treated_sink) == _debate_entries(control_sink)
|
||||||
|
|
||||||
|
|
||||||
async def test_t9_a_budget_stop_inside_generation_still_leaves_the_record(tmp_path: Path) -> None:
|
async def test_t9_a_budget_stop_inside_generation_still_leaves_the_record(tmp_path: Path) -> None:
|
||||||
|
|
@ -1129,7 +1163,16 @@ def test_t13_the_flag_answers_the_review_from_a_real_argv_and_the_answer_is_used
|
||||||
|
|
||||||
(The plan's separate T19 — "a bare, unscripted flag parse" — is folded in here: without a
|
(The plan's separate T19 — "a bare, unscripted flag parse" — is folded in here: without a
|
||||||
script that argv would call a live model, so the argparse witness is this child's own
|
script that argv would call a live model, so the argparse witness is this child's own
|
||||||
``unrecognized arguments`` assertion.)"""
|
``unrecognized arguments`` assertion.)
|
||||||
|
|
||||||
|
**The attempt indices are SC4's count PROXY at this door, and nothing more — stated rather
|
||||||
|
than left to be read as a prompt-level count.** The child runs in its own interpreter, so the
|
||||||
|
``ScriptedChatClient`` sink lives in that process and there is no CLI flag that dumps it: the
|
||||||
|
prompt-level half of SC4 (the feedback VERBATIM in the second generation prompt, and the
|
||||||
|
debate's turn count unchanged against a control) is measured in-process, where the sink is
|
||||||
|
reachable — T1 for the verbatim injection, T5-run and T8 for the debate pairing. What this arm
|
||||||
|
settles that none of those can is that a real argv, a real pipe and a real ``run_project``
|
||||||
|
bought a second attempt whose OUTCOME differs (40 000, entry 2 of the step list)."""
|
||||||
proc = _child(_cli_argv(tmp_path, outbox="outbox"), stdin=f"revise {_CLI_FEEDBACK}\napprove\n")
|
proc = _child(_cli_argv(tmp_path, outbox="outbox"), stdin=f"revise {_CLI_FEEDBACK}\napprove\n")
|
||||||
|
|
||||||
assert "unrecognized arguments" not in proc.stderr, proc.stderr
|
assert "unrecognized arguments" not in proc.stderr, proc.stderr
|
||||||
|
|
@ -1142,7 +1185,9 @@ def test_t13_the_flag_answers_the_review_from_a_real_argv_and_the_answer_is_used
|
||||||
("approve", True),
|
("approve", True),
|
||||||
]
|
]
|
||||||
assert payload["reviews"][0]["feedback"] == _CLI_FEEDBACK # VERBATIM
|
assert payload["reviews"][0]["feedback"] == _CLI_FEEDBACK # VERBATIM
|
||||||
assert [r["attempt"] for r in payload["reviews"]] == [0, 1] # exactly two generation attempts
|
# SC4's count proxy at this door (see the docstring): two attempt indices, not two counted
|
||||||
|
# prompts — the child's sink is in the child.
|
||||||
|
assert [r["attempt"] for r in payload["reviews"]] == [0, 1]
|
||||||
|
|
||||||
outcome = json.loads(
|
outcome = json.loads(
|
||||||
(tmp_path / "outbox" / f"{_RUN_ID}-proposal.json").read_text(encoding="utf-8")
|
(tmp_path / "outbox" / f"{_RUN_ID}-proposal.json").read_text(encoding="utf-8")
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue