The pilot invitations asked for feedback without specifying what to run or
what we expect to see. That returns a description, not a measurement: only a
stated expectation lets someone else's run falsify our model instead of
merely confirming it.
Each of the three tests now carries a numbered procedure, numbered expected
outcomes, and an explicit list of what would surprise us. Naming the
surprises is the load-bearing part -- a consumer who sees something odd but
passing otherwise has no reason to mention it.
Test A (producer, portfolio-optimiser-claude) runs one already-working
manifest twice with the same explicit ingested_at, once under DEFAULT as
baseline and once under OKF_V0_2, and diffs. A-E1 is the stop condition: if
the DEFAULT run is not byte-identical to their current pinned output, we have
broken a v0.1 consumer and the pilot ends there. The named surprises include
`at` differing from the ingested_at they passed, which would mean a
wall-clock crept in, and a collision refusal on a file that IS theirs, which
is the inverse of the fail-safe we predicted -- we expect foreign files to be
refused, so a false refusal of their own is the defect.
Test B (gate, catalog) runs their unmodified gate on the v0.2 fixture and on
a 0.1 control. B-E3 -- that no gate other than the version gate behaves
differently between the two -- is what converts our reading of their form
regex into a measurement. A membership list or equality comparison anywhere
in their chain is exactly what this is paid for.
Test C (expressiveness, wiki) is run by us, read-only at a recorded commit,
and the report goes to them. What we ask them to check is the part we cannot
measure from outside: whether a field we called an optional addition is in
fact load-bearing in their pipeline, and whether "no change required" holds
operationally rather than only formally.
Feedback gains a per-expectation verdict line so three independent runs are
comparable and a disagreement is located rather than merely reported. The
request also says outright that a wrong expectation is a better result than a
clean run, since a clean run only confirms what we already believed.
Inputs come from this repo at the pre-release tag; nothing is transported
through the mailbox except the specification.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
Kjell Tore Guttormsen
5de590ebdf