docs(okf-v0.2): the pilot tests state their expected results up front
The pilot invitations asked for feedback without specifying what to run or what we expect to see. That returns a description, not a measurement: only a stated expectation lets someone else's run falsify our model instead of merely confirming it. Each of the three tests now carries a numbered procedure, numbered expected outcomes, and an explicit list of what would surprise us. Naming the surprises is the load-bearing part -- a consumer who sees something odd but passing otherwise has no reason to mention it. Test A (producer, portfolio-optimiser-claude) runs one already-working manifest twice with the same explicit ingested_at, once under DEFAULT as baseline and once under OKF_V0_2, and diffs. A-E1 is the stop condition: if the DEFAULT run is not byte-identical to their current pinned output, we have broken a v0.1 consumer and the pilot ends there. The named surprises include `at` differing from the ingested_at they passed, which would mean a wall-clock crept in, and a collision refusal on a file that IS theirs, which is the inverse of the fail-safe we predicted -- we expect foreign files to be refused, so a false refusal of their own is the defect. Test B (gate, catalog) runs their unmodified gate on the v0.2 fixture and on a 0.1 control. B-E3 -- that no gate other than the version gate behaves differently between the two -- is what converts our reading of their form regex into a measurement. A membership list or equality comparison anywhere in their chain is exactly what this is paid for. Test C (expressiveness, wiki) is run by us, read-only at a recorded commit, and the report goes to them. What we ask them to check is the part we cannot measure from outside: whether a field we called an optional addition is in fact load-bearing in their pipeline, and whether "no change required" holds operationally rather than only formally. Feedback gains a per-expectation verdict line so three independent runs are comparable and a disagreement is located rather than merely reported. The request also says outright that a wrong expectation is a better result than a clean run, since a clean run only confirms what we already believed. Inputs come from this repo at the pre-release tag; nothing is transported through the mailbox except the specification. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
This commit is contained in:
parent
a27017e635
commit
5de590ebdf
1 changed files with 94 additions and 2 deletions
|
|
@ -244,10 +244,97 @@ stated:
|
|||
deprecation cycle. Saying this up front is what buys the freedom to act on
|
||||
feedback; discovering it later is what makes a pilot a de-facto release.
|
||||
|
||||
### Test specifications — stated before the run, with expected results
|
||||
|
||||
A pilot that asks "run it and tell us how it went" returns a description. A pilot
|
||||
that states the expected result first returns a *measurement*, because only then
|
||||
can the consumer see that something differed. So each test below ships with its
|
||||
expected outcomes numbered, plus an explicit list of what would surprise us —
|
||||
naming the surprises is what lets someone else's run falsify our model instead of
|
||||
merely confirming it.
|
||||
|
||||
All three read their inputs from this repo at the pre-release tag (`v0.5.0a1`);
|
||||
nothing is transported through the mailbox except the specification itself.
|
||||
|
||||
#### Test A — producer path (`portfolio-optimiser-claude`)
|
||||
|
||||
**Procedure.** In a scratch venv, install the package at the `v0.5.0a1` tag. Take
|
||||
**one manifest they have already run under `DEFAULT`**, so the existing v0.1
|
||||
output is the baseline. Materialize it twice with the *same* explicit
|
||||
`ingested_at`: once with `profile=DEFAULT`, once with `profile=OKF_V0_2`. Diff.
|
||||
|
||||
**Expected:**
|
||||
- **A-E1** The `DEFAULT` run is byte-identical to what their currently pinned
|
||||
version produces. Support is additive; if this fails we have broken a v0.1
|
||||
consumer and the pilot stops here.
|
||||
- **A-E2** The v0.2 run differs from `DEFAULT` **only inside the frontmatter
|
||||
block**. Body bytes identical, filenames identical.
|
||||
- **A-E3** `generated` is an inline flow mapping
|
||||
`{ by: llm-ingestion-okf/<version>, at: <ingested_at> }` — not the string
|
||||
`true` — and `at` equals the `ingested_at` argument **exactly**.
|
||||
- **A-E4** `sources` is present as a block list carrying at least a `resource`
|
||||
derived from the manifest's source.
|
||||
- **A-E5** Re-running with the same `ingested_at` is byte-identical.
|
||||
- **A-E6** The root `index.md` declares `okf_version: "0.2"`.
|
||||
|
||||
**What would surprise us — report immediately:** any diff outside the frontmatter
|
||||
block (the profile seam leaks); `at` differing from the `ingested_at` they passed
|
||||
(a wall-clock crept in — a determinism defect); a collision refusal on a target
|
||||
file that **is** theirs (`_is_ingest_owned` mis-firing under the new `generated`
|
||||
shape — we predict fail-safe refusal of *foreign* files, so a false refusal of
|
||||
their *own* is the failure mode); and anything about constructing the profile that
|
||||
required reading our source to work out.
|
||||
|
||||
#### Test B — gate acceptance (`catalog`)
|
||||
|
||||
**Procedure.** Run their **existing, unmodified** gate on two bundles from
|
||||
`examples/` at the tag: the v0.2 fixture, and the same fixture with `okf_version`
|
||||
set to `"0.1"` as a control. No pin, no code change, two gate runs.
|
||||
|
||||
**Expected:**
|
||||
- **B-E1** Both pass. The form regex accepts both values, so the two runs should
|
||||
be indistinguishable.
|
||||
- **B-E2** No gate emits a WARNING or ERROR mentioning the version.
|
||||
- **B-E3** No gate *other than* the version gate behaves differently between the
|
||||
two runs.
|
||||
|
||||
**What would surprise us:** any gate anywhere in their chain that keys off the
|
||||
version **value** rather than its form — a membership list, an equality
|
||||
comparison, a switch. We asserted from reading their regex that none exists;
|
||||
B-E3 is what converts that assertion into a measurement. Also: a gate requiring
|
||||
`timestamp` on a concept (superseded in v0.2), or anything downstream of the gate
|
||||
— their re-pin flow, the discovery mechanism — that assumes `0.1`.
|
||||
|
||||
#### Test C — expressiveness (`claude-code-llm-wiki`, run by us)
|
||||
|
||||
**Procedure (ours).** Construct a v0.2 variant of `STRICT_V1` here, read their
|
||||
bundle **read-only at a recorded commit**, validate all 522 documents, send them
|
||||
the report. They run nothing.
|
||||
|
||||
**Expected:**
|
||||
- **C-E1** All 522 documents pass the v0.2 variant while keeping `timestamp` and
|
||||
emitting no `generated` — the §13.1 legacy path.
|
||||
- **C-E2** The variant requires **no** change to their eight required keys.
|
||||
- **C-E3** Everything a v0.2 posture would add is optional: `generated`,
|
||||
`verified`, `status`, `stale_after`, `sources`.
|
||||
- **C-E4** Their per-directory index policy and root frontmatter
|
||||
(`okf_version` / `bundle_profile` / `okf_spec_commit`) are expressible
|
||||
unchanged.
|
||||
|
||||
**What we ask them to check in our report** — the parts we cannot measure from
|
||||
outside: whether any field we classified as an optional addition is in fact
|
||||
load-bearing in their pipeline; whether our reading of their contract still holds
|
||||
at their current HEAD; and whether "no change required" is true *operationally*
|
||||
and not just formally, since they know their consumers and we do not.
|
||||
|
||||
### Feedback shape
|
||||
|
||||
Unstructured feedback is not comparable across three repos, so ask for five
|
||||
things by name:
|
||||
Unstructured feedback is not comparable across three repos, so ask for a
|
||||
**per-expectation verdict plus five things by name**.
|
||||
|
||||
First, one line per numbered expectation — `A-E1: as expected` / `A-E3: differed,
|
||||
<what we saw>`. That is the part that makes three independent runs comparable and
|
||||
that turns a disagreement into a located one. Then:
|
||||
|
||||
1. What was run, against which tag.
|
||||
2. What failed — with the bytes or the actual error, not a summary.
|
||||
|
|
@ -256,6 +343,11 @@ things by name:
|
|||
4. What was needed that the profile could not express.
|
||||
5. Whether they would adopt it as-is.
|
||||
|
||||
**An expectation we got wrong is a better result than a clean run**, and the
|
||||
request says so explicitly — otherwise a consumer who sees something odd but
|
||||
passing has no reason to mention it. A clean run confirms what we already believed;
|
||||
a differed expectation is the only thing that can change the design.
|
||||
|
||||
### GA exit criteria
|
||||
|
||||
1. Every P1–P3 finding is either fixed or explicitly accepted with the reason
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue