open/llm-ingestion-okf
1
0
Fork 0
Code Issues Pull requests Releases Activity

docs(okf-v0.2): the pilot answered before the tag existed, and it changed the tag

Browse source 
All three pilots accepted. Their measurements land in the plan now rather than
as failed runs later, which is the entire return on asking before building.

Six corrections, each measured by the consumer rather than reasoned by us:

- A-E6 specified `okf_version: "0.2"` quoted. Catalog measured the quoted form
  failing their shape regex with exit 1, and a UTF-8 BOM making the marker
  invisible while still exiting 0. Both now stated as raw-byte expectations.
- A-E2 said the diff lies only *inside* the frontmatter block. No bundle whose
  root index has no frontmatter block can satisfy that together with A-E6.
- B-E3 covers one of catalog's three gates; the other two cannot see our
  fixture. Recorded so the report is not read as broader than it is.
- Our claim that no gate keys on the version value was wrong as formulated:
  check-okf-parity.mjs puts the raw value in its signature. It compares without
  judging -- verdict-invariant, not value-blind.
- C-E1's ground truth is 524 non-reserved documents, not 522; 522 was correct
  at d2c12d2.
- `materialize_bundle` gains keyword-only `profile=DEFAULT` on po-claude's
  request. The signature already has a `*`, so their three-positional call path
  stays source-compatible and additivity becomes a property of the signature
  instead of something a consumer measures.

Also recorded: po-claude closed their index-parser defect themselves (f41264f),
so do not design around it; the assumption under it -- "a generated index has no
frontmatter" -- is not peculiar to them. Dropping the superseded `timestamp`
draws one catalog warning per concept file, decided after the pilot, not before.

The estimate's open caveat is discharged by reading profiles.py:197: emit is one
formatting path, the validator already admits list values, so only the emitter
side is missing and the sizing stands.

CLAUDE.md gains a standing constraint: consumer content stays at form level in
public files. The wiki's bundle is private pending an Anthropic ToS assessment,
their report goes back through coord, and a push is not reversible.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01A2aKJxLejT9S8jYwoZ9fut
This commit is contained in:
Kjell Tore Guttormsen 2026-07-26 17:03:05 +02:00
commit f7385fafdb
2 changed files with 160 additions and 19 deletions
Download patch file Download diff file Expand all files Collapse all files

8
CLAUDE.md
View file

@ -152,6 +152,14 @@ and fixtures, never code.
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
session state or internal briefs. No secrets, sober English prose, no
marketing language.
- **Consumer content stays at form level in public files.** Some consumers we
read are private (`claude-code-llm-wiki` is, pending an Anthropic ToS
assessment). Key names, counts, gate names, profile fields and contract shapes
are publishable; page bodies, full title or path lists from a consumer's
bundle, and Anthropic-derived prose are not. Findings about a private
consumer's data go back to them through coord, never as a file here. This
costs nothing — every question this library asks of a consumer is about shapes
and key sets — and it is not reversible once pushed.
- After `git commit`: push to Forgejo (`git push origin`) immediately.
Never GitHub.

171
docs/plan/okf-v0.2-alignment.md
View file

@ -203,7 +203,7 @@ Three repos, each exercising a different axis, and deliberately cheap for them:
|---|---|---|---|
| `portfolio-optimiser-claude` | **Producer path.** Real Door A ingestion emitting v0.2 from a real manifest | Pin the pre-release tag, run one real manifest, report | one run |
| `catalog` | **Gate acceptance.** Does a bundle *declaring* `okf_version: "0.2"` actually pass their gate — measured against their gate, not our reading of its regex | Run their existing gate on our fixture. No code change | one gate run |
| `claude-code-llm-wiki` | **Expressiveness.** Can the profile object express a strict consumer at v0.2, over 522 real documents | Nothing. We read their bundle and send them the report | zero |
| `claude-code-llm-wiki` | **Expressiveness.** Can the profile object express a strict consumer at v0.2, over 524 real documents | Nothing. We read their bundle and send them the report | zero |
The wiki entry is the important design detail: **most of the pilot is read-only
on our side.** We run the v0.2 reader over real consumer bundles and report what
@ -229,6 +229,92 @@ deciding V1 — a participant, not a test site); `llm-ingestion-pipeline-securit
- **P4 — fix, then GA.** Address findings, then flip `OKF_LATEST` as its own
release.
### Pilot responses — all three accepted, and the asking paid for itself
All three replied on 2026-07-26, before the tag exists. That timing is the whole
return on asking early: every finding below changes the tag's *design* rather
than showing up as a failed run afterwards.
- **`portfolio-optimiser-claude` — yes, conditional.** Their conditions are their
own hygiene, not reservations about us: scratch venv, no edit to their
`ingest.py`, no bump of their production pin (`v0.3.2` = `f14c075`). Their
exposure to both backward-breaking changes is **measured at zero** against
their byte-frozen golden: they have no `timestamp` field at all (they have
`ingested_at`, a different field with a different mandate), and no `# Citations`
heading in any body they produce or read. Their current form is literally
`generated: true`, so A-E3 is calibrated correctly against them. Baseline:
`examples/ingest-golden-file/manifest.json`, chosen because it is byte-frozen
alongside its `expected-bundle/` and `ingested-at.txt` **and** is the shared
fixture against the sibling implementation — so A-E1 is an exact byte
comparison against a baseline two independent implementations already ratified.
- **`catalog` — yes.** Accepted the provisional terms explicitly. Runs after
2026-07-30. Pre-measured their own gate on a synthetic proxy (6 runs,
`okf-check.mjs` unmodified at `6a72b26`) because we asked them not to trust a
*reading* of their regex — so they declined to trust one either. They are
fixing neither the quoting hole nor the BOM hole before the run, deliberately:
we asked for their existing unmodified gate, and hardening it now would make
Test B measure something else.
- **`claude-code-llm-wiki` — yes to both the read and the report**, with one
condition on our public track (below). They pre-accept none of C-E1..C-E4 and
will answer per expectation. One qualifier on what they can answer: "load-
bearing" for them means load-bearing for `validate.py`'s gates **and** for
consumers they do not control — they will say which of the two any answer rests
on.
**A premise of ours they corrected.** The pilot request said "no code change on
your side beyond the pin". That is false for po-claude's call path: their adapter
calls `materialize_bundle` with three positional arguments and exposes no
`profile` parameter, so the procedure's `profile=DEFAULT` / `profile=OKF_V0_2`
has nowhere to land. They solve it in scratch without touching their repo, so
their condition holds — but it means **A-E1 measures the library API, not their
adapter seam**, and we must read the result as such. Requirement 6 above is the
design response.
**A defect they closed rather than had us design around.** po-claude's index
parser died with `ValueError` on any `index.md` carrying frontmatter without a
`type` key — which is exactly what A-E6 produces, and it would have killed all of
`navigate_bundle`, not just the index read. Fixed on their side in `f41264f`
before we designed anything, mutation-proven both directions. **Do not design
around it.** The generalizable part is the assumption underneath it — "a
generated index has no frontmatter" — which was frozen into their golden where
nothing could catch it, and is not peculiar to them. Any consumer stamping
`okf_version` into a generated index may hold it.
**Their frontmatter parser is line-oriented, not YAML** (`line.partition(":")`),
which is the same shape as ours and gives v0.2's new families a predictable
failure mode elsewhere: `sources` as a block list pollutes the dict with false
keys (`- resource`, `author`), and `generated` as an inline flow mapping is
stored as its literal string. Latent and not reached today: last key wins, so a
nested `type:` anywhere in a block list would overwrite a file's own `type`. None
of the fields we specified is named `type`, so this is a warning about the
*form*, not about v0.2 — and it is one more reason the structured reader (D1)
is real work rather than a formality.
**Dropping `timestamp` is not free at the gate.** Catalog does not require it —
it sits in RECOMMENDED, so absence is a WARNING with no exit effect. But a v0.2
bundle that drops the superseded field draws one warning **per concept file**,
noise that grows with bundle size. Whether their gate should stay silent about a
superseded field is a change on their side, to be decided after the pilot, not
before.
### The wiki's condition on our public track
Their bundle and code are **private** pending an Anthropic ToS assessment (their
E3, operator decision 2026-07-26). This repo pushes to `open/`. The condition,
which they name as the one thing that cannot be reversed:
- The Test C report goes to them **via coord**, never as a file in a public tree.
- In public files here, stay at **form level**. Key names, counts, gate names and
profile fields are fine and already present, and they have no objection to
those. **Not** page bodies, **not** full title or path lists from their
`bundle/`, **not** Anthropic-derived text (release notes, documentation prose).
This is not an objection to the pilot, and it costs the pilot nothing: Test C
measures whether a profile can *express* their contract, which is a question
about shapes and key sets. Nothing it needs to answer requires quoting their
content. The constraint is recorded in `CLAUDE.md` as well, because it binds
every future session writing into this repo, not just the ones running Test C.
### What `v0.5.0a1` minimally requires — much less than the deliverable list
Walking each pilot test back to what it actually exercises produces a result the
@ -246,14 +332,27 @@ structured reader.**
The structured reader exists to **consume third-party v0.2 bundles** — that is
D3/Door C, and no pilot repo sends us one. So the tag needs:
1. `FrontmatterSchema.emit` accepts a block-list value (`sources`).
1. `FrontmatterSchema.emit` accepts a block-list value (`sources`). Verified
2026-07-26: `emit` + `_render` are a single formatting path rendering values
verbatim, and the *validator* already admits "a string or a non-empty list of
strings" (`_is_legal_value`). Only the emitter side is missing, so the sizing
caveat below is discharged — the estimate holds.
2. An `OKF_V0_2` profile constant — same shape as the existing `STRICT_V1` work.
3. `_is_ingest_owned` becomes profile-aware, since the ownership marker now
differs per profile.
4. `sources` derived from the manifest source in the materializer.
5. One v0.2 golden fixture in `examples/`, generated by the code rather than
hand-authored, or it does not test the emitter.
6. The tag.
hand-authored, or it does not test the emitter. Its root `index.md` carries
`okf_version: 0.2` **unquoted and BOM-free** — both measured as gate-breaking
by catalog (see A-E6), so both get an assertion on the raw bytes rather than
on the parsed value, which would mask exactly these two defects.
6. **`materialize_bundle` takes `profile` as a keyword-only argument defaulting
to `DEFAULT`.** Added on po-claude's request, and it is nearly free: the
signature already has a `*` marker, so the parameter goes behind it and every
existing three-positional call site stays source-compatible. The reason to
take the request is that it converts the additivity A-E1 asserts from
something a consumer measures into a property of the signature itself.
7. The tag.
Plus TDD steps 1–3 first as the safety net; they change no behavior.
@ -267,9 +366,13 @@ characterization test (TDD step 1) is written before it is touched. Item 1 also
runs through `DEFAULT`'s code path, so V-A6 must be *proven* by the golden suite,
not assumed.
**Caveat on the estimate:** it treats `FrontmatterSchema.emit` as a single
formatting function, inferred from its tests rather than read. Verify that before
committing to the sizing — the estimate is a premise like any other.
**Caveat on the estimate — discharged 2026-07-26.** It treated
`FrontmatterSchema.emit` as a single formatting function, inferred from its tests
rather than read. Now read: `profiles.py:197` renders one `key: value` line per
key through one `_render` helper, verbatim except for `source_query`'s
whitespace collapse. The premise held, and the sizing stands. Recorded rather
than deleted, because a premise that survives verification is only known to have
survived if the check is visible.
### Shipping a provisional surface without owing stability
@ -309,15 +412,25 @@ output is the baseline. Materialize it twice with the *same* explicit
- **A-E1** The `DEFAULT` run is byte-identical to what their currently pinned
version produces. Support is additive; if this fails we have broken a v0.1
consumer and the pilot stops here.
- **A-E2** The v0.2 run differs from `DEFAULT` **only inside the frontmatter
block**. Body bytes identical, filenames identical.
- **A-E2** The v0.2 run differs from `DEFAULT` **only in frontmatter**. Body
bytes identical, filenames identical. *Revised on po-claude's measurement:*
the original text said "only **inside** the frontmatter block", which no
bundle whose root `index.md` has no frontmatter block can satisfy — there,
A-E6 necessarily **adds** a block rather than editing within one. Adding a
frontmatter block where the profile requires one satisfies A-E2; a diff in
body bytes does not.
- **A-E3** `generated` is an inline flow mapping
`{ by: llm-ingestion-okf/<version>, at: <ingested_at> }` — not the string
`true` — and `at` equals the `ingested_at` argument **exactly**.
- **A-E4** `sources` is present as a block list carrying at least a `resource`
derived from the manifest's source.
- **A-E5** Re-running with the same `ingested_at` is byte-identical.
- **A-E6** The root `index.md` declares `okf_version: "0.2"`.
- **A-E6** The root `index.md` declares `okf_version: 0.2` — **unquoted**, and
the file carries **no UTF-8 BOM**. Both are load-bearing, not stylistic:
catalog measured that a quoted value fails their shape regex `exit 1`, and
that a BOM makes the marker invisible to them while still exiting `0`. The
emitter renders values verbatim, so this is a property of the value we hand
it. Verifying the raw bytes here is part of the expectation.
**What would surprise us — report immediately:** any diff outside the frontmatter
block (the profile seam leaks); `at` differing from the `ingested_at` they passed
@ -338,24 +451,44 @@ set to `"0.1"` as a control. No pin, no code change, two gate runs.
be indistinguishable.
- **B-E2** No gate emits a WARNING or ERROR mentioning the version.
- **B-E3** No gate *other than* the version gate behaves differently between the
two runs.
two runs — **scoped to `okf-check.mjs`**, the one gate of their three that
takes an arbitrary bundle root. `check-okf-parity.mjs` runs over a fixed
corpus and `check-nav-golden.mjs` over their own bundle directory; neither can
see our fixture without a corpus change we excluded. Their verdict for those
two is "not run", and it must not be read as "as expected". Widening this is a
separate request with its own price, and it is the operator's call.
**What would surprise us:** any gate anywhere in their chain that keys off the
version **value** rather than its form — a membership list, an equality
comparison, a switch. We asserted from reading their regex that none exists;
B-E3 is what converts that assertion into a measurement. Also: a gate requiring
`timestamp` on a concept (superseded in v0.2), or anything downstream of the gate
— their re-pin flow, the discovery mechanism — that assumes `0.1`.
comparison, a switch. Also: a gate requiring `timestamp` on a concept
(superseded in v0.2), or anything downstream of the gate — their re-pin flow,
the discovery mechanism — that assumes `0.1`.
**Correction, already measured (catalog, 2026-07-26).** Our assertion that *no*
gate keys on the value was wrong as formulated, and they falsified it before the
run rather than during it: `check-okf-parity.mjs:75` puts the **raw**
`okf_version` value into its cross-implementation signature. It *compares* the
value across implementations without *judging* it — verdict-invariant, but not
value-blind. The practical consequence for us is nil, since both implementations
read the same file and see the same value. The distinction is kept because
compare-vs-judge is precisely what B-E3 was written to settle, and recording it
imprecisely would discard the finding.
#### Test C — expressiveness (`claude-code-llm-wiki`, run by us)
**Procedure (ours).** Construct a v0.2 variant of `STRICT_V1` here, read their
bundle **read-only at a recorded commit**, validate all 522 documents, send them
the report. They run nothing.
bundle **read-only at a recorded commit**, validate all non-reserved documents,
send them the report **through coord**. They run nothing.
**Expected:**
- **C-E1** All 522 documents pass the v0.2 variant while keeping `timestamp` and
emitting no `generated` — the §13.1 legacy path.
- **C-E1** All **524** non-reserved documents pass the v0.2 variant while keeping
`timestamp` and emitting no `generated` — the §13.1 legacy path. The count is
theirs, given as ground truth before the run: 529 `.md` total, minus 4
`index.md`, minus 1 `README`/`log`, at HEAD `b9b557b` (ingest run
`run-20260726T053004Z`). 522 was correct at `d2c12d2` and is now stale — the
delta is exactly Claude Code v2.1.219 + v2.1.220. Reading at a fresher HEAD is
allowed provided the commit is recorded; the count that must pass is whatever
their stated ground truth says at the commit we actually read.
- **C-E2** The variant requires **no** change to their eight required keys.
- **C-E3** Everything a v0.2 posture would add is optional: `generated`,
`verified`, `status`, `stale_after`, `sources`.