llm-ingestion-okf/docs/upstream-okf-upgrade-runbook.md

Upstream OKF upgrade — runbook

Standing policy (operator, 2026-07-26): this library always supports the current latest version of Google OKF. This document is the procedure that policy runs on. It exists because a standing policy without a procedure is an intention, and because the v0.1 → v0.2 round produced five findings that a spec reading alone had settled wrongly — each step below names the concrete failure it prevents.

Written after the v0.2 round. Every prevented-failure note is something that actually happened, not a hypothetical.

The goal that shapes the procedure: this repo is a black box

An upstream release is our problem, not our consumers'. The target cost to a repo that consumes OKF through this library is a re-run, and nothing else.

That is a design commitment with teeth, not an aspiration:

• Support is additive — a new profile, never a migration. Existing profiles stay byte-stable. A consumer who does not opt in sees no change at all.
• New public parameters are keyword-only with defaults. A consumer's existing positional call sites stay source-compatible across an upgrade. This is why profile is specified as *, profile: BundleProfile = DEFAULT — decided, not yet shipped: measured 2026-07-26, no door takes the argument at all. It turns additivity from something a consumer has to measure into a property of the signature.
• Consumer golden fixtures must not churn. If an upgrade rewrites bytes in a consumer's frozen fixtures, the upgrade is wrong, not the fixture.

The honest boundary — state it every time, never let the black box be oversold. The library absorbs shape changes. It cannot absorb upstream changes to content the consumer authored. v0.2 superseded timestamp with generated.at and the body # Citations list with sources; no library change makes a consumer's existing timestamp field become something else. For that class the deliverable is not absorption but a measured exposure report, per consumer, before they ask — which is what step 5 produces.

Trigger

"Always latest" decays silently: nothing fails when upstream ships and we do not notice. So the re-check is an item on the release checklist — run it at every release of this library, and record the result even when unchanged, because an unrecorded check is indistinguishable from a skipped one.

Since 2026-08-23 the trigger also fires without a release, weekly, from tools/okf_watch.py. See § The weekly watch below. The watch decides whether this procedure runs; it never substitutes for it.

Check GoogleCloudPlatform/open-knowledge-format. That is the canonical home of the spec, the reference agent and the sample bundles as of 2026-08-21. A version bump appears as a commit against SPEC.md §12 and, in the v0.2 round, as an explicit migration commit (okf: migrate format and tooling to Open Knowledge Format v0.2).

Do not check GoogleCloudPlatform/knowledge-catalog, path okf/. That copy is a frozen snapshot by upstream's own notice (6265173, "anything built against it will drift out of date"), and this repo was pinned to it until the 2026-08-23 round. Two consequences, both measured that round and neither hypothetical:

• The two trees have already diverged, and not only in the direction you would expect: the frozen repository's head carries a fix (38c713f, eight tags: values written as sequences rather than as one plain scalar) that the canonical repo does not. The canonical tree is authoritative for the spec; it is not automatically a superset. Note the pin-level precision, measured 2026-08-23: 38c713f is not an ancestor of the old pin 3fcbb9f either, so moving the pin lost nothing — canonical simply ships a form its own frozen predecessor has already repaired. Enumerated in full in docs/plan/okf-2026-08-timestamp-tightening.md § Known divergence.
• A round run against the frozen tree reports "no change" truthfully and uselessly — the exact shape of a negative result that is not a measurement.

Prevents: tracking a tree that has stopped being the thing you promised to track. "Always latest" is a promise about an object with an identity, and the identity can move without the version number moving.

Step 1 — Pin before reading anything

Record the exact commit SHA and read everything at it.

Prevents: reading a moving branch and treating the result as fact. In the v0.2 round both this repo and portfolio-optimiser-commons independently read main, which felt like two confirmations and was one unstable reference read twice. Worse, SPEC.md was edited after the v0.2 migration commit — so "the v0.2 commit" and "the current spec text" were different objects, and either alone would have been a partial answer.

Step 2 — Enumerate the whole REPOSITORY tree, not just okf/, not just SPEC.md

List every file and directory at the pinned commit before deciding what to read. Start at the repository root:

git ls-files | awk -F/ '{print $1}' | sort -u      # top-level entries
git ls-files | sed 's/.*\.//' | sort | uniq -c     # what kinds of file exist

Prevents: assuming the directory you know about is the one that matters. This step has now failed twice at two different scopes, which is why its title names the root rather than a subdirectory:

• v0.2 round, inside okf/. okf/ held SPEC.md, README.md, pyproject.toml, src/, tests/, samples/ and bundles/. The last one held the actual v0.2 example bundles and was not on anyone's list until the tree was enumerated.
• 2026-07-31, one level up — the same mistake against okf/ itself. Scoping enumeration to okf/ (as this step previously instructed) hides that the repository root also carries toolbox/ and samples/. Measured at 3fcbb9f: 265 tracked files, of which 48 are Python and 43 TypeScript, with toolbox/ holding two complete tools (mdcode, enrichment) plus a second copy of an OKF bundle under toolbox/mdcode/demo/okf/catalog/. None of it was on any list.

The cost of getting this wrong is not only unread files: a negative claim derived from a partial enumeration reads exactly like a measured one. "There is no validator in okf/" was recorded as fact and used to plan a weakening of V-A8; okf/src/reference_agent/bundle/document.py:58 defines validate(), and the same module is a working v0.2 reader. Enumerate before concluding that something is absent — see Step 3a.

Step 3 — Read the shipped examples, not only the normative text

This is the step the v0.2 round skipped, and it is the reason this document exists.

A spec says what is permitted. An example shows what upstream actually emits. Those differ, and when we are about to freeze a fixture and hand it to consumers, the second one is what we are being measured against.

Two traps, both hit in the v0.2 round:

• A directory called samples/ was not samples of the format. okf/samples/ga4_merch_store/ is a runner config (README.md + seeds.txt) that drives the reference agent to generate a bundle. It says nothing about output shape. The real examples were in okf/bundles/ — four of them.
• Fetch tooling that converts to markdown will strip or reformat frontmatter. Byte-level questions need byte-level fetches. Check for a BOM, check the trailing newline, check whether a value is quoted. Those three are exactly the properties that break downstream gates and exactly the ones a summarizing fetch destroys.

What reading the four v0.2 bundles produced, none of it derivable from SPEC.md:

Finding	Consequence
No reference bundle declares okf_version at all; root index.md has no frontmatter	§12 is a MAY and upstream declines it. Our stricter posture became a decision instead of an assumption
generated is written for human-authored content (by: human:…)	It can never be an ownership or machine-generated predicate. Governs the collision gate
Real frontmatter is multi-line block YAML: block lists of multi-key mappings, nested mappings, flow sequences, booleans, dates	Re-sized the emitter work. It had been scoped against a list of strings
§7's canonical tool actor is <producer>/<version>, and upstream uses it	Counter-evidence to a recommendation we had already sent to another repo

Step 3a — Run upstream's own reader against our fixture (V-A8)

Every other test in the suite asks whether we agree with ourselves. This one asks an independent implementation, and it is cheap enough that there is no excuse for skipping it.

It is not a dependency. okf/src/reference_agent/bundle/document.py imports only yaml, so it runs standalone under system Python against the pinned clone — nothing is installed, .venv is never touched (the one-runtime-dependency rule and its packaging test both stay intact), and nothing is written.

import sys; sys.path.insert(0, "<clone>/okf/src")
from reference_agent.bundle.document import OKFDocument, trust_tier, is_stale

Assert, over each file of the current version's golden bundle:

1. OKFDocument.parse() accepts our bytes. This is the load-bearing one. Our own parser is line-oriented and reads inline flow forms as opaque strings, so it structurally cannot tell us whether a real YAML consumer recovers them as structures. Only an outside parser can.
2. The values arrive as the shapes the profile intends — generated as a mapping, sources as a list of mappings — not merely as something that parsed.
3. validate() passes. Note what this does and does not buy: REQUIRED_FRONTMATTER_KEYS = ("type",), so it checks one key — measured unchanged at 3fcbb9f and at ad30107. Treating a green validate() as "upstream accepts our bundle" would overclaim badly; the parse in (1) and the shape assertions in (2) are where the signal is.
4. The semantic readers upstream ships run over our frontmatter without raising — at 3fcbb9f and at ad30107: trust_tier (§5.3), normalize_verified (§5.2), is_stale (§5.5). They take the frontmatter mapping, not the document. Passing the OKFDocument raises AttributeError: 'OKFDocument' object has no attribute 'get' — a harness bug that reads exactly like an upstream incompatibility if the traceback is not read. Cost this round: one re-run. OKFDocument.parse() likewise takes the text only; there is no path= keyword.

Record the type each value arrives as, not just that it parsed. A real YAML parser coerces, and the coercions are load-bearing facts about the value space — this is where the v0.2 round's okf_version float finding came from (see the alignment plan's A-E6). Upstream's serialize() also reflows inline flow forms to block form, so an upstream round-trip produces bytes our own line-oriented parser cannot read. That is expected and one-directional; do not "fix" it by emitting block form.

Result, v0.2 round at 3fcbb9f: 13/13 green against examples/ingest-golden-okf-v0-2/.

Result, 2026-08 round at ad30107: 9/9 of our bundle documents parse, across all four goldens, with the three semantic readers running over each without raising; 78/78 of upstream's own bundle documents parse. Re-run 2026-08-23 when the pin moved to ad30107: 9/9 parsed, 0 failures, 27/27 semantic-reader calls clean. Do not compare the two counts — the second harness asserted validate() on root index.md files too, which correctly fails (an index carries no type), so it counts differently by construction. A clean sweep is worth nothing until the harness is shown able to fail: feed it a deliberate type: [unclosed and confirm it raises before you report the zeroes.

Step 4 — Produce the diff, classified

From the spec's own "Changes from vN" section plus the example evidence, classify every change as:

• Breaking — name it, and measure consumer exposure in step 5.
• Additive — new optional fields, new conventional headings.
• Deferred upstream — a format specified but its runtime protocol left to a future revision. The format is supportable; an unspecified runtime is not something to build against. Record it as out of scope on upstream's deferral, not on our preference.
• Observed but undocumented — present in an official bundle, not found in the spec sections read. Record as unresolved. Do not silently promote it to either category. (v0.2 round: a not: family in metrics/gross-margin.md.)
• Tightened in place, no version bump — the normative text narrows what a key may hold, under the same version number. A value that conformed last round does not conform now, and nothing in the version string says so. (2026-08 round: every timestamp-valued key became an ISO 8601 datetime with an explicit offset, and SPEC.md still declares Version 0.2.) Two things follow. First, do not reach for the additive-profile machinery on reflex — there is no new version to express, and minting one would declare a version upstream has not. Second, "conformant with v0.2" stops identifying one document; the spec commit has to be named alongside the version, which is what okf_spec_commit is for.

Step 5 — Measure our exposure and each consumer's, in that order

Ours: grep the profiles, the fixtures, and the emitter for every field the diff touches. Cite file:line. Never reason about a key set — measure it.

Theirs: for each breaking change, state what would have to be true for it to reach them, then ask them to measure it against their real data rather than telling them whether they are affected. In the v0.2 round both portfolio-optimiser-claude and linkedin-studio came back with measured zero exposure and found the framing useful precisely because it was falsifiable.

Step 6 — Plan additively, then pilot before general availability

The plan is a new profile, sequenced deliverables, and stated invariants. Rollout is pilot-first: a pre-release tag to a small pilot set chosen for signal, revised on their feedback, then GA. Flipping the OKF_LATEST alias is the GA event, not a merge side effect.

State each pilot test's expected results before the run, numbered, plus what would surprise us. An expectation we get wrong is a better result than a clean run; only a stated expectation can be falsified by someone else's run.

Name fixtures by repo and path, never by path alone. In the v0.2 round both this repo and portfolio-optimiser-claude had a directory named examples/ingest-golden-file/ holding different fixtures with zero content overlap. A pilot baseline was agreed by name, and the name pointed at two files.

Step 7 — Inform every OKF-consuming repo

All of them, not only the pilots, and not only when something breaks.

Send a bounded loop of directed messages, not a broadcast — a broadcast is inherited by every future repo, which is the wrong lifetime for a version-specific notice.

Each message carries:

1. The version, the pinned commit, and where to read it.
2. The breaking changes by name, with the shape of the exposure — so they can measure themselves rather than take our word.
3. The black-box promise and its boundary: their cost is a re-run; existing profiles are byte-stable; new parameters are keyword-only. And the part we cannot absorb, said plainly.
4. What we want back, if anything, and what happens if they say nothing.

Consumers to cover (2026-07): portfolio-optimiser-commons (spec authorship), portfolio-optimiser, portfolio-optimiser-claude, claude-code-llm-wiki, catalog, okr, linkedin-studio, ms-ai-architect. Re-derive this list each round rather than trusting it — it is a premise like any other.

Step 8 — Close the loop in the repo

The plan doc records the pinned commit, the classified diff, the measured exposure, and every correction a consumer sent back. Corrections stay visible: a premise that survived verification is only known to have survived if the check is recorded, and a claim we withdrew is only safely withdrawn if the withdrawal is written where the claim was.

The weekly watch — tools/okf_watch.py

Answers one question on a schedule: has canonical moved past our pin, and does the move touch anything that bears the contract? On a hit it sends a coord message to this repo and, as FYI, to .claude. On a miss it prints one line and exits 0.

It is deliberately not part of the package: it lives in tools/, outside src/, so it never enters a wheel and a consumer's install surface is unchanged. tests/test_packaging.py holds that as a promise rather than an accident.

Run it:

python3 tools/okf_watch.py                 # the real weekly run
python3 tools/okf_watch.py --dry-run       # print the messages, send nothing
python3 tools/okf_watch.py --pin <sha>     # demonstrate the hit path (implies --dry-run)

Cadence: weekly is the floor. It costs one git fetch against a blob:none mirror, so running it daily is not meaningfully more expensive.

Three properties, and why each is load-bearing

1. A failed call is never an empty result. Every git invocation raises on a non-zero exit and carries stderr. The failure mode this closes is specific and has been met before: grep … | head; echo $? reports the exit status of head, and a query that failed then reads as a query that found nothing.
2. It proves it can find, on every run. Before believing any zero, the watch re-runs its full detect-and-classify path over ad30107^1..ad30107 — a range known to have changed SPEC.md. If that comes back empty the query is broken, and the run aborts loudly instead of reporting a clean sweep. This is Verification-law face 4 made executable rather than remembered.
3. It reports on change, not on state. A JSON state file records which commits have already been announced, keyed on the pin. Moving the pin resets it, because a pin move means everything behind it was absorbed.

Quiet is the enumerated list; signal is not

QUIET_PREFIXES names the paths measured not to bear the contract (.github/, CONTRIBUTING.md, CODE_OF_CONDUCT.md, LICENSE.md, the HTML viewer, generated viz.html). Everything else reports.

The inverse design — enumerate what counts as normative — can only match what upstream has already invented, so anything new falls outside the list and the watch goes quiet about it. That fails in the direction nobody notices. Over-firing is visible and fixable by widening the quiet list; under-firing is neither. If the watch becomes noisy, widen QUIET_PREFIXES. Do not narrow the signal.

README.md is deliberately not quiet: upstream announced the repository move in a README commit, and that move is the change with the longest reach this library has seen.

What the watch cannot do — state this when reporting it

It sees commits. It cannot see meaning.

When upstream tightened v0.2 in place on 2026-08-21, the watch would have fired correctly, because SPEC.md changed. But no commit list says "a value that conformed last month does not conform now", and none says is_stale has reversed for date-only inputs. Those were found by reading the diff and running both readers against the same input. So the watch's output is always run the runbook, never here is your exposure — and the message it sends says so in as many words.

Two further blind spots, named rather than left to be discovered:

• A silent relocation. The last move was caught only because upstream committed a notice to README.md. A move announced anywhere other than this git history is invisible here.
• A tightening with no commit at all — a spec whose meaning is changed by an external document, an errata page, a changed reference implementation shipped under a different repository. Nothing local can see that. The release-checklist trigger, which reads rather than diffs, is the only cover.

The watch narrows the window between an upstream change and our noticing it. It does not close it, and a session that treats a quiet watch as proof that upstream is unchanged has made exactly the mistake the watch was built to prevent.

Invariants this procedure protects

• No profile hard-codes an upstream version.
• No bundle declares a version its shape has not earned. The spec may permit declaring conformance without a checkpoint; we decline. Conform first, claim after.
• DEFAULT states commons' ingest-spec §5 layer — raised there, never patched here.
• STRICT_V1 mirrors a consumer's ratified contract — never changed from here.
• The value of okf_version belongs to the catalog.
• Security stays the guard's domain in every version.