open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

feat(profiles): SEGMENTED_OKF_V0_2 so a segmented bundle can declare its spec

Browse source 
Neither shipped profile could do both: SEGMENTED_V1 names `bundle_id` and not
`okf_version`, OKF_V0_2 names `okf_version` and has no segmentation at all. The
sixth profile is where the two intersect. Additive, as upstream support always
is here -- a new profile, never a migration.

THE INDEX POLICY IS DECIDED, NOT INHERITED, and that is the part that was easy
to get wrong. Measured: OKF_V0_2.index has facets=None and per_directory=False,
while SEGMENTED_V1.index has both. Building on OKF_V0_2's index would have
produced a segmented bundle with no faceted index -- structurally valid, SPEC
conformant, and missing the surface a consumer reads. So the index comes from
SEGMENTED_V1 with both root keys named, and the spec declaration from OKF_V0_2.
The premise is asserted in the suite rather than trusted to stay true.

`SegmentationPolicy.adjudication_key` is the discriminator BETWEEN segmented
profiles. It has to be a field with a value: every 1-to-N branch keys on
`profile.segmentation is not None`, which both segmented profiles satisfy, so a
later step surfacing the adjudication marker on that check would write it into
SEGMENTED_V1 too and move a byte-pinned golden. `None` means the profile does
not surface adjudication state at all.

`okf_version`'s VALUE is not in this module and must never be: a profile names
a key, the caller owns its value (decision E1). A constant here would claim a
decision this library does not own and would be the one thing to chase on every
upstream release. Asserted.

Five existing profiles untouched, each still equal to a freshly constructed
copy, root keys pinned literally, and OKF_LATEST still DEFAULT -- adding a
profile is not a GA event.

Suite 926 -> 931.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-02 14:21:15 +02:00
commit ea64b9915d
2 changed files with 135 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

42
src/llm_ingestion_okf/profiles.py
View file

@ -892,6 +892,17 @@ class SegmentationPolicy:
segment_id_key: str = "segment_id"
offset_key: str = "source_offset"
nav_label: str = "index"
# The discriminator BETWEEN segmented profiles, and the reason it has to be
# a field with a value rather than a presence check: every 1-to-N branch
# keys on `profile.segmentation is not None`, which BOTH segmented profiles
# satisfy. A step that surfaced the adjudication state on that check would
# write it into `SEGMENTED_V1` as well and move a byte-pinned golden.
#
# `None` means this profile does not surface adjudication state at all,
# matching the shape used everywhere else here. When set, it NAMES the
# frontmatter key and index facet; the value written under it is the
# adjudicator's, never this module's.
adjudication_key: str | None = None
@dataclass(frozen=True)
@ -1222,6 +1233,37 @@ SEGMENTED_V1 = BundleProfile(
)
# The sixth profile. A segmented bundle could not declare which upstream spec
# it targets: `SEGMENTED_V1` names `bundle_id`, `OKF_V0_2` names `okf_version`,
# and the two never intersected. Additive, as upstream support always is here --
# a new profile, never a migration of an existing one.
#
# THE INDEX POLICY IS DECIDED HERE, NOT INHERITED, and that is the one thing in
# this construction that is easy to get wrong. Measured: `OKF_V0_2.index` has
# `facets=None` and `per_directory=False`, while `SEGMENTED_V1.index` has both.
# Building the sixth profile on `OKF_V0_2`'s index would have produced a
# segmented bundle with NO faceted index -- structurally valid, conformant, and
# missing the surface a consumer reads. So the index comes from `SEGMENTED_V1`
# with both root keys named, and the spec declaration comes from `OKF_V0_2`.
#
# `okf_version`'s VALUE is not here and must never be: a profile names a key,
# the caller owns its value (decision E1). The value tracks the upstream Google
# version and belongs to catalog; a constant here would claim a decision this
# library does not own, and would be the one thing to chase on every upstream
# release.
SEGMENTED_OKF_V0_2 = BundleProfile(
types=OKF_V0_2.types,
frontmatter=OKF_V0_2.frontmatter,
paths=SEGMENTED_V1.paths,
index=replace(SEGMENTED_V1.index, root_frontmatter=("okf_version", "bundle_id")),
ownership=OKF_V0_2.ownership,
# Constructed rather than `replace`d off `SEGMENTED_V1.segmentation`: that
# attribute is typed `| None`, and the equality is asserted in the suite so
# this stays a fresh copy of the same policy plus the discriminator.
segmentation=SegmentationPolicy(adjudication_key="adjudication"),
)
# "The latest version supported as STABLE", not the latest present in this
# module. It therefore keeps v0.1 UPSTREAM semantics for as long as v0.2 is
# provisional, and flipping it is the GA event — one auditable action rather