feat(check): a skill and a payload naming different bundles is a finding

`okf check` had fifteen rules and none asked whether the skill and the payload
were talking about the same bundle. Reproduced on this HEAD before any code
moved: three pairs reported `conformant: 15 rules over 8 excerpts and 438
withheld entries, 0 findings` -- a skill generated from one corpus against
another corpus's payload, the unfilled template against that payload, and a
payload sharing the skill's `bundle_id` at a foreign `ref`. All three now exit 1
with one `bundle_mismatch` finding over 16 rules.

BOTH halves are compared and the `ref` half is load-bearing: three distinct
builds on this machine carry one `bundle_id`, so an id comparison would pass a
stale skill. SS 3.3: "a version is the producer's assertion; a ref is a fact
about bytes". An identity the rule cannot read is a finding, never a silent
pass -- that is what refuses the unfilled template.

No new field: the identity was already in the generated skill's prose, now
factored into `skill.identity_line` and read back by
`contract_check.skill_identity`. Generated skill bytes unchanged, measured on
both tracked bundles on one interpreter.

The rule's first real find is this repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
identity a reader can act on: 1 of 1. Nine tests that asserted the old, false
conformance now pair a skill with its own bundle's payload.

Measured, nothing else moved: `~/okf-test/dokumenter` `diff -r` empty old
source vs new on one interpreter (52 files, 26 concepts), `okf project` still
byte-equal to `okf build`, K2 pin unmodified and green (453 concepts, ranks
1,1,1,1,1,5), known-negative `{}` unchanged at 9 findings.

Report: docs/2026-09-10-k3-runde15-bundle-mismatch.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-10 23:42:06 +02:00
commit 7cca9e079e
9 changed files with 468 additions and 56 deletions

View file

@ -36,6 +36,7 @@ from __future__ import annotations
import argparse
import json
import re
from collections.abc import Callable, Mapping, Sequence
from dataclasses import dataclass
from pathlib import Path
@ -62,6 +63,23 @@ ADJUDICATION_STATES = ("proposed", "adjudicated", "unknown")
#: SS 6.2, from SPEC SS 5.3: derived from `verified`, lowest to highest.
TRUST_TIERS = ("unverified", "machine-confirmed", "human-reviewed")
#: SS 3.1 and SS 3.3, read back out of the skill's own prose. `okf skill` writes
#: the bundle it was generated for in one authored sentence, and this is the
#: pattern that reads it: there is no structured skill model, only its text.
#: Held to the generator by a test, because the two live in different files.
SKILL_IDENTITY = re.compile(r"for one bundle: `([^`<>]+)` at ref\s+`([^`<>]+)`")
def skill_identity(skill_text: str) -> tuple[str, str] | None:
"""The `(bundle_id, ref)` the skill declares, or `None` when it declares
none a reader could act on. `None` is a finding, never a silent pass: the
unfilled template's `<CORPUS>` and `<REF>` are placeholders, and the
template's own rule is that a copy leaving one unfilled is not configured,
it is unfinished."""
match = SKILL_IDENTITY.search(skill_text)
return (match.group(1), match.group(2)) if match else None
#: The level-2 headings a conformant skill carries. Checked by name because the
#: alternative -- searching the prose for the obligation -- passes on any
#: document that happens to use the words.
@ -174,6 +192,73 @@ def rule_bundle_ref(ctx: Context) -> list[Finding]:
return findings
def rule_bundle_identity(ctx: Context) -> list[Finding]:
"""SS 3.1 and SS 3.3: the skill and the payload must name one bundle.
Added 2026-09-10 on a measurement this checker had published about itself
since 2026-09-08 and not closed: it reported `conformant, 15 rules, 0
findings` for a skill generated from one corpus against a payload assembled
from another, for the unfilled template against that payload, and for a
payload sharing the skill's `bundle_id` at a foreign `ref`.
**Both halves are compared, and the `ref` half is the load-bearing one.**
Three distinct builds on one machine were measured carrying the same
`bundle_id`, so an id comparison would pass a stale skill -- exactly the
case the generated skill warns about in its own words. SS 3.3: "a version
is the producer's assertion; a ref is a fact about bytes".
**It compares a DECLARED identity against a DECLARED identity** and never
opens the bundle, so a payload misreporting its own `ref` passes here.
Proving a ref against bytes is `okf consume --ref`'s job and needs a bundle
path this command deliberately does not take.
A payload that declares no identity at all is `rule_bundle_ref`'s defect,
not this one's: restating it would report one hole twice.
"""
declared = skill_identity(ctx.skill)
if declared is None:
return [
Finding(
"bundle_mismatch",
"the skill declares no readable bundle identity, so no payload "
"can be shown to belong to it; a `<PLACEHOLDER>` left unfilled "
"is not an identity, and neither is its absence (SS 3.1, SS 3.3)",
)
]
skill_id, skill_ref = declared
bundle = _mapping(ctx.payload.get("bundle"))
payload_id, payload_ref = _text(bundle.get("bundle_id")), _text(bundle.get("ref"))
disagreements = [
f"{key} (skill {mine!r}, payload {theirs!r})"
for key, mine, theirs in (
("bundle_id", skill_id, payload_id),
("ref", skill_ref, payload_ref),
)
if theirs and theirs != mine
]
if disagreements:
return [
Finding(
"bundle_mismatch",
"the skill was generated for a bundle the payload does not "
f"describe: {'; '.join(disagreements)}. Every number in the "
"skill was measured against its own bundle's bytes (SS 3.1, "
"SS 3.3)",
)
]
return [
Finding(
"bundle_mismatch",
f"excerpt {position} names bundle {found!r}, which is not the "
f"payload's {payload_id!r}; identity across bundles is the "
"(bundle_id, concept_id) tuple, so an excerpt naming another "
"bundle is another bundle's excerpt (SS 3.1)",
)
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts")))
if payload_id and (found := _text(_mapping(raw).get("bundle_id"))) and found != payload_id
]
def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
if not ctx.payload_is_mapping:
return []
@ -435,6 +520,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
rule_payload_shape,
rule_contract_version,
rule_bundle_ref,
rule_bundle_identity,
rule_excerpt_source_marking,
rule_excerpt_named,
rule_excerpt_states,