feat(check): a skill and a payload naming different bundles is a finding
`okf check` had fifteen rules and none asked whether the skill and the payload
were talking about the same bundle. Reproduced on this HEAD before any code
moved: three pairs reported `conformant: 15 rules over 8 excerpts and 438
withheld entries, 0 findings` -- a skill generated from one corpus against
another corpus's payload, the unfilled template against that payload, and a
payload sharing the skill's `bundle_id` at a foreign `ref`. All three now exit 1
with one `bundle_mismatch` finding over 16 rules.
BOTH halves are compared and the `ref` half is load-bearing: three distinct
builds on this machine carry one `bundle_id`, so an id comparison would pass a
stale skill. SS 3.3: "a version is the producer's assertion; a ref is a fact
about bytes". An identity the rule cannot read is a finding, never a silent
pass -- that is what refuses the unfilled template.
No new field: the identity was already in the generated skill's prose, now
factored into `skill.identity_line` and read back by
`contract_check.skill_identity`. Generated skill bytes unchanged, measured on
both tracked bundles on one interpreter.
The rule's first real find is this repository's own hand-made
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
identity a reader can act on: 1 of 1. Nine tests that asserted the old, false
conformance now pair a skill with its own bundle's payload.
Measured, nothing else moved: `~/okf-test/dokumenter` `diff -r` empty old
source vs new on one interpreter (52 files, 26 concepts), `okf project` still
byte-equal to `okf build`, K2 pin unmodified and green (453 concepts, ranks
1,1,1,1,1,5), known-negative `{}` unchanged at 9 findings.
Report: docs/2026-09-10-k3-runde15-bundle-mismatch.md
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
b5df3355c5
commit
7cca9e079e
9 changed files with 468 additions and 56 deletions
42
CHANGELOG.md
42
CHANGELOG.md
|
|
@ -7,6 +7,48 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||||
|
|
||||||
## [Unreleased]
|
## [Unreleased]
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **`okf check` refuses a skill and a payload that name different bundles
|
||||||
|
(`bundle_mismatch`).** The checker had published this hole about itself since
|
||||||
|
2026-09-08 and not closed it: reproduced on this repository's HEAD, it
|
||||||
|
reported `conformant: 15 rules over 8 excerpts and 438 withheld entries, 0
|
||||||
|
findings` for a skill generated from one corpus against a payload assembled
|
||||||
|
from another; the same line for the UNFILLED template against that payload;
|
||||||
|
and the same line again for a payload sharing the skill's `bundle_id` at a
|
||||||
|
foreign `ref`. All three now exit **1** with one finding. The right pair is
|
||||||
|
untouched at exit 0 with 0 findings, and a payload declaring no identity at
|
||||||
|
all stays `ref_missing`'s defect at 9 findings -- no rule restates another.
|
||||||
|
- **BOTH halves are compared, and the `ref` half is the load-bearing one.**
|
||||||
|
Three distinct builds on one machine were measured carrying the same
|
||||||
|
`bundle_id`, so an id comparison alone would pass a stale skill -- the case
|
||||||
|
the generated skill warns about in its own words ("if the bundle moves, the
|
||||||
|
ref moves with it and this file is stale"). SS 3.3: "a version is the
|
||||||
|
producer's assertion; a ref is a fact about bytes".
|
||||||
|
- **An identity the rule cannot read is a finding, never a silent pass.**
|
||||||
|
That is what refuses the unfilled template, whose `<CORPUS>` and `<REF>`
|
||||||
|
are not an identity. It also refuses the repository's own hand-made
|
||||||
|
`skills/okf-consume/SKILL.md`, which predates `okf skill` and declares no
|
||||||
|
bundle identity a reader can act on -- **1 of 1** shipped hand-made skill,
|
||||||
|
a real find and not a fixture.
|
||||||
|
- **The rule compares a DECLARED identity against a DECLARED identity and
|
||||||
|
never opens the bundle**, so a payload misreporting its own `ref` still
|
||||||
|
passes. Proving a ref against bytes is `okf consume --ref`'s job and needs
|
||||||
|
a bundle path this command deliberately does not take.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **The checker's rule count is 16, not 15**, and `Report.rules_evaluated` is
|
||||||
|
the denominator every report line quotes -- so `15 rules` becomes `16 rules`
|
||||||
|
in every published line. A consumer citing the old number is citing a number
|
||||||
|
that has changed. No payload bytes move: this is the checker, not the
|
||||||
|
pre-pass.
|
||||||
|
- `skill.identity_line` is now the single authored copy of the sentence a
|
||||||
|
generated skill declares its bundle in, read back by
|
||||||
|
`contract_check.skill_identity` and held to it by a test. Generated skill
|
||||||
|
bytes are unchanged -- measured, both tracked bundles byte-identical before
|
||||||
|
and after on the same interpreter.
|
||||||
|
|
||||||
## [0.8.1] — 2026-09-10
|
## [0.8.1] — 2026-09-10
|
||||||
|
|
||||||
### Added
|
### Added
|
||||||
|
|
|
||||||
44
CLAUDE.md
44
CLAUDE.md
|
|
@ -687,13 +687,49 @@ and fixtures, never code.
|
||||||
names on PATH. The template and `docs/consumption-contract.md` (the § 7.4
|
names on PATH. The template and `docs/consumption-contract.md` (the § 7.4
|
||||||
known-positive) are force-included into the wheel from the file they are
|
known-positive) are force-included into the wheel from the file they are
|
||||||
authored in — one authored copy, no committed duplicate.
|
authored in — one authored copy, no committed duplicate.
|
||||||
The form was chosen on a measurement: the contract checker passes the
|
The form was chosen on a measurement: the contract checker passed the
|
||||||
UNFILLED template and passes a skill built for another bundle, so it cannot
|
UNFILLED template and passed a skill built for another bundle, so it could not
|
||||||
tell the two apart — the choice rests on § 5/§ 6.4/§ 7.6 being per-bundle
|
tell the two apart — the choice rests on § 5/§ 6.4/§ 7.6 being per-bundle
|
||||||
numbers a generic skill can only leave as holes or state falsely.
|
numbers a generic skill can only leave as holes or state falsely, and that
|
||||||
|
argument NEVER rested on conformance, so closing the measurement leaves it
|
||||||
|
standing. **That half is CLOSED 2026-09-10 by `bundle_mismatch`, the checker's
|
||||||
|
SIXTEENTH rule** (`RULES` is a tuple and `Report.rules_evaluated` is
|
||||||
|
`len(RULES)`, so every published «15 rules» line is now «16 rules» — a
|
||||||
|
contract change for anyone quoting it). It compares the identity a skill
|
||||||
|
DECLARES against the identity its payload declares, **both `bundle_id` AND
|
||||||
|
`ref`**: three distinct builds on this machine carry the one `bundle_id`
|
||||||
|
`k2-trinn1-20260903` at three refs, so an id-only rule would pass a stale
|
||||||
|
skill, which is the case the generated skill warns about in its own words.
|
||||||
|
SS 3.3 is the ground — «a version is the producer's assertion; a ref is a fact
|
||||||
|
about bytes» — and SS 3.1 for the same question one level down, an excerpt
|
||||||
|
naming a bundle its payload does not. Reproduced on `113b3f8` before any code
|
||||||
|
moved, three pairs at `conformant: 15 rules over 8 excerpts and 438 withheld
|
||||||
|
entries, 0 findings` and rc 0; after, all three at rc 1 with **one** finding
|
||||||
|
over 16 rules — a foreign corpus's payload, the unfilled template, and (the
|
||||||
|
arm that separates a whole rule from half of one) a payload sharing the id at
|
||||||
|
a foreign ref. **An identity the rule cannot READ is a finding, never a silent
|
||||||
|
pass**, or the template passes again. The right pair is untouched at rc 0 / 0
|
||||||
|
findings, and `{}` is unchanged at **9** findings because a payload declaring
|
||||||
|
no identity stays `ref_missing`'s defect — no rule restates another. NO new
|
||||||
|
field was needed: the identity already lived in the generated skill's prose,
|
||||||
|
now one authored copy in `skill.identity_line` read back by
|
||||||
|
`contract_check.skill_identity`, generated skill bytes byte-identical before
|
||||||
|
and after on both tracked bundles. **The rule compares DECLARED against
|
||||||
|
DECLARED and never opens the bundle**, so a payload lying about its own `ref`
|
||||||
|
still passes — that is `okf consume --ref`'s job and needs a bundle path this
|
||||||
|
command deliberately does not take; «closed» means the three measured forms
|
||||||
|
now fell, not that no fourth exists. Report:
|
||||||
|
`docs/2026-09-10-k3-runde15-bundle-mismatch.md`.
|
||||||
The first instantiated consumption skill is `skills/okf-consume/`; the
|
The first instantiated consumption skill is `skills/okf-consume/`; the
|
||||||
measurement behind it, including the control that FAILED, is
|
measurement behind it, including the control that FAILED, is
|
||||||
`docs/2026-09-07-okf-konsumskill-maaling.md`. **The ranking is this
|
`docs/2026-09-07-okf-konsumskill-maaling.md`. **That skill is the new rule's
|
||||||
|
first REAL find**: hand-filled before `okf skill` existed, it declares no
|
||||||
|
bundle identity a reader can act on, so `okf check` now refuses it against its
|
||||||
|
own shipped example payload (rc 1, 1 finding) — **1 of 1** shipped hand-made
|
||||||
|
instantiated skill. Behind it sits a second fact: its prose states K2 numbers
|
||||||
|
while the payload beside it is regenerated from the tracked golden bundle,
|
||||||
|
because K2 cannot ship. What that artefact should BE is an operator question,
|
||||||
|
not a patch. **The ranking is this
|
||||||
repository's own choice** — the contract binds a payload, not a retrieval
|
repository's own choice** — the contract binds a payload, not a retrieval
|
||||||
algorithm (§ 10) — and it has FOUR optional widenings. Three are **off by
|
algorithm (§ 10) — and it has FOUR optional widenings. Three are **off by
|
||||||
default** and keep the default payload byte-identical; the fourth
|
default** and keep the default payload byte-identical; the fourth
|
||||||
|
|
|
||||||
|
|
@ -531,8 +531,9 @@ puts the project somewhere other than the current directory.
|
||||||
|
|
||||||
Measured 2026-09-09 from a fresh `uv tool install` with this repository nowhere
|
Measured 2026-09-09 from a fresh `uv tool install` with this repository nowhere
|
||||||
on the path: 5 documents in, **26** concepts out, a skill carrying **0** paths
|
on the path: 5 documents in, **26** concepts out, a skill carrying **0** paths
|
||||||
into any checkout, and `okf check` conformant on its own payload (15 rules, 0
|
into any checkout, and `okf check` conformant on its own payload (16 rules, 0
|
||||||
findings). The 2026-09-08 run of the same measurement reported 15 concepts, and
|
findings; the count was 15 until `bundle_mismatch` landed 2026-09-10, and the
|
||||||
|
pair was re-measured rather than carried over). The 2026-09-08 run of the same measurement reported 15 concepts, and
|
||||||
that number was the defect rather than the result: `okf project` was calling
|
that number was the defect rather than the result: `okf project` was calling
|
||||||
`build()` as a function and reading its signature's defaults, which disagreed
|
`build()` as a function and reading its signature's defaults, which disagreed
|
||||||
with argparse's on two flags. Two tests now hold the two default sets equal. Before that day the same result took a `PYTHONPATH`, a snapshot of a
|
with argparse's on two flags. Two tests now hold the two default sets equal. Before that day the same result took a `PYTHONPATH`, a snapshot of a
|
||||||
|
|
|
||||||
155
docs/2026-09-10-k3-runde15-bundle-mismatch.md
Normal file
155
docs/2026-09-10-k3-runde15-bundle-mismatch.md
Normal file
|
|
@ -0,0 +1,155 @@
|
||||||
|
# K3 round 15: which bundle was this skill generated for?
|
||||||
|
|
||||||
|
**Date:** 2026-09-10 · **HEAD at start:** `113b3f8` (`v0.8.1` tagged locally on
|
||||||
|
`3daf983`) · **Red test:** `b5df335` · No push, no tag, no version bump.
|
||||||
|
|
||||||
|
## 1. What was wrong
|
||||||
|
|
||||||
|
`okf skill --help` had carried this sentence about its own tooling since
|
||||||
|
2026-09-08:
|
||||||
|
|
||||||
|
> Measured 2026-09-08: the contract checker passes the UNFILLED template
|
||||||
|
> against a real payload (exit 0, 15 rules, 0 findings), and passes a skill
|
||||||
|
> built for a different bundle against this one's payload. So the checker
|
||||||
|
> cannot tell the two forms apart.
|
||||||
|
|
||||||
|
Documented, not closed. `okf check` had fifteen rules and none of them asked
|
||||||
|
whether the skill and the payload were talking about the same bundle.
|
||||||
|
|
||||||
|
## 2. Reproduced first, on this HEAD
|
||||||
|
|
||||||
|
Every figure below was re-measured on `113b3f8` in a frozen export
|
||||||
|
(`git archive` + `uv sync --frozen --extra extract`), because the order's
|
||||||
|
figures were taken on `958e9bc`. **Zero divergence**: all five rows reproduced
|
||||||
|
byte for byte, including the exact report lines.
|
||||||
|
|
||||||
|
Two real artefacts, both read and never written: a SKILL generated by
|
||||||
|
`okf skill` from the pinned K2 bundle (`k2-trinn1-20260903` at
|
||||||
|
`sha256-tree:18ae18ab...`, 453 concepts) and a payload from an unrelated corpus
|
||||||
|
(`vegnormal-n100-2023` at `sha256-tree:da6b8204...`, 446 concepts).
|
||||||
|
|
||||||
|
| # | pair | rc before | rc after | line after |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| A | skill(k2) x payload(vegnormal-n100) | 0 | **1** | `NOT conformant: 16 rules over 8 excerpts and 438 withheld entries, 1 findings` |
|
||||||
|
| B | unfilled template x payload(vegnormal-n100) | 0 | **1** | same denominators, 1 finding |
|
||||||
|
| C | skill(k2) x payload(k2 id, foreign ref) | 0 | **1** | same denominators, 1 finding, message names **`ref` only** |
|
||||||
|
| KP | skill(k2) x its own payload | 0 | **0** | `conformant: 16 rules over 8 excerpts and 445 withheld entries, 0 findings` |
|
||||||
|
| KN | skill(k2) x `{}` | 1 | **1** | `NOT conformant: 16 rules over 0 excerpts and 0 withheld entries, 9 findings` |
|
||||||
|
|
||||||
|
Before-rows were measured with the frozen export of `113b3f8`; after-rows with
|
||||||
|
the working tree. Exit codes were captured directly (`out=$(okf check ...);
|
||||||
|
rc=$?`) and never through a pipe, which would report `tail`'s status.
|
||||||
|
|
||||||
|
## 3. Why the rule compares the ref and not only the id
|
||||||
|
|
||||||
|
Three distinct builds on this machine carry the **same** `bundle_id`
|
||||||
|
`k2-trinn1-20260903` at three different refs (`f14872a0...`, `9a4e5561...`,
|
||||||
|
`18ae18ab...`). So `bundle_id` does not identify the bytes, and an id-only rule
|
||||||
|
would pass a stale skill at exit 0 -- the case the generated skill warns about
|
||||||
|
in its own words: "If the bundle moves, the ref moves with it and this file is
|
||||||
|
stale -- regenerate". `docs/consumption-contract.md` SS 3.3 is the ground:
|
||||||
|
"a version is the producer's assertion; a ref is a fact about bytes". SS 3.1 is
|
||||||
|
the ground for the excerpt-level half: "identity across bundles is the tuple,
|
||||||
|
never `concept_id` alone".
|
||||||
|
|
||||||
|
**Row C is the arm that separates a whole rule from half of one.** It is the
|
||||||
|
only row where an id comparison and an identity comparison disagree.
|
||||||
|
|
||||||
|
## 4. What was built
|
||||||
|
|
||||||
|
One new rule, `contract_check.rule_bundle_identity`, one new code
|
||||||
|
`bundle_mismatch`, `RULES` 15 -> 16. No existing rule weakened, no existing code
|
||||||
|
renamed, no rule short-circuiting another.
|
||||||
|
|
||||||
|
**No new field was needed, and that was verified rather than assumed.** The
|
||||||
|
identity is already in the generated skill's prose. It was factored into one
|
||||||
|
authored copy at `skill.py:395` (`identity_line`), which `_rewrite` uses at the
|
||||||
|
place it was previously inlined, and `contract_check.skill_identity` reads back.
|
||||||
|
Generated skill bytes are **unchanged**: both tracked bundles produce
|
||||||
|
byte-identical skills before and after when rendered on the same interpreter
|
||||||
|
(`e67827a6...` and `890aeac2...` both ways). The earlier apparent 9-byte
|
||||||
|
difference was the bundle path's length in two different export trees, not the
|
||||||
|
code -- measured and discarded.
|
||||||
|
|
||||||
|
Three cases, in one rule:
|
||||||
|
|
||||||
|
1. The skill declares no readable identity -> finding. This is what refuses the
|
||||||
|
unfilled template, and it is load-bearing: a silent pass here would let row B
|
||||||
|
through and leave the round undone.
|
||||||
|
2. The declared identities disagree in `bundle_id`, in `ref`, or in both ->
|
||||||
|
one finding naming which halves disagreed.
|
||||||
|
3. An excerpt names a bundle the payload does not -> one finding per excerpt.
|
||||||
|
Same question one level down (SS 3.1).
|
||||||
|
|
||||||
|
A payload declaring no identity at all is left to `rule_bundle_ref`: restating
|
||||||
|
it would report one hole twice. Measured: the known-negative `{}` is unchanged
|
||||||
|
at **9 findings**, and `bundle_mismatch` is not among them.
|
||||||
|
|
||||||
|
## 5. What the rule found in this repository
|
||||||
|
|
||||||
|
`skills/okf-consume/SKILL.md` -- the first instantiated skill, filled in by hand
|
||||||
|
before `okf skill` existed -- **declares no bundle identity a reader can act
|
||||||
|
on**, so `okf check` now refuses it against its own shipped example payload:
|
||||||
|
`NOT conformant: 16 rules over 3 excerpts and 0 withheld entries, 1 findings`,
|
||||||
|
rc 1. **1 of 1** shipped hand-made instantiated skill. A second fact sits behind
|
||||||
|
it: that skill's prose states K2 numbers while the payload shipped beside it is
|
||||||
|
regenerated from the tracked golden bundle, because K2 cannot ship. Repairing it
|
||||||
|
is a decision about what that artefact is for, and is left to the operator
|
||||||
|
rather than taken here.
|
||||||
|
|
||||||
|
Nine tests asserted the old, false conformance by pairing a skill with a
|
||||||
|
payload from another bundle (or with the template standing in for a skill).
|
||||||
|
They now pair correctly -- the checker's control is an INSTANTIATED skill, since
|
||||||
|
the template can no longer be one.
|
||||||
|
|
||||||
|
## 6. Nothing else moved, measured on the bytes
|
||||||
|
|
||||||
|
- **Build output identical.** `~/okf-test/dokumenter` built with the old source
|
||||||
|
and the new source **on the same interpreter**: `diff -r` empty, 52 files, 26
|
||||||
|
concepts. (Comparing across the two venvs shows one line in `log.md` -- the
|
||||||
|
converter's own path and Python version -- which is the venv, not this change.)
|
||||||
|
- **`okf project` is still byte-equal to `okf build`** on that folder: `diff -r`
|
||||||
|
empty against `<out>/.okf/<id>`.
|
||||||
|
- **K2 pin untouched and green.** `tests/test_default_bundle_pin.py` is
|
||||||
|
unmodified and RAN (7 passed, not skipped): 453 concepts, per-row ranks
|
||||||
|
`(1, 1, 1, 1, 1, 5)`.
|
||||||
|
- **The repository's own generated pair is conformant at the full 16 rules**:
|
||||||
|
`conformant: 16 rules over 2 excerpts and 1 withheld entries, 0 findings`, and
|
||||||
|
the `okf project` skill likewise, `16 rules over 8 excerpts and 18 withheld
|
||||||
|
entries, 0 findings`.
|
||||||
|
- **Gates**, ruff 0.16.6: `ruff check src tests tools` clean, `ruff format
|
||||||
|
--check .` 119 files, `mypy src` clean, `pytest -q` **1593 passed, 1 skipped**
|
||||||
|
(baseline 1582 + 1 on this HEAD; the frozen export additionally fails
|
||||||
|
`test_segmented_golden.py::test_the_four_existing_goldens_are_untouched`,
|
||||||
|
which is the export artefact and was red before this round too).
|
||||||
|
|
||||||
|
## 7. Honesty limits
|
||||||
|
|
||||||
|
- **The rule compares a DECLARED identity against a DECLARED identity and never
|
||||||
|
opens the bundle.** A payload that misreports its own `ref` passes. Proving a
|
||||||
|
ref against bytes is `okf consume --ref`'s job and needs a bundle path this
|
||||||
|
command deliberately does not take. Nothing here makes a ref true.
|
||||||
|
- **The denominator for the reproduction is two real SKILL/payload pairs plus
|
||||||
|
arms built from this repository's own two tracked bundles -- not a corpus.**
|
||||||
|
Every fixture is generated from bundles already in the tree, so no sentence
|
||||||
|
from any document, no corpus path and no gold question reaches a tracked file.
|
||||||
|
- **The excerpt-level arm is built, not found.** Across the two real payloads
|
||||||
|
reproduced this round, **0 of 16** excerpts carried a `bundle_id` differing
|
||||||
|
from the payload's own. The arm covers a case that has not been observed in
|
||||||
|
the wild; it is cheap, and its absence has a denominator.
|
||||||
|
- **Conformance is the floor, never the proof.** The checker's own docstring
|
||||||
|
says why: the division of labour (SS 2) and the prohibitions (SS 9) are
|
||||||
|
properties of a RUN -- whether an agent went looking for context the pre-pass
|
||||||
|
withheld -- and no static check reaches them. A green `okf check` is not a
|
||||||
|
quality gate, and this repository measured that directly on
|
||||||
|
2026-09-10: three arms over one corpus all returned 0 findings from `okf
|
||||||
|
check` while their hit@k ranged from 6 of 6 to 0 of 6.
|
||||||
|
- **"Closed" means the three measured forms now fail, not that no fourth form
|
||||||
|
exists.** A skill whose identity sentence has been edited by hand to name a
|
||||||
|
bundle it was not generated from would pass. So would a payload and a skill
|
||||||
|
that agree with each other and disagree with the bytes.
|
||||||
|
- **The rule count is a contract change downstream.** Every consumer quoting
|
||||||
|
"15 rules" is quoting a number that has changed, and one pair
|
||||||
|
`portfolio-optimiser` already runs goes from rc 0 to rc 1. That is the rule
|
||||||
|
working, not a regression there -- but it is their build's exit code, so it
|
||||||
|
was sent to them rather than left to be discovered.
|
||||||
|
|
@ -36,6 +36,7 @@ from __future__ import annotations
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
import json
|
import json
|
||||||
|
import re
|
||||||
from collections.abc import Callable, Mapping, Sequence
|
from collections.abc import Callable, Mapping, Sequence
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
@ -62,6 +63,23 @@ ADJUDICATION_STATES = ("proposed", "adjudicated", "unknown")
|
||||||
#: SS 6.2, from SPEC SS 5.3: derived from `verified`, lowest to highest.
|
#: SS 6.2, from SPEC SS 5.3: derived from `verified`, lowest to highest.
|
||||||
TRUST_TIERS = ("unverified", "machine-confirmed", "human-reviewed")
|
TRUST_TIERS = ("unverified", "machine-confirmed", "human-reviewed")
|
||||||
|
|
||||||
|
#: SS 3.1 and SS 3.3, read back out of the skill's own prose. `okf skill` writes
|
||||||
|
#: the bundle it was generated for in one authored sentence, and this is the
|
||||||
|
#: pattern that reads it: there is no structured skill model, only its text.
|
||||||
|
#: Held to the generator by a test, because the two live in different files.
|
||||||
|
SKILL_IDENTITY = re.compile(r"for one bundle: `([^`<>]+)` at ref\s+`([^`<>]+)`")
|
||||||
|
|
||||||
|
|
||||||
|
def skill_identity(skill_text: str) -> tuple[str, str] | None:
|
||||||
|
"""The `(bundle_id, ref)` the skill declares, or `None` when it declares
|
||||||
|
none a reader could act on. `None` is a finding, never a silent pass: the
|
||||||
|
unfilled template's `<CORPUS>` and `<REF>` are placeholders, and the
|
||||||
|
template's own rule is that a copy leaving one unfilled is not configured,
|
||||||
|
it is unfinished."""
|
||||||
|
match = SKILL_IDENTITY.search(skill_text)
|
||||||
|
return (match.group(1), match.group(2)) if match else None
|
||||||
|
|
||||||
|
|
||||||
#: The level-2 headings a conformant skill carries. Checked by name because the
|
#: The level-2 headings a conformant skill carries. Checked by name because the
|
||||||
#: alternative -- searching the prose for the obligation -- passes on any
|
#: alternative -- searching the prose for the obligation -- passes on any
|
||||||
#: document that happens to use the words.
|
#: document that happens to use the words.
|
||||||
|
|
@ -174,6 +192,73 @@ def rule_bundle_ref(ctx: Context) -> list[Finding]:
|
||||||
return findings
|
return findings
|
||||||
|
|
||||||
|
|
||||||
|
def rule_bundle_identity(ctx: Context) -> list[Finding]:
|
||||||
|
"""SS 3.1 and SS 3.3: the skill and the payload must name one bundle.
|
||||||
|
|
||||||
|
Added 2026-09-10 on a measurement this checker had published about itself
|
||||||
|
since 2026-09-08 and not closed: it reported `conformant, 15 rules, 0
|
||||||
|
findings` for a skill generated from one corpus against a payload assembled
|
||||||
|
from another, for the unfilled template against that payload, and for a
|
||||||
|
payload sharing the skill's `bundle_id` at a foreign `ref`.
|
||||||
|
|
||||||
|
**Both halves are compared, and the `ref` half is the load-bearing one.**
|
||||||
|
Three distinct builds on one machine were measured carrying the same
|
||||||
|
`bundle_id`, so an id comparison would pass a stale skill -- exactly the
|
||||||
|
case the generated skill warns about in its own words. SS 3.3: "a version
|
||||||
|
is the producer's assertion; a ref is a fact about bytes".
|
||||||
|
|
||||||
|
**It compares a DECLARED identity against a DECLARED identity** and never
|
||||||
|
opens the bundle, so a payload misreporting its own `ref` passes here.
|
||||||
|
Proving a ref against bytes is `okf consume --ref`'s job and needs a bundle
|
||||||
|
path this command deliberately does not take.
|
||||||
|
|
||||||
|
A payload that declares no identity at all is `rule_bundle_ref`'s defect,
|
||||||
|
not this one's: restating it would report one hole twice.
|
||||||
|
"""
|
||||||
|
declared = skill_identity(ctx.skill)
|
||||||
|
if declared is None:
|
||||||
|
return [
|
||||||
|
Finding(
|
||||||
|
"bundle_mismatch",
|
||||||
|
"the skill declares no readable bundle identity, so no payload "
|
||||||
|
"can be shown to belong to it; a `<PLACEHOLDER>` left unfilled "
|
||||||
|
"is not an identity, and neither is its absence (SS 3.1, SS 3.3)",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
skill_id, skill_ref = declared
|
||||||
|
bundle = _mapping(ctx.payload.get("bundle"))
|
||||||
|
payload_id, payload_ref = _text(bundle.get("bundle_id")), _text(bundle.get("ref"))
|
||||||
|
disagreements = [
|
||||||
|
f"{key} (skill {mine!r}, payload {theirs!r})"
|
||||||
|
for key, mine, theirs in (
|
||||||
|
("bundle_id", skill_id, payload_id),
|
||||||
|
("ref", skill_ref, payload_ref),
|
||||||
|
)
|
||||||
|
if theirs and theirs != mine
|
||||||
|
]
|
||||||
|
if disagreements:
|
||||||
|
return [
|
||||||
|
Finding(
|
||||||
|
"bundle_mismatch",
|
||||||
|
"the skill was generated for a bundle the payload does not "
|
||||||
|
f"describe: {'; '.join(disagreements)}. Every number in the "
|
||||||
|
"skill was measured against its own bundle's bytes (SS 3.1, "
|
||||||
|
"SS 3.3)",
|
||||||
|
)
|
||||||
|
]
|
||||||
|
return [
|
||||||
|
Finding(
|
||||||
|
"bundle_mismatch",
|
||||||
|
f"excerpt {position} names bundle {found!r}, which is not the "
|
||||||
|
f"payload's {payload_id!r}; identity across bundles is the "
|
||||||
|
"(bundle_id, concept_id) tuple, so an excerpt naming another "
|
||||||
|
"bundle is another bundle's excerpt (SS 3.1)",
|
||||||
|
)
|
||||||
|
for position, raw in enumerate(_sequence(ctx.payload.get("excerpts")))
|
||||||
|
if payload_id and (found := _text(_mapping(raw).get("bundle_id"))) and found != payload_id
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
|
def rule_excerpt_source_marking(ctx: Context) -> list[Finding]:
|
||||||
if not ctx.payload_is_mapping:
|
if not ctx.payload_is_mapping:
|
||||||
return []
|
return []
|
||||||
|
|
@ -435,6 +520,7 @@ RULES: tuple[Callable[[Context], list[Finding]], ...] = (
|
||||||
rule_payload_shape,
|
rule_payload_shape,
|
||||||
rule_contract_version,
|
rule_contract_version,
|
||||||
rule_bundle_ref,
|
rule_bundle_ref,
|
||||||
|
rule_bundle_identity,
|
||||||
rule_excerpt_source_marking,
|
rule_excerpt_source_marking,
|
||||||
rule_excerpt_named,
|
rule_excerpt_named,
|
||||||
rule_excerpt_states,
|
rule_excerpt_states,
|
||||||
|
|
|
||||||
|
|
@ -6,11 +6,20 @@ Filling it by hand is what produced `skills/okf-consume/` for one corpus. This
|
||||||
command does the same thing for any bundle, from values it measures rather than
|
command does the same thing for any bundle, from values it measures rather than
|
||||||
values someone remembered.
|
values someone remembered.
|
||||||
|
|
||||||
**Why a generator rather than one generic skill.** Measured 2026-09-08: the
|
**Why a generator rather than one generic skill.** The measurement this
|
||||||
contract checker passes the UNFILLED template against a real payload (exit 0, 15
|
paragraph used to rest on is CLOSED 2026-09-10. It read: the contract checker
|
||||||
rules, 0 findings), and passes a skill built for a different bundle against this
|
passes the UNFILLED template against a real payload (exit 0, 15 rules, 0
|
||||||
one's payload. So the checker cannot tell the two forms apart, and the choice
|
findings), and passes a skill built for a different bundle against this one's
|
||||||
could not be made on conformance. It was made on what the skill has to state:
|
payload. `contract_check.rule_bundle_identity` now compares the identity a
|
||||||
|
skill declares with the identity its payload declares, so all three measured
|
||||||
|
pairs are refused at exit 1 with one `bundle_mismatch` finding over 16 rules:
|
||||||
|
a skill against another bundle's payload, the unfilled template against a real
|
||||||
|
payload, and -- the arm an id comparison would miss -- a payload sharing the
|
||||||
|
skill's `bundle_id` at a foreign `ref`. The right pair is untouched at exit 0
|
||||||
|
with 0 findings.
|
||||||
|
|
||||||
|
**The argument for a generator never rested on conformance, and still does
|
||||||
|
not.** It was made on what the skill has to state:
|
||||||
§ 5's denominators, § 7.6's breaking point and § 6.4's conditional-field list
|
§ 5's denominators, § 7.6's breaking point and § 6.4's conditional-field list
|
||||||
are all per-bundle numbers. A generic skill can either leave them as holes -- the
|
are all per-bundle numbers. A generic skill can either leave them as holes -- the
|
||||||
template's own definition of unfinished -- or carry another corpus's numbers,
|
template's own definition of unfinished -- or carry another corpus's numbers,
|
||||||
|
|
@ -383,6 +392,18 @@ def render(
|
||||||
return header + text, payload
|
return header + text, payload
|
||||||
|
|
||||||
|
|
||||||
|
def identity_line(bundle_id: str, ref: str) -> str:
|
||||||
|
"""The one sentence that says which bundle a generated skill belongs to.
|
||||||
|
|
||||||
|
Authored here because the generator writes it, and read back by
|
||||||
|
`contract_check.skill_identity`, whose `bundle_mismatch` rule is the reason
|
||||||
|
it has to be findable rather than merely present. Two copies of this
|
||||||
|
sentence would drift, and the copy nobody reads is the one that goes wrong,
|
||||||
|
so the coupling has its own test.
|
||||||
|
"""
|
||||||
|
return f"generated by `okf skill` for one bundle: `{bundle_id}` at ref\n`{ref}`"
|
||||||
|
|
||||||
|
|
||||||
def _description(bundle_id: str, total: int, ref: str) -> str:
|
def _description(bundle_id: str, total: int, ref: str) -> str:
|
||||||
return (
|
return (
|
||||||
f"Answer one question about the OKF bundle `{bundle_id}` ({total} concepts, "
|
f"Answer one question about the OKF bundle `{bundle_id}` ({total} concepts, "
|
||||||
|
|
@ -417,9 +438,9 @@ def _rewrite(
|
||||||
(
|
(
|
||||||
TEMPLATE_HEADER,
|
TEMPLATE_HEADER,
|
||||||
"**This file is an instantiated copy of "
|
"**This file is an instantiated copy of "
|
||||||
"`skills/okf-consume-template/SKILL.md`,** generated by "
|
"`skills/okf-consume-template/SKILL.md`,** "
|
||||||
f"`okf skill` for one bundle: `{bundle_id}` at ref\n"
|
f"{identity_line(bundle_id, ref)}. Every value below was measured "
|
||||||
f"`{ref}`. Every value below was measured against those bytes. If the\n"
|
"against those bytes. If the\n"
|
||||||
"bundle moves, the ref moves with it and this file is stale — regenerate\n"
|
"bundle moves, the ref moves with it and this file is stale — regenerate\n"
|
||||||
"it rather than editing a number here. The section headings are fixed:\n"
|
"it rather than editing a number here. The section headings are fixed:\n"
|
||||||
"the contract checker reads them by name.",
|
"the contract checker reads them by name.",
|
||||||
|
|
|
||||||
|
|
@ -33,6 +33,7 @@ PROJECT_ROOT = Path(__file__).resolve().parents[1]
|
||||||
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
||||||
|
|
||||||
import okf_contract_check # noqa: E402
|
import okf_contract_check # noqa: E402
|
||||||
|
import okf_skill # noqa: E402
|
||||||
|
|
||||||
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
|
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
|
||||||
EXAMPLE = TEMPLATE.parent / "references" / "example-payload.json"
|
EXAMPLE = TEMPLATE.parent / "references" / "example-payload.json"
|
||||||
|
|
@ -44,6 +45,32 @@ def load_example() -> dict[str, Any]:
|
||||||
return payload
|
return payload
|
||||||
|
|
||||||
|
|
||||||
|
def declaring(skill_text: str, payload: dict[str, Any]) -> str:
|
||||||
|
"""The template with the one hole `bundle_mismatch` reads filled in.
|
||||||
|
|
||||||
|
The template alone can no longer be a conformant skill: `<CORPUS>` and
|
||||||
|
`<REF>` are placeholders, and an identity the checker cannot read is a
|
||||||
|
finding by design. The sentence is the generator's own, taken from it
|
||||||
|
rather than copied, so the two cannot drift apart.
|
||||||
|
"""
|
||||||
|
bundle = payload["bundle"]
|
||||||
|
return skill_text.replace(
|
||||||
|
okf_skill.TEMPLATE_HEADER,
|
||||||
|
okf_skill.identity_line(bundle["bundle_id"], bundle["ref"]) + ".",
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def instantiated() -> str:
|
||||||
|
return declaring(TEMPLATE.read_text(encoding="utf-8"), load_example())
|
||||||
|
|
||||||
|
|
||||||
|
def instantiated_file(tmp_path: Path) -> Path:
|
||||||
|
target = tmp_path / "SKILL.md"
|
||||||
|
target.write_text(instantiated(), encoding="utf-8")
|
||||||
|
return target
|
||||||
|
|
||||||
|
|
||||||
def codes(skill_text: str, payload: Any) -> list[str]:
|
def codes(skill_text: str, payload: Any) -> list[str]:
|
||||||
return [finding.code for finding in okf_contract_check.check(skill_text, payload).findings]
|
return [finding.code for finding in okf_contract_check.check(skill_text, payload).findings]
|
||||||
|
|
||||||
|
|
@ -51,16 +78,20 @@ def codes(skill_text: str, payload: Any) -> list[str]:
|
||||||
# --- The known-positive, which every negative arm depends on ----------------
|
# --- The known-positive, which every negative arm depends on ----------------
|
||||||
|
|
||||||
|
|
||||||
def test_shipped_template_and_example_payload_are_conformant() -> None:
|
def test_an_instantiated_skill_and_its_own_example_payload_are_conformant() -> None:
|
||||||
"""The CONTROL. A checker that cannot pass anything proves nothing below."""
|
"""The CONTROL. A checker that cannot pass anything proves nothing below.
|
||||||
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), load_example())
|
|
||||||
|
It is the INSTANTIATED skill, not the template: since 2026-09-10 a skill
|
||||||
|
declaring no readable bundle identity is `bundle_mismatch`, measured on
|
||||||
|
the template itself in `test_bundle_identity.py`."""
|
||||||
|
report = okf_contract_check.check(instantiated(), load_example())
|
||||||
assert report.findings == ()
|
assert report.findings == ()
|
||||||
|
|
||||||
|
|
||||||
def test_the_report_carries_its_denominators() -> None:
|
def test_the_report_carries_its_denominators() -> None:
|
||||||
""" "Conformant" without a denominator is unmeasured, per the contract's own
|
""" "Conformant" without a denominator is unmeasured, per the contract's own
|
||||||
section 5. The report says how many rules ran over how many units."""
|
section 5. The report says how many rules ran over how many units."""
|
||||||
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), load_example())
|
report = okf_contract_check.check(instantiated(), load_example())
|
||||||
assert report.rules_evaluated == len(okf_contract_check.RULES)
|
assert report.rules_evaluated == len(okf_contract_check.RULES)
|
||||||
assert report.rules_evaluated > 0
|
assert report.rules_evaluated > 0
|
||||||
assert report.excerpts_examined == 3
|
assert report.excerpts_examined == 3
|
||||||
|
|
@ -76,112 +107,112 @@ def test_the_report_carries_its_denominators() -> None:
|
||||||
def test_broken_denominator_identity_is_named() -> None:
|
def test_broken_denominator_identity_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["denominators"]["considered"] = 6
|
payload["denominators"]["considered"] = 6
|
||||||
assert "denominator_identity" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "denominator_identity" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_delivered_count_disagreeing_with_the_list_is_named() -> None:
|
def test_delivered_count_disagreeing_with_the_list_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["excerpts"] = payload["excerpts"][:2]
|
payload["excerpts"] = payload["excerpts"][:2]
|
||||||
assert "denominator_list_mismatch" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "denominator_list_mismatch" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_missing_adjudication_key_is_not_absence() -> None:
|
def test_a_missing_adjudication_key_is_not_absence() -> None:
|
||||||
"""The contract's sharpest rule: `unknown` is written, never omitted."""
|
"""The contract's sharpest rule: `unknown` is written, never omitted."""
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
del payload["excerpts"][0]["adjudication"]
|
del payload["excerpts"][0]["adjudication"]
|
||||||
assert "state_missing" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "state_missing" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_an_adjudication_value_outside_the_closed_set_is_named() -> None:
|
def test_an_adjudication_value_outside_the_closed_set_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["excerpts"][0]["adjudication"] = "absent"
|
payload["excerpts"][0]["adjudication"] = "absent"
|
||||||
assert "state_not_in_set" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "state_not_in_set" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_trust_tier_outside_the_closed_set_is_named() -> None:
|
def test_a_trust_tier_outside_the_closed_set_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["excerpts"][1]["trust_tier"] = "trusted"
|
payload["excerpts"][1]["trust_tier"] = "trusted"
|
||||||
assert "state_not_in_set" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "state_not_in_set" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.parametrize("key", ["bundle_id", "concept_id", "sha256"])
|
@pytest.mark.parametrize("key", ["bundle_id", "concept_id", "sha256"])
|
||||||
def test_an_excerpt_missing_its_source_marking_is_named(key: str) -> None:
|
def test_an_excerpt_missing_its_source_marking_is_named(key: str) -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
del payload["excerpts"][0][key]
|
del payload["excerpts"][0][key]
|
||||||
assert "source_marking_incomplete" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "source_marking_incomplete" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_bundle_without_a_ref_is_named() -> None:
|
def test_a_bundle_without_a_ref_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["bundle"]["ref"] = ""
|
payload["bundle"]["ref"] = ""
|
||||||
assert "ref_missing" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "ref_missing" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_withheld_concept_without_a_rule_is_an_undeclared_cut() -> None:
|
def test_a_withheld_concept_without_a_rule_is_an_undeclared_cut() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
del payload["withheld"][0]["rule"]
|
del payload["withheld"][0]["rule"]
|
||||||
assert "cut_undeclared" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "cut_undeclared" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_spending_over_the_limit_is_the_gate_firing() -> None:
|
def test_spending_over_the_limit_is_the_gate_firing() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["budget"]["spent"] = payload["budget"]["limit"] + 1
|
payload["budget"]["spent"] = payload["budget"]["limit"] + 1
|
||||||
assert "budget_exceeded" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "budget_exceeded" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_an_instrument_that_missed_its_known_positive_is_named() -> None:
|
def test_an_instrument_that_missed_its_known_positive_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["budget"]["known_positive"]["measured"] = 10405
|
payload["budget"]["known_positive"]["measured"] = 10405
|
||||||
assert "instrument_unvalidated" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "instrument_unvalidated" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_budget_without_an_instrument_is_named() -> None:
|
def test_a_budget_without_an_instrument_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
payload["budget"]["instrument"] = ""
|
payload["budget"]["instrument"] = ""
|
||||||
assert "budget_undeclared" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "budget_undeclared" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_payload_naming_no_contract_revision_is_named() -> None:
|
def test_a_payload_naming_no_contract_revision_is_named() -> None:
|
||||||
payload = load_example()
|
payload = load_example()
|
||||||
del payload["contract"]
|
del payload["contract"]
|
||||||
assert "contract_unversioned" in codes(TEMPLATE.read_text(encoding="utf-8"), payload)
|
assert "contract_unversioned" in codes(instantiated(), payload)
|
||||||
|
|
||||||
|
|
||||||
def test_a_payload_that_is_not_a_mapping_is_refused_not_crashed() -> None:
|
def test_a_payload_that_is_not_a_mapping_is_refused_not_crashed() -> None:
|
||||||
assert "payload_invalid" in codes(TEMPLATE.read_text(encoding="utf-8"), ["not", "a", "mapping"])
|
assert "payload_invalid" in codes(instantiated(), ["not", "a", "mapping"])
|
||||||
|
|
||||||
|
|
||||||
# --- Skill mutations --------------------------------------------------------
|
# --- Skill mutations --------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def test_a_missing_required_section_is_named() -> None:
|
def test_a_missing_required_section_is_named() -> None:
|
||||||
text = TEMPLATE.read_text(encoding="utf-8").replace("## Denominators", "## Counts")
|
text = instantiated().replace("## Denominators", "## Counts")
|
||||||
assert "skill_section_missing" in codes(text, load_example())
|
assert "skill_section_missing" in codes(text, load_example())
|
||||||
|
|
||||||
|
|
||||||
def test_a_missing_required_marking_is_named() -> None:
|
def test_a_missing_required_marking_is_named() -> None:
|
||||||
text = TEMPLATE.read_text(encoding="utf-8").replace("[sourced-not-sufficient]", "[thin]")
|
text = instantiated().replace("[sourced-not-sufficient]", "[thin]")
|
||||||
assert "skill_marking_missing" in codes(text, load_example())
|
assert "skill_marking_missing" in codes(text, load_example())
|
||||||
|
|
||||||
|
|
||||||
def test_a_translated_marking_literal_does_not_count() -> None:
|
def test_a_translated_marking_literal_does_not_count() -> None:
|
||||||
"""One literal string, no variants, no translations."""
|
"""One literal string, no variants, no translations."""
|
||||||
text = TEMPLATE.read_text(encoding="utf-8").replace(
|
text = instantiated().replace("[unverifiable-from-bundle]", "[ikke-verifiserbar-fra-bundle]")
|
||||||
"[unverifiable-from-bundle]", "[ikke-verifiserbar-fra-bundle]"
|
|
||||||
)
|
|
||||||
assert "skill_marking_missing" in codes(text, load_example())
|
assert "skill_marking_missing" in codes(text, load_example())
|
||||||
|
|
||||||
|
|
||||||
def test_a_skill_omitting_an_adjudication_value_is_named() -> None:
|
def test_a_skill_omitting_an_adjudication_value_is_named() -> None:
|
||||||
text = TEMPLATE.read_text(encoding="utf-8").replace("`unknown`", "`missing`")
|
text = instantiated().replace("`unknown`", "`missing`")
|
||||||
assert "skill_state_missing" in codes(text, load_example())
|
assert "skill_state_missing" in codes(text, load_example())
|
||||||
|
|
||||||
|
|
||||||
# --- The command line -------------------------------------------------------
|
# --- The command line -------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
def test_the_command_exits_zero_on_the_shipped_pair() -> None:
|
def test_the_command_exits_zero_on_an_instantiated_pair(tmp_path: Path) -> None:
|
||||||
exit_code = okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(EXAMPLE)])
|
exit_code = okf_contract_check.main(
|
||||||
|
["--skill", str(instantiated_file(tmp_path)), "--payload", str(EXAMPLE)]
|
||||||
|
)
|
||||||
assert exit_code == 0
|
assert exit_code == 0
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -190,7 +221,12 @@ def test_the_command_exits_one_on_a_non_conformant_payload(tmp_path: Path) -> No
|
||||||
payload["denominators"]["considered"] = 6
|
payload["denominators"]["considered"] = 6
|
||||||
broken = tmp_path / "payload.json"
|
broken = tmp_path / "payload.json"
|
||||||
broken.write_text(json.dumps(payload), encoding="utf-8")
|
broken.write_text(json.dumps(payload), encoding="utf-8")
|
||||||
assert okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(broken)]) == 1
|
assert (
|
||||||
|
okf_contract_check.main(
|
||||||
|
["--skill", str(instantiated_file(tmp_path)), "--payload", str(broken)]
|
||||||
|
)
|
||||||
|
== 1
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
def test_the_command_separates_could_not_run_from_non_conformant(tmp_path: Path) -> None:
|
def test_the_command_separates_could_not_run_from_non_conformant(tmp_path: Path) -> None:
|
||||||
|
|
@ -204,13 +240,13 @@ def test_the_command_separates_could_not_run_from_non_conformant(tmp_path: Path)
|
||||||
assert okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(unreadable)]) == 2
|
assert okf_contract_check.main(["--skill", str(TEMPLATE), "--payload", str(unreadable)]) == 2
|
||||||
|
|
||||||
|
|
||||||
def test_the_module_runs_as_a_script() -> None:
|
def test_the_module_runs_as_a_script(tmp_path: Path) -> None:
|
||||||
result = subprocess.run(
|
result = subprocess.run(
|
||||||
[
|
[
|
||||||
sys.executable,
|
sys.executable,
|
||||||
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
|
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
|
||||||
"--skill",
|
"--skill",
|
||||||
str(TEMPLATE),
|
str(instantiated_file(tmp_path)),
|
||||||
"--payload",
|
"--payload",
|
||||||
str(EXAMPLE),
|
str(EXAMPLE),
|
||||||
],
|
],
|
||||||
|
|
|
||||||
|
|
@ -32,6 +32,7 @@ import sys
|
||||||
import unicodedata
|
import unicodedata
|
||||||
from collections.abc import Mapping
|
from collections.abc import Mapping
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
import pytest
|
import pytest
|
||||||
|
|
||||||
|
|
@ -41,10 +42,28 @@ sys.path.insert(0, str(PROJECT_ROOT / "tools"))
|
||||||
import okf_consume # noqa: E402
|
import okf_consume # noqa: E402
|
||||||
import okf_consume_measure # noqa: E402
|
import okf_consume_measure # noqa: E402
|
||||||
import okf_contract_check # noqa: E402
|
import okf_contract_check # noqa: E402
|
||||||
|
import okf_skill # noqa: E402
|
||||||
|
|
||||||
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
|
from llm_ingestion_okf.materialize import parse_frontmatter # noqa: E402
|
||||||
|
|
||||||
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
|
TEMPLATE = PROJECT_ROOT / "skills" / "okf-consume-template" / "SKILL.md"
|
||||||
|
|
||||||
|
|
||||||
|
def _skill_declaring(payload: dict[str, Any]) -> str:
|
||||||
|
"""A skill declaring the bundle THIS payload declares.
|
||||||
|
|
||||||
|
The template cannot stand in for one any more: `<CORPUS>` and `<REF>` are
|
||||||
|
placeholders, and since 2026-09-10 an identity `okf check` cannot read is
|
||||||
|
a `bundle_mismatch` finding. The sentence comes from the generator rather
|
||||||
|
than being copied beside it."""
|
||||||
|
bundle = payload["bundle"]
|
||||||
|
return TEMPLATE.read_text(encoding="utf-8").replace(
|
||||||
|
okf_skill.TEMPLATE_HEADER,
|
||||||
|
okf_skill.identity_line(bundle["bundle_id"], bundle["ref"]) + ".",
|
||||||
|
1,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
GOLDEN = PROJECT_ROOT / "examples" / "ingest-golden-segmented-okf-v0-2" / "expected-bundle"
|
GOLDEN = PROJECT_ROOT / "examples" / "ingest-golden-segmented-okf-v0-2" / "expected-bundle"
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -730,8 +749,9 @@ def _payload(
|
||||||
return okf_consume.build_payload(root, question=question, **kwargs) # type: ignore[arg-type]
|
return okf_consume.build_payload(root, question=question, **kwargs) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
def test_the_payload_passes_the_checker_against_the_template_with_zero_findings() -> None:
|
def test_the_payload_passes_the_checker_against_a_skill_for_its_own_bundle() -> None:
|
||||||
report = okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), _payload())
|
payload = _payload()
|
||||||
|
report = okf_contract_check.check(_skill_declaring(payload), payload)
|
||||||
assert report.findings == ()
|
assert report.findings == ()
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -855,7 +875,7 @@ def test_a_question_with_no_answer_returns_a_measured_empty_set_not_a_guess() ->
|
||||||
|
|
||||||
def test_the_empty_payload_still_passes_the_checker() -> None:
|
def test_the_empty_payload_still_passes_the_checker() -> None:
|
||||||
payload = _payload(question="Hva er reglene for sveising av titan i vakuum?")
|
payload = _payload(question="Hva er reglene for sveising av titan i vakuum?")
|
||||||
assert okf_contract_check.check(TEMPLATE.read_text(encoding="utf-8"), payload).findings == ()
|
assert okf_contract_check.check(_skill_declaring(payload), payload).findings == ()
|
||||||
|
|
||||||
|
|
||||||
# --- Corpus-conditional arms --------------------------------------------------
|
# --- Corpus-conditional arms --------------------------------------------------
|
||||||
|
|
@ -1072,12 +1092,16 @@ def test_the_payload_written_by_the_cli_passes_the_checker(tmp_path: Path) -> No
|
||||||
assert (
|
assert (
|
||||||
_run(str(FIXTURE), "--question", "Hvordan skal prisene fylles ut?", "--out", str(out))
|
_run(str(FIXTURE), "--question", "Hvordan skal prisene fylles ut?", "--out", str(out))
|
||||||
).returncode == 0
|
).returncode == 0
|
||||||
|
skill_file = tmp_path / "SKILL.md"
|
||||||
|
skill_file.write_text(
|
||||||
|
_skill_declaring(json.loads(out.read_text(encoding="utf-8"))), encoding="utf-8"
|
||||||
|
)
|
||||||
checked = subprocess.run(
|
checked = subprocess.run(
|
||||||
[
|
[
|
||||||
sys.executable,
|
sys.executable,
|
||||||
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
|
str(PROJECT_ROOT / "tools" / "okf_contract_check.py"),
|
||||||
"--skill",
|
"--skill",
|
||||||
str(TEMPLATE),
|
str(skill_file),
|
||||||
"--payload",
|
"--payload",
|
||||||
str(out),
|
str(out),
|
||||||
],
|
],
|
||||||
|
|
@ -1134,9 +1158,14 @@ def test_every_rule_the_pre_pass_can_emit_is_named_in_the_skill() -> None:
|
||||||
assert rule in text, rule
|
assert rule in text, rule
|
||||||
|
|
||||||
|
|
||||||
def test_the_skill_and_a_real_payload_pass_the_checker_together() -> None:
|
def test_the_skill_and_a_real_payload_pass_the_checker_together(tmp_path: Path) -> None:
|
||||||
payload = _payload()
|
# A GENERATED skill, against a payload from the bundle it was generated for.
|
||||||
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
|
# The shipped `skills/okf-consume/SKILL.md` cannot serve here: it predates
|
||||||
|
# `okf skill` and declares no bundle identity a reader can act on, which is
|
||||||
|
# a `bundle_mismatch` finding and is recorded as one rather than worked
|
||||||
|
# around.
|
||||||
|
text, payload = okf_skill.render(GOLDEN, out=tmp_path / "skill")
|
||||||
|
assert okf_contract_check.check(text, payload).findings == ()
|
||||||
|
|
||||||
|
|
||||||
def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte() -> None:
|
def test_the_shipped_example_payload_is_current_and_regenerates_byte_for_byte() -> None:
|
||||||
|
|
@ -1793,9 +1822,8 @@ def test_a_payload_carrying_a_reservation_still_passes_the_checker(tmp_path: Pat
|
||||||
# SS 8 permits additional members; a declaration the checker refuses would
|
# SS 8 permits additional members; a declaration the checker refuses would
|
||||||
# buy honesty at the price of conformance.
|
# buy honesty at the price of conformance.
|
||||||
root = _eviction_bundle(tmp_path / "bundle")
|
root = _eviction_bundle(tmp_path / "bundle")
|
||||||
report = okf_contract_check.check(
|
payload = _eviction_payload(root, reserve_top_rank=True)
|
||||||
TEMPLATE.read_text(encoding="utf-8"), _eviction_payload(root, reserve_top_rank=True)
|
report = okf_contract_check.check(_skill_declaring(payload), payload)
|
||||||
)
|
|
||||||
assert report.findings == ()
|
assert report.findings == ()
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -2492,9 +2520,10 @@ def test_the_checker_refuses_an_excerpt_that_cannot_be_named() -> None:
|
||||||
# `title` is the defect po measured, and a checker that passes it certifies
|
# `title` is the defect po measured, and a checker that passes it certifies
|
||||||
# a payload a model cannot cite from.
|
# a payload a model cannot cite from.
|
||||||
payload = json.loads((SKILL.parent / "references" / "example-payload.json").read_text("utf-8"))
|
payload = json.loads((SKILL.parent / "references" / "example-payload.json").read_text("utf-8"))
|
||||||
assert okf_contract_check.check(SKILL.read_text(encoding="utf-8"), payload).findings == ()
|
skill_text = _skill_declaring(payload)
|
||||||
|
assert okf_contract_check.check(skill_text, payload).findings == ()
|
||||||
del payload["excerpts"][0]["title"]
|
del payload["excerpts"][0]["title"]
|
||||||
codes = [f.code for f in okf_contract_check.check(SKILL.read_text("utf-8"), payload).findings]
|
codes = [f.code for f in okf_contract_check.check(skill_text, payload).findings]
|
||||||
assert codes == ["excerpt_unnamed"]
|
assert codes == ["excerpt_unnamed"]
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,9 +2,15 @@
|
||||||
|
|
||||||
`tools/okf_skill.py` turns one OKF bundle into one instantiated `SKILL.md` that
|
`tools/okf_skill.py` turns one OKF bundle into one instantiated `SKILL.md` that
|
||||||
`tools/okf_contract_check.py` accepts. The discipline here is the one measurement
|
`tools/okf_contract_check.py` accepts. The discipline here is the one measurement
|
||||||
that decided the form: **the checker cannot tell an instantiated skill from an
|
that decided the form: the checker could not tell an instantiated skill from an
|
||||||
unfilled template**, and passes a skill built for a different bundle against this
|
unfilled template, and passed a skill built for a different bundle against this
|
||||||
one's payload. So every gate the checker does not have is a test here.
|
one's payload. So every gate the checker does not have is a test here.
|
||||||
|
|
||||||
|
**That measurement is closed on its identity half since 2026-09-10.** The
|
||||||
|
`bundle_mismatch` rule refuses both forms, and `tests/test_bundle_identity.py`
|
||||||
|
holds the arms. The gates below are the ones it still does not have: what the
|
||||||
|
generated skill MEASURES -- the per-bundle denominators, the breaking point,
|
||||||
|
the conditional-field list -- is not something any static pairing check reaches.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from __future__ import annotations
|
from __future__ import annotations
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue