docs(consume): the Claude Code recipe, measured end to end on two bundles
Four questions, two bundles, one run each, in a scratch project outside this repository with a generated skill per bundle. All four passed, and zero numbers or identifiers appeared in any answer that were not in the delivered set or in the payload's own identities (62, 45 and 35 unique numeric tokens checked). The skill triggered WITHOUT being named in the prompt and selected the right one of two installed skills from the question alone, so no special invocation syntax is needed: the generated `description`, which carries the bundle id, the concept count and the ref, is enough to route on. One defect the runs found, and it was in the prose rather than the payload. The citation guidance listed the four locator keys this library writes, so on the 270-concept third-party bundle the model reported "no page locator, the address is at document level" while the excerpt in front of it carried `source_element_id` - that bundle's own locator, correctly delivered by the prefix rule. The guidance now tells the reader to cite whichever `source_*` keys are present. On the re-run the same question returned the element id. Two runs of one question, the second measuring a changed artefact and not retrying the first. One finding that is not a defect in this chain: the first attempt at a known-negative was not one. The bundle covers water and frost protection on 17 of its 270 concepts and the ranker put none of them in the cut. The consumer behaved exactly as the contract asks - refused, named its denominator, reported its own zero as unmeasured because `withheld` entries carry no titles, and did not go around the cut. Recorded as a retrieval miss rather than replaced, and it is the same shape as the open fusion finding. A correction to this session's own measurement is in the record too: a first sweep used `grep -rhoE "^source_[a-z_]+:"`, whose character class excludes digits, and so missed `source_sha256` on 270 of 270 concepts. A pattern that cannot match what it is looking for returns a zero that reads like a fact. README gains "Consume in Claude Code": folder to answer in three commands, every one of them run in this session. A test holds that the recipe invokes only scripts this repository ships, at the paths it names. Suite 1373 (1339 at the session baseline), ruff clean, mypy src clean. No version bump, no tag, no push. Co-Authored-By: Claude <claude-opus-5>
This commit is contained in:
parent
c95d18905a
commit
171798ed32
5 changed files with 477 additions and 15 deletions
61
README.md
61
README.md
|
|
@ -191,6 +191,67 @@ hit@8 was **5 of 6** questions at rank 1 against a chance baseline of **1.35 of
|
|||
6** — with one control that failed, and both are in
|
||||
`docs/2026-09-07-okf-konsumskill-maaling.md` with the honesty limits stated.
|
||||
|
||||
## Consume in Claude Code
|
||||
|
||||
A folder of documents to an answer a model can cite, in three commands. Every
|
||||
command below was run end to end on 2026-09-08 against a nine-document folder
|
||||
and a 270-concept third-party bundle; nothing here is untested.
|
||||
|
||||
```sh
|
||||
SRC=/tmp/c1-fresh-src # the folder of documents
|
||||
BUNDLE=/tmp/c1-fresh-bundle # where the OKF bundle goes
|
||||
PROJECT=/tmp/c1-scratch # the project you will ask the question from
|
||||
```
|
||||
|
||||
**1. Build the bundle.**
|
||||
|
||||
```sh
|
||||
okf build "$SRC" --bundle "$BUNDLE" --bundle-id c1-fresh-20260908 --okf-version 0.2 --ingested-at 2026-09-08T00:00:00Z
|
||||
```
|
||||
|
||||
**2. Generate a skill for that bundle**, straight into the project's skill
|
||||
directory. The skill is instantiated for these bytes: its id, ref, concept
|
||||
count, per-field denominators, whole-bundle cost and breaking point are all
|
||||
measured from the bundle, and it ships a reference payload the checker accepts.
|
||||
|
||||
```sh
|
||||
python3 tools/okf_skill.py "$BUNDLE" --out "$PROJECT/.claude/skills/c1-fresh-20260908-consume"
|
||||
```
|
||||
|
||||
Repeat for every bundle you want reachable; each one gets its own skill named
|
||||
after its `bundle_id`, which is what lets a model pick between them. A bundle
|
||||
you only have read access to is fine — the generator only reads it.
|
||||
|
||||
**3. Ask.** From `$PROJECT`, in Claude Code:
|
||||
|
||||
```sh
|
||||
claude -p "Hvordan skal prisene fylles ut?"
|
||||
```
|
||||
|
||||
Measured with two bundles installed side by side: the model selected the right
|
||||
skill from the question alone, ran the pre-pass and the contract check itself,
|
||||
quoted the requirement verbatim, and named the document, the requirement number,
|
||||
the source resource and the locator inside it. Across three questions, **0**
|
||||
numbers or identifiers appeared in an answer that were not in the delivered set
|
||||
or in the payload's own identities. On a question the bundle does not cover it
|
||||
answered `[sourced-not-sufficient]` and reported the denominator rather than
|
||||
inventing an answer.
|
||||
|
||||
The pre-pass and the checker are the same two commands the skill runs for you,
|
||||
if you want to see the payload first:
|
||||
|
||||
```sh
|
||||
python3 tools/okf_consume.py "$BUNDLE" --question "your question" --out /tmp/payload.json
|
||||
python3 tools/okf_contract_check.py --skill "$PROJECT/.claude/skills/c1-fresh-20260908-consume/SKILL.md" --payload /tmp/payload.json
|
||||
```
|
||||
|
||||
The honest limits: this was measured on **four questions** across two bundles,
|
||||
which is a demonstration and not a hit rate. The ranking is lexical, and one of
|
||||
the four found a topic the bundle **does** cover and did not rank it into the
|
||||
cut — the skill then said so with its denominator instead of answering, which is
|
||||
the behaviour the contract asks for, but a miss is still a miss.
|
||||
`docs/2026-09-08-claude-code-skill-vilkaarlig-bundle.md` has the runs.
|
||||
|
||||
## Implemented scope (v1)
|
||||
|
||||
The library provides three entry points for getting content into an OKF
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue