docs(o6): the measurement record, including the query that could not find
Restates the order's premises on HEAD (README is 679 lines, not 676; the `okf project` install line is 320, not 317), records the defect the reproduction turned up (`okf project` 15 concepts / 30 files against `okf build`'s 26 / 52, one signature default apart), and states what the three end-to-end runs and the two-bundle routing measured with the denominator beside each figure. Two of this repository's own published figures are corrected: O5's "4 absolute paths -> 0" was measured with `grep -c "^/"` against paths indented by two spaces, so the zero was never a measurement; and the order's own "0 occurrences of hypothes/task/write/produce" is false per term (`write` 4, `produce` 7, all unrelated) even though the modes were genuinely absent. Honesty limits name N = 1 per row, that routing is 2 questions over 2 bundles and worked on the bundle ID rather than the description, that run 2 overstated one negative (NS 8407 is in 3 of 52 files, all withheld), and that run 3's document numbers are for a non-default payload it changed and declared. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
1260fac154
commit
fbad5b0eab
1 changed files with 323 additions and 0 deletions
323
docs/2026-09-09-o6-forsteskjerm-og-skillmodi.md
Normal file
323
docs/2026-09-09-o6-forsteskjerm-og-skillmodi.md
Normal file
|
|
@ -0,0 +1,323 @@
|
||||||
|
# O6: the first screen an agent reads, three modes, and one flag that made two builds
|
||||||
|
|
||||||
|
Session 119, order `20260909T155135Z-8080812956-from-.claude`, on `9567145`.
|
||||||
|
Commit `1260fac`, tag `v0.7.0` (local, unpushed).
|
||||||
|
|
||||||
|
Everything below is a measurement with the command beside it. Where a figure
|
||||||
|
this repository published before is wrong, the correction is stated rather than
|
||||||
|
quietly replaced. § 7 states what was **not** measured.
|
||||||
|
|
||||||
|
## 1 The red measurement, and the query that could not find
|
||||||
|
|
||||||
|
### 1.1 The order's own premises, restated on HEAD
|
||||||
|
|
||||||
|
The order's PM figures (i)–(iii) were measured on an export of `a364ef4`, one
|
||||||
|
commit behind HEAD. Restated on `9567145`:
|
||||||
|
|
||||||
|
| Premise | PM on `a364ef4` | Measured on `9567145` | Command |
|
||||||
|
|---|---|---|---|
|
||||||
|
| suite | 1498 | **1498** | `pytest -q` |
|
||||||
|
| ruff | clean | **clean** | `ruff check src tests tools` |
|
||||||
|
| mypy | clean, 21 files | **clean, 21 files** | `mypy --strict src` |
|
||||||
|
| template `SKILL.md` | 164 lines | **164** | `wc -l < skills/okf-consume-template/SKILL.md` |
|
||||||
|
| a GENERATED skill | 231 lines | **231** | `okf skill …/expected-bundle --out <scratch>` |
|
||||||
|
| `README.md` | 676 lines | **679** | `wc -l < README.md` |
|
||||||
|
| `okf project` install line | line 317 | **line 320** | `grep -n "@v0\." README.md` |
|
||||||
|
| `~/okf-test/dokumenter` default | **52 md** | **30 md** — see § 2 | `okf project ~/okf-test/dokumenter` |
|
||||||
|
|
||||||
|
The last row is not a restatement. It is a defect, and it is § 2.
|
||||||
|
|
||||||
|
### 1.2 The pins disagreed
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -n "@v0\." README.md
|
||||||
|
26: … llm-ingestion-okf.git@v0.4.0
|
||||||
|
40: … llm-ingestion-pipeline-security.git@v0.2.0
|
||||||
|
41: … llm-ingestion-okf.git@v0.4.0
|
||||||
|
320: … llm-ingestion-okf.git@v0.6.0
|
||||||
|
$ grep -n "@v0\." llms.txt
|
||||||
|
10: … llm-ingestion-okf.git@v0.4.0
|
||||||
|
```
|
||||||
|
|
||||||
|
An agent reading either file from the top installed `v0.4.0`, which has no
|
||||||
|
`okf project`. `llms.txt` — the file an agent fetches *first* — named neither
|
||||||
|
`okf project` nor Claude Code at all.
|
||||||
|
|
||||||
|
### 1.3 The query that could not find (face 4)
|
||||||
|
|
||||||
|
`docs/2026-09-08-o5-okf-project.md` lines 16 and 177 measure absolute paths in a
|
||||||
|
generated skill with `grep -c "^/"` and report **4 → 0**. The generator indents
|
||||||
|
those paths by two spaces, so `^/` cannot match one. Control, run before any
|
||||||
|
zero was read:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ printf 'prose with no path\n /Users/x/bundle\nokf consume /Users/x/other\n' > kp.txt
|
||||||
|
$ grep -c "^/" kp.txt # O5's query
|
||||||
|
0 # <- cannot find a known positive
|
||||||
|
$ grep -cE '(^|[[:space:]])/[A-Za-z]' kp.txt
|
||||||
|
2 # <- can
|
||||||
|
```
|
||||||
|
|
||||||
|
Applied to a generated skill on `9567145`:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -nE '(^|[[:space:]])/[A-Za-z]' <skill>/SKILL.md
|
||||||
|
26: /Users/ktg/repos/llm-ingestion-okf/examples/…/expected-bundle \
|
||||||
|
29: --out /tmp/payload.json
|
||||||
|
41: --skill /private/tmp/o6-red/skillgen/SKILL.md \
|
||||||
|
42: --payload /tmp/payload.json
|
||||||
|
$ grep -n "$HOME" <skill>/SKILL.md
|
||||||
|
26: /Users/ktg/repos/…
|
||||||
|
```
|
||||||
|
|
||||||
|
**Two** paths into the machine the skill was generated on (26 and 41), confirming
|
||||||
|
the order's (iii). The other two are `/tmp/payload.json`, a scratch write target
|
||||||
|
that names no checkout. O5's "0" was never a measurement — the query it used
|
||||||
|
could not have produced anything else.
|
||||||
|
|
||||||
|
### 1.4 The modes, counted per term
|
||||||
|
|
||||||
|
The order's (ii) is confirmed in substance and corrected in form. Counted with
|
||||||
|
`grep -ic <term> skills/okf-consume-template/SKILL.md`:
|
||||||
|
|
||||||
|
| term | template | generated |
|
||||||
|
|---|---|---|
|
||||||
|
| `hypothes` | 0 | 0 |
|
||||||
|
| `task` | 0 | 0 |
|
||||||
|
| `oppgave` | 0 | 0 |
|
||||||
|
| `notat` | 0 | 0 |
|
||||||
|
| `write` | **4** | 2 |
|
||||||
|
| `produce` | **7** | 15 |
|
||||||
|
| `confirm` | 1 | 1 |
|
||||||
|
| `refut` | 0 | 0 |
|
||||||
|
|
||||||
|
Every `write` and `produce` hit is unrelated ("the producer wrote none", "Write
|
||||||
|
to `<OUT>`"). The modes were absent; the claim "0 occurrences of
|
||||||
|
hypothes/task/write/produce" was still false as a measurement.
|
||||||
|
|
||||||
|
## 2 The defect the reproduction found: two builds, one command
|
||||||
|
|
||||||
|
`okf project ~/okf-test/dokumenter` wrote **15 concepts / 30 files**. `okf build`
|
||||||
|
on the same folder wrote **26 / 52**. `CLAUDE.md` states the opposite — "it owns
|
||||||
|
NO flag that moves a bundle's bytes and a test holds it byte-equal to
|
||||||
|
`okf build`" — and a test by that name exists and was green.
|
||||||
|
|
||||||
|
**Mechanism.** `cli.build`'s Python signature carried
|
||||||
|
`keep_table_heading: bool = False` and `sheet_section_rows: bool = False` while
|
||||||
|
argparse carried `DEFAULT_KEEP_TABLE_HEADING = True` and
|
||||||
|
`DEFAULT_SHEET_SECTION_ROWS = True`. Every other flag reads its `DEFAULT_`
|
||||||
|
constant in both places; these two did not. `project.create` calls `build()` as a
|
||||||
|
function and passes no flag list, so it read the signature. The whole 22-file
|
||||||
|
difference was inside `del-ii-bilag-7-prisskjema/` — the priced spreadsheet, which
|
||||||
|
is the document a question about price has to reach.
|
||||||
|
|
||||||
|
**Why the test could not see it.** `test_the_project_bundle_is_the_bytes_okf_build_writes`
|
||||||
|
compares `project.create` against the same `build()` function, so both sides
|
||||||
|
carried the same wrong value; and its two fixture documents (`krav.md`,
|
||||||
|
`notat.md`) contain neither a table nor a sheet, so the difference could not have
|
||||||
|
appeared even against argparse. A test and the code agreeing over a set where the
|
||||||
|
difference cannot appear.
|
||||||
|
|
||||||
|
**Repair, red first.** Two tests, both red before the change:
|
||||||
|
|
||||||
|
- `test_the_build_signature_defaults_are_the_build_command_defaults` — compares
|
||||||
|
the signature's default against argparse's for every parameter argparse passes
|
||||||
|
through at the same type. Red output named exactly the two:
|
||||||
|
`{'keep_table_heading': (False, True), 'sheet_section_rows': (False, True)}`.
|
||||||
|
Parameters argparse converts (`segments` `'on'`→bool, `pdf_headings`
|
||||||
|
`'none'`→bool, `okf_version` caller-owned) are excluded by the type test, not
|
||||||
|
by a list.
|
||||||
|
- `test_a_sheet_reaches_the_project_bundle_as_it_reaches_the_build_command` —
|
||||||
|
the same folder through `project.create` and through `okf_main(["build", …])`,
|
||||||
|
on a document whose concept count actually moves with the two flags. On the
|
||||||
|
first draft (a two-row table) it passed; that draft is recorded here because it
|
||||||
|
is the same failure as the invariant it replaces — a green over an empty set.
|
||||||
|
With five numeric rows it went red (2 segments vs 7) and is the shipped form.
|
||||||
|
|
||||||
|
**After.** `okf project` writes 26 concepts / 52 files, and `diff -rq` against the
|
||||||
|
`okf build` bundle of the same folder reports **0 differences**.
|
||||||
|
|
||||||
|
## 3 A — the first screen
|
||||||
|
|
||||||
|
`README.md` now opens with, in this order: what it is in one sentence; one
|
||||||
|
`uv tool install` line; `okf project <folder>` and `claude`; the three shapes of
|
||||||
|
request; one line saying where the rest is. The phase-status paragraph that used
|
||||||
|
to be lines 3–19 moved down under `## What this library is` — moved, not deleted.
|
||||||
|
|
||||||
|
`llms.txt` was rewritten in the same shape: the new tag, the `okf project` form,
|
||||||
|
the three modes, and a pointer into the Claude Code section.
|
||||||
|
|
||||||
|
Acceptance, measured:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -o "@v0\.[0-9.]*" README.md llms.txt | sed 's/.*://' | sort -u
|
||||||
|
@v0.7.0
|
||||||
|
$ printf 'x %s/y\n' "$HOME" > kp.txt; grep -c "$HOME" kp.txt # control: can find
|
||||||
|
1
|
||||||
|
$ grep -c "$HOME" README.md llms.txt
|
||||||
|
README.md:0
|
||||||
|
llms.txt:0
|
||||||
|
```
|
||||||
|
|
||||||
|
Earlier tags are kept under `### Earlier tags, as history`, labelled as history
|
||||||
|
and not as commands. The guard pairing is measured, not guessed: `v0.7.0`'s own
|
||||||
|
`[tool.uv.sources]` names `llm-ingestion-guard` `v1.3.0`, and § 5 shows the
|
||||||
|
install resolving exactly that.
|
||||||
|
|
||||||
|
## 4 B — three modes, relative paths
|
||||||
|
|
||||||
|
`## Modes` in the template, the instantiated `skills/okf-consume/SKILL.md`, and
|
||||||
|
therefore in every generated skill. Question (the default), Hypothesis, and Task
|
||||||
|
that produces a document. The five markings are untouched — a test asserts every
|
||||||
|
literal `okf check` fixes is still present, so the modes added no sixth.
|
||||||
|
|
||||||
|
Relative paths: `skill.project_root_of(out)` reads `<root>/.claude/skills/<name>`
|
||||||
|
off the path and returns `<root>`; `skill.as_written(path, base=root)` writes a
|
||||||
|
path relative to it **when it is under it**, and leaves it absolute otherwise. A
|
||||||
|
path outside the project root stays absolute on purpose: `../../..` is not more
|
||||||
|
portable, only harder to read.
|
||||||
|
|
||||||
|
Measured on a bundle built by `okf project` in a cold directory:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ grep -nE '(^|[[:space:]])/[A-Za-z]' .claude/skills/dokumenter-consume/SKILL.md
|
||||||
|
29: --out /tmp/payload.json
|
||||||
|
42: --payload /tmp/payload.json
|
||||||
|
$ grep -c "$HOME" .claude/skills/dokumenter-consume/SKILL.md
|
||||||
|
0
|
||||||
|
$ grep -n "\.okf/dokumenter\|\.claude/skills/dokumenter-consume/SKILL.md" <skill>
|
||||||
|
26: .okf/dokumenter \
|
||||||
|
41: --skill .claude/skills/dokumenter-consume/SKILL.md \
|
||||||
|
```
|
||||||
|
|
||||||
|
Two paths into the machine → **0**. Both commands run from the project root:
|
||||||
|
|
||||||
|
```
|
||||||
|
$ okf consume .okf/dokumenter --question … --ref … --out /tmp/payload.json ; echo $?
|
||||||
|
0
|
||||||
|
$ okf check --skill .claude/skills/dokumenter-consume/SKILL.md --payload /tmp/payload.json
|
||||||
|
conformant: 15 rules over 8 excerpts and 18 withheld entries, 0 findings
|
||||||
|
```
|
||||||
|
|
||||||
|
## 5 The install measured from a cold, empty directory
|
||||||
|
|
||||||
|
`uv tool install` cannot point at an unpushed tag over git, so the tag was
|
||||||
|
exported and installed from a local path. This is stated rather than glossed: the
|
||||||
|
bytes are the tag's (`git archive v0.7.0`), the channel is not.
|
||||||
|
|
||||||
|
```
|
||||||
|
$ git archive v0.7.0 | tar -x -C /tmp/o6-install/export
|
||||||
|
$ uv tool install --force "llm-ingestion-okf[extract] @ file:///tmp/o6-install/export"
|
||||||
|
+ llm-ingestion-guard==1.3.0 (from git+…@44e2b31…)
|
||||||
|
+ llm-ingestion-okf==0.7.0 (from file:///tmp/o6-install/export)
|
||||||
|
Installed 1 executable: okf
|
||||||
|
```
|
||||||
|
|
||||||
|
The guard pairing the README states is what resolved. Then, in an empty
|
||||||
|
directory, with no `PYTHONPATH`, no `VIRTUAL_ENV`, and this repository nowhere on
|
||||||
|
`PATH` (`which okf` → `~/.local/bin/okf`):
|
||||||
|
|
||||||
|
| | |
|
||||||
|
|---|---|
|
||||||
|
| documents in | **5** |
|
||||||
|
| concepts out | **26** |
|
||||||
|
| files out | **52** md |
|
||||||
|
| left out of the bundle | **0 of 5** |
|
||||||
|
| landed whole as one concept | 1 of 5 (named in the output) |
|
||||||
|
| `okf check` | conformant, 15 rules, **0 findings** |
|
||||||
|
| paths into any checkout in the skill | **0** |
|
||||||
|
|
||||||
|
Order of operations, as the order required: the tag was created (§ 6) before this
|
||||||
|
was run, because the measurement is of the tag.
|
||||||
|
|
||||||
|
## 6 The three end-to-end runs (C1's instrument)
|
||||||
|
|
||||||
|
Skills installed in a project directory outside this repository;
|
||||||
|
`cd <project> && claude -p "<the question>"`; **one run per row, no re-run for a
|
||||||
|
better answer**. Sources grep-verified against the bundle tree with the
|
||||||
|
denominator stated.
|
||||||
|
|
||||||
|
| # | Mode | Asked | Sources verified | Denominator |
|
||||||
|
|---|---|---|---|---|
|
||||||
|
| 1 | question | "hva er kravene til pris?" | **8 of 8** sha256 prefixes found in the payload | 8 excerpts in the payload |
|
||||||
|
| 2 | hypothesis | "stemmer det at totalentreprenøren bærer risikoen for massehåndtering, og at massene prises som fastpris i basiskostnaden?" | **2 of 2** sha256 prefixes; the verbatim quote found in **1** file | 8 excerpts; 52 md files in the bundle |
|
||||||
|
| 3 | document task | "lag `krav-pris.md` med alle krav til pris, ett avsnitt per krav, med dokument og kravnummer" | **28 of 28** 64-hex digests; **24 of 24** concept paths exist on disk | **80** known digests (excerpt fields + 52 concept files + 5 source documents + ref) |
|
||||||
|
|
||||||
|
Invented numbers: **0** across the three runs, against the denominators above.
|
||||||
|
Every digest and every concept path a run cited was found.
|
||||||
|
|
||||||
|
**Run 2 answered per premise**, which is what the mode asks for: premise 2
|
||||||
|
`refuted` with the contradicting sentence quoted, premise 1 split into mengderisiko
|
||||||
|
(`refuted`, `derived`), enhetsprisrisiko (`confirmed`, `extracted`), the
|
||||||
|
tiltaksklasse question `undecidable-from-bundle` marked
|
||||||
|
`[sourced-not-sufficient]` **on that premise**, and the hypothesis-level verdict
|
||||||
|
stated as `derived`. No fourth verdict value appeared.
|
||||||
|
|
||||||
|
**Run 3 wrote the document.** 221 lines, 37 requirement paragraphs, and — the
|
||||||
|
part that matters — the discipline is *in the file*: 24 `extracted`, 18 `derived`,
|
||||||
|
3 `[unverifiable-from-bundle]`, 1 `[unread]`, **2 `[sourced-not-sufficient]`**
|
||||||
|
standing where the requirement belongs rather than dropped, the bundle ref, and
|
||||||
|
`considered / withheld / delivered` declared in the document's own header table.
|
||||||
|
It also reported that `req_number` is **0 of 26** in this bundle and marked its
|
||||||
|
own numbering `derived` rather than inventing requirement numbers — which is the
|
||||||
|
failure the mode exists to prevent.
|
||||||
|
|
||||||
|
## 7 C — two bundles in one project
|
||||||
|
|
||||||
|
Two `okf project` runs with different `--id` on two topically different document
|
||||||
|
sets, split from `~/okf-test/dokumenter` into scratch (`~/okf-test` read-only):
|
||||||
|
|
||||||
|
```
|
||||||
|
.okf/pris 13 concepts ref sha256-tree:477ed72de1d3be18…
|
||||||
|
.okf/teknikk 10 concepts ref sha256-tree:e53bed17f7b8a5e2…
|
||||||
|
.claude/skills/pris-consume .claude/skills/teknikk-consume
|
||||||
|
```
|
||||||
|
|
||||||
|
Both exist side by side with different refs. The order's (vii) is confirmed on
|
||||||
|
the descriptions: the two `description:` lines differ **only** in the bundle id,
|
||||||
|
the concept count and the ref — there is no topic signal in either.
|
||||||
|
|
||||||
|
Routing measured with **no skill named in the prompt**, one run per question:
|
||||||
|
|
||||||
|
| Question | Belongs to | Chosen | Ref in the answer |
|
||||||
|
|---|---|---|---|
|
||||||
|
| "hva kreves ved prøvedrift?" | `teknikk` | **`teknikk`** | `e53bed17f7b8a5e2…` |
|
||||||
|
| "hva er påslagsprosenten for materialer?" | `pris` | **`pris`** | `477ed72de1d3be18…` |
|
||||||
|
|
||||||
|
**2 of 2.** The first run's output names *both* bundles before settling on
|
||||||
|
`teknikk`, so the choice was made rather than defaulted to. No topic signal was
|
||||||
|
added to the generator, because the measurement did not call for one: the bundle
|
||||||
|
**id** carried enough. That is a property of ids chosen meaningfully, not of the
|
||||||
|
description line, and § 8 says so.
|
||||||
|
|
||||||
|
## 8 Honesty limits
|
||||||
|
|
||||||
|
- **N = 1 per row in § 6 and § 7.** Four `claude -p` runs total, no re-runs. A
|
||||||
|
single run is not a rate.
|
||||||
|
- **Routing is 2 questions over 2 bundles, on one corpus split in two.** It was
|
||||||
|
*not* measured for more than two bundles, for topically overlapping bundles, or
|
||||||
|
for ids that carry no topic (`alfa`/`beta`). The mechanism that worked was the
|
||||||
|
**id**, and a caller who passes `--id a` and `--id b` has removed it. The
|
||||||
|
description line still carries no topic signal.
|
||||||
|
- **Run 2 overstated one negative.** It wrote "NS 8407-bestemmelsene er ikke i
|
||||||
|
bundelen". `grep -ril '8407' .okf/dokumenter --include='*.md'` finds **3 of 52**
|
||||||
|
files (control: `pris` matches 21 of 52, so the query can find). All three are
|
||||||
|
in **withheld** concepts, and the answer did name
|
||||||
|
`dokument-for-avtaleinngåelse/tabell-linje-30` as the one withheld concept that
|
||||||
|
might carry contract terms — so the coverage limitation was reported and the
|
||||||
|
sentence above it was still stronger than what was measured.
|
||||||
|
- **Run 3 changed the instrument mid-task and said so.** It re-ran the pre-pass
|
||||||
|
with `--k 26` and `--cost-vocabulary` after the default `k=8` delivered 8 of 26,
|
||||||
|
and declared both cuts in the file. The document's numbers are therefore for a
|
||||||
|
**non-default** payload; the § 6 default figures are the ones for the shipped
|
||||||
|
configuration.
|
||||||
|
- **The install in § 5 came from a local export, not a git URL.** The tag is
|
||||||
|
unpushed by instruction, so the channel differs from the README's line even
|
||||||
|
though the bytes do not.
|
||||||
|
- **No `pptx` was in any measured folder.** The `pptx` fasit stands where round 7
|
||||||
|
left it.
|
||||||
|
- **The K2 reference bundle was not rebuilt.** This session changed no
|
||||||
|
segmentation rule and no proposer default; the `cli.build` signature change
|
||||||
|
affects only callers who used the Python API without passing the two flags, of
|
||||||
|
which `project.create` was the one in this repository. `tests/test_default_bundle_pin.py`
|
||||||
|
is untouched and green.
|
||||||
Loading…
Add table
Add a link
Reference in a new issue