open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

docs(front-page): the gate numbers the gate actually prints, and a breaking point that was measured

Browse source 
Four claims on the front page were false on this commit, and one of them was a
number no division ever produced.

**The retrieval gate.** README reported it RED on rows 3, 4, 5, 7, 8 and 9,
with row 3 at 2 of 5 and row 4 at 3 of 6. Run on this commit it is RED on rows
5, 7, 8 and 9, with row 3 at 5 of 5 and row 4 at 6 of 6: `f81683e` made a
withheld concept carry the rule that actually decided it, and `05cb190` gave
the payload a `coverage` block, and neither updated the table. Row 8 is `0 of 3
| NOT RUN` on the default run and was published as `44 of 64 questions`, which
is what it scores the day all three private sets are handed to it -- now
labelled with the day and the machine rather than printed as a row. The same
four figures were stale in `CLAUDE.md`.

**The breaking point in a generated skill.** `int(LIMIT / per_withheld) if
per_withheld else 0` printed `At roughly 0 concepts the bookkeeping alone
reaches the 120000-byte limit` whenever the generation run withheld nothing --
the absence of a measurement, rendered as one, and read as a bundle that breaks
before it holds anything. A run with no withheld entry has no slope to
extrapolate from, so the sentence is withheld with its reason. The shipped
`skills/okf-consume/SKILL.md` is generated with the question its
`references/README.md` names, withholds nothing, and carried exactly that `0`;
it is regenerated. Two arms in the test, because one would pass on an empty
set: the bundles that withhold something must still state a positive figure.

The sentence for that arm also stopped saying `**4 bytes** for 3 concepts`
where the 4 bytes were the cost of 0 withheld entries. It is now `for N of M
concepts`, which moves two generated skills' line counts and therefore the
published comparison: 280 of 312 and 310 -> 281 of 313 and 311, re-measured,
with the 62 differing lines unchanged.

**Four tools.** A single-bundle server exposes three: `okf_list` is absent
where there is nothing to list. README's table already said so in a cell; the
heading and the CHANGELOG did not.

**What `--accounting` accounts for.** The account is over the element classes
each format's vocabulary names, verified against `accounting._READERS` rather
than against the report: a file whose suffix has no reader is accounted at file
level only, `.docx` reads `document.xml` and `footnotes.xml` (so headers,
footers, endnotes and comments are outside), `.pptx` reads the slides (so
speaker notes are outside), `.xlsx` reads the worksheets (so cell comments are
outside and a cell contributes its cached value, never its formula), and `.rtf`
skips its header and footer groups. A hidden slide or sheet IS counted -- it
lives in the same part as a visible one. Nothing is built for this; the list is
what `0 unaccounted` does not claim.

Gates re-run on the commit: retrieval `GATE RED: rows 5, 7, 8, 9` (exit 1),
MCP `GATE RED: rows 2` (exit 1), both matching what is now written.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-20 15:40:20 +02:00
commit d300338e4d
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
9 changed files with 203 additions and 754 deletions
Download patch file Download diff file Expand all files Collapse all files

21
CHANGELOG.md
View file

@ -7,6 +7,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [0.10.1] — 2026-09-19 ## [0.10.1] — 2026-09-19
### Removed
- **`tools/okf_adjudicate.py`.** It shelled out to a model CLI at an absolute
path on one machine, which is the one thing nothing in this repository does:
no code here starts another program to judge anything. Its tests go with it.
The two entries below under earlier versions describe what that tool did
while it existed and are left standing — a changelog that edits its own past
is not a record. The K3/K4/K5 reports that used it now say so in the past
tense.
### Added ### Added
- **An MCP surface over OKF bundles, in two shapes, plus a generic - **An MCP surface over OKF bundles, in two shapes, plus a generic
@ -14,7 +24,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
`okf mcp --root <dir>` (repeatable) serves every bundle under the roots and `okf mcp --root <dir>` (repeatable) serves every bundle under the roots and
knows none of them by name. Four tools — `okf_list`, `okf_describe`, knows none of them by name. Four tools — `okf_list`, `okf_describe`,
`okf_ask`, `okf_fetch` — each with its reason written into the description a `okf_ask`, `okf_fetch` — each with its reason written into the description a
client reads. The eval was written RED first (`tools/okf_mcp_gate.py`, client reads. A single-bundle server exposes **three**: `okf_list` is absent
where there is nothing to list, because a tool that always returns the same
one row invites a client to treat discovery as available when the deployment
does not have it. The eval was written RED first (`tools/okf_mcp_gate.py`,
`5f1772e`); the capability follows. `5f1772e`); the capability follows.
- **The protocol is written narrowly with stdlib only, and that is the - **The protocol is written narrowly with stdlib only, and that is the
packaging invariant kept rather than an aesthetic.** An MCP SDK would be packaging invariant kept rather than an aesthetic.** An MCP SDK would be
@ -130,6 +143,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
gate and `--gate none`. gate and `--gate none`.
- **Opt-in, measured.** On the 43-document reference corpus the build took - **Opt-in, measured.** On the 43-document reference corpus the build took
+744 s (+19 %) and +0.53 GB peak RSS. +744 s (+19 %) and +0.53 GB peak RSS.
- **The account is over the element classes the vocabulary knows.** A file
whose suffix has no reader is accounted at file level only, and a part of
a document no vocabulary names is not counted — `.docx` headers, footers,
endnotes and comments, `.pptx` speaker notes, `.xlsx` cell comments and
formulas, the `.rtf` header/footer groups. Content there can go missing
under exit 0 and `0 unaccounted`; README states the list.
- **The reference corpus fails the check, with 24 real losses:** 22 images - **The reference corpus fails the check, with 24 real losses:** 22 images
on PDF pages without a text layer, which the reader drops together with on PDF pages without a text layer, which the reader drops together with
the page, and 2 docx Title paragraphs, which the converter moves into the page, and 2 docx Title paragraphs, which the converter moves into

35
CLAUDE.md
View file

@ -1263,10 +1263,11 @@ and fixtures, never code.
bytes moved is exit 2, never a quiet continuation, because **a gold set is bytes moved is exit 2, never a quiet continuation, because **a gold set is
an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and an INPUT and never a constant here** (`okf_consume_measure.py`'s rule, and
this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since this repo is public). **THE CORPUS IS PINNED THE SAME WAY** since
2026-09-19 (`SPECS_SHA256`). Today: **1 (10/10) - 2 (7/7) - 3 (2/5) - 2026-09-19 (`SPECS_SHA256`). Today (measured on the default run
4 (3/6) - 5 (0/1) - 6 (10/10) - 7 (12/14) - 8 (44/64 questions, all three 2026-09-20): **1 (10/10) - 2 (7/7) - 3 (5/5) - 4 (6/6) - 5 (0/1) -
real sets measured 2026-09-19) - 9 (0/6)**, `GATE RED: rows 3, 4, 5, 7, 8, 6 (10/10) - 7 (12/14) - 8 (0 of 3 sets, NOT RUN without `--real`; 44 of 64
9`. questions the day all three were given to it, 2026-09-19) - 9 (0/6)**,
`GATE RED: rows 5, 7, 8, 9`.
**FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED **FOUR OF EIGHT CHEATING ATTACKS WENT THROUGH IT AND ARE CLOSED
(2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could (2026-09-19).** PM's checkpoint on `2c8296b` found rows 3, 5, 7 and 8 could
go GREEN without one label becoming true or one concept ranking better, and go GREEN without one label becoming true or one concept ranking better, and
@ -1305,15 +1306,19 @@ and fixtures, never code.
(index built UNDER the patch: every unit a hit; warmed first: every unit a (index built UNDER the patch: every unit a hit; warmed first: every unit a
miss). Row 9 takes `--k2 SET SHA BUNDLE` and stays RED without one, because miss). Row 9 takes `--k2 SET SHA BUNDLE` and stays RED without one, because
ITS denominator is known -- a deliberate divergence from the order, stated. ITS denominator is known -- a deliberate divergence from the order, stated.
Report: `docs/2026-09-19-gjenfinningsgaten-motstand.md`. **Row 3 is the finding to read first**: in a bundle built Report: `docs/2026-09-19-gjenfinningsgaten-motstand.md`. **ROWS 3 AND 4 WERE THE TWO FINDINGS AND BOTH ARE
from ONE source document every concept past the first two carries that CLOSED** (`f81683e`, `05cb190`, 2026-09-20). Row 3: in a bundle built from
document's `source_file`, so a concept the RANK had already lost is withheld ONE source document every concept past the first two carries that document's
as `source_quota_exceeded` -- the truth is decided by the SAME cut run `source_file`, so a concept the RANK had already lost came back withheld as
without the quota, and PM measured the same lie on **13 of 25** real misses `source_quota_exceeded` -- PM measured the same lie on **13 of 25** real
17.09. **Row 4 is the second**: no payload key says "this bundle does not misses 17.09. A drop now keeps the rule the SAME cut without the quota would
answer that", so the only honest case is `delivered == 0`; 3 of 6 controls have given it, decided by the code the run itself uses: **2 of 5 -> 5 of
come out right and the three that do not get eight excerpts and no 5**. Row 4: no payload key said "this bundle does not answer that", so the
statement. **A miss has exactly ONE class** (a fasit not in the bundle, b only honest case was `delivered == 0`. `coverage` states the terms the
pre-pass read, the terms no concept in the bundle answers and the terms no
delivered excerpt answers -- facts and no verdict, both candidate verdicts
falsified over 81 questions -- with the gate's own `UNANSWERED_BAR = 2/3`:
**3 of 6 -> 6 of 6**, 0 of 65 real questions marked. **A miss has exactly ONE class** (a fasit not in the bundle, b
below k, c cut by quota/budget, d delivered without the citation, e other), below k, c cut by quota/budget, d delivered without the citation, e other),
each FORCED by its own fixture, and an unplaced miss takes the row to 0 of each FORCED by its own fixture, and an unplaced miss takes the row to 0 of
N. **The judge opens the bundle**: `BundleIndex` reads every concept off N. **The judge opens the bundle**: `BundleIndex` reads every concept off
@ -1449,8 +1454,8 @@ and fixtures, never code.
skill for ANY bundle: it carries no bundle's identity and no bundle's numbers, skill for ANY bundle: it carries no bundle's identity and no bundle's numbers,
and the property that makes that checkable rather than asserted is that and the property that makes that checkable rather than asserted is that
`skill.render_generic()` **takes no argument** -- there is no bundle it could `skill.render_generic()` **takes no argument** -- there is no bundle it could
have read. Measured 2026-09-20: two per-bundle skills are identical on **280 have read. Measured 2026-09-20: two per-bundle skills are identical on **281
of 312** and **310** lines, and the 62 that differ are exactly identity, of 313** and **311** lines, and the 62 that differ are exactly identity,
concept count, the conditional-field table, the whole-bundle cost and the concept count, the conditional-field table, the whole-bundle cost and the
breaking point -- the five things that go stale on a rebuild. The update breaking point -- the five things that go stale on a rebuild. The update
drill, four artefact classes: MCP one-to-one **0 artefacts / 0 steps**, MCP drill, four artefact classes: MCP one-to-one **0 artefacts / 0 steps**, MCP

69
README.md
View file

@ -464,6 +464,27 @@ short element such as a section label can be found elsewhere in the same
document. The judge is `tools/okf_accounting_gate.py`, which compares the document. The judge is `tools/okf_accounting_gate.py`, which compares the
inventory against an independent witness. inventory against an independent witness.
**The account covers the element classes the vocabulary knows, and no others.**
`accounting._READERS` names twelve suffixes, each with its own tuple of classes
(`.md`: heading, paragraph, table, table_row, image, code_block; `.pdf`: image
and page only, which is the approved exception below). Two consequences are
stated here rather than left to be discovered, because "0 unaccounted" reads
like a statement about the document and is a statement about those classes:
- **A file whose suffix has no reader is accounted at FILE level only** —
carried, merged or rejected — never element by element.
- **Parts of a document that no vocabulary names are not counted, so content
there can go missing under exit 0 and `0 unaccounted`.** Verified against the
readers: `.docx` reads `word/document.xml` and `word/footnotes.xml`, so
headers, footers, endnotes and comments are outside; `.pptx` reads
`ppt/slides/slideN.xml`, so speaker notes, masters and layouts are outside;
`.xlsx` reads the worksheets, the shared strings and the drawings, so cell
comments are outside and a cell contributes its cached value or inline
string, never its formula; `.rtf` skips the `header`, `footer`, `info`,
`pict`, `stylesheet`, `fonttbl` and `colortbl` groups. A hidden slide or
sheet IS counted — it lives in the same part as a visible one. Nothing here
is built for now: the list is what the account does not claim.
**Two operator decisions, 2026-09-17.** The accounting stays OPT-IN until the **Two operator decisions, 2026-09-17.** The accounting stays OPT-IN until the
losses it reports on the reference corpus are fixed, because a default-on door losses it reports on the reference corpus are fixed, because a default-on door
would fail builds that pass today. And of the three exceptions the gate would fail builds that pass today. And of the three exceptions the gate
@ -1018,30 +1039,33 @@ set names a consumer's documents and this repository is public. **The corpus
is pinned the same way** (`SPECS_SHA256`): every row counts against those is pinned the same way** (`SPECS_SHA256`): every row counts against those
documents, so moving them without moving the pin is exit 2. documents, so moving them without moving the pin is exit 2.
**It is RED today, on rows 3, 4, 5, 7, 8 and 9**, and each of those is a **It is RED today, on rows 5, 7, 8 and 9**, and each of those is a
finding rather than a defect in the gate: finding rather than a defect in the gate:
| row | what it asks | today | | row | what it asks | today |
|---|---|---| |---|---|---|
| 1 | hit@payload, one fasit entry = one unit | 10 of 10 | | 1 | hit@payload, one fasit entry = one unit | 10 of 10 |
| 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 | | 2 | every miss carries exactly one class, each forced by its own fixture | 7 of 7 |
| 3 | the `rule` the payload prints for a withheld fasit is the true one | 2 of 5 | | 3 | the `rule` the payload prints for a withheld fasit is the true one | 5 of 5 |
| 4 | an uncovered question comes back marked, a covered one does not | 3 of 6 | | 4 | an uncovered question comes back marked, a covered one does not | 6 of 6 |
| 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 | | 5 | a hold-out set, frozen and with its threshold written first | 0 of 1 |
| 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 | | 6 | every delivery confirmed against the bundle's own bytes | 10 of 10 |
| 7 | mechanical mutants of the ranking and the cut, felled | 12 of 14 | | 7 | mechanical mutants of the ranking and the cut, felled | 12 of 14 |
| 8 | the three real sets, from path + sha256 | 44 of 64 questions | | 8 | the three real sets, from path + sha256 | 0 of 3 sets, NOT RUN without `--real` |
| 9 | K2 | 0 of 6, no gold set exists | | 9 | K2 | 0 of 6, no gold set exists |
Row 3 is the one to read first: in a bundle built from ONE source document, Rows 3 and 4 were this gate's two findings and both are closed, which is what
every concept past the first two carries that document's `source_file`, so a a gate written before the capability is for. Row 3: in a bundle built from ONE
concept the RANK had already lost is withheld as `source_quota_exceeded`. The source document, every concept past the first two carries that document's
gate decides the truth with the same cut run without the quota, and a `source_file`, so a concept the RANK had already lost came back withheld as
consumer reading `rule` today gets the wrong reason. Row 4 is the second: the `source_quota_exceeded`. A drop now keeps the rule the same cut without the
payload has no key a consumer can read as "this bundle does not answer that", quota would have given it, and only a candidate that cut would have delivered
so an uncovered question comes back with excerpts and no statement — the only is named as the quota's — 2 of 5 to 5 of 5. Row 4: the payload had no key a
honest case today is the one where nothing matched at all and nothing was consumer could read as "this bundle does not answer that", so an uncovered
delivered. question came back with excerpts and no statement. `coverage` states the terms
the pre-pass read, the terms no concept in the bundle answers and the terms no
delivered excerpt answers — facts, no verdict, and the one bar is the gate's
own `UNANSWERED_BAR = 2/3` — 3 of 6 to 6 of 6.
Row 7 reports two survivors with what they moved rather than with a shrug: Row 7 reports two survivors with what they moved rather than with a shrug:
killing the document prior and flattening the fusion (`RRF_K`) each moved killing the document prior and flattening the fusion (`RRF_K`) each moved
@ -1064,10 +1088,13 @@ Granularity is stated on every line and the two forms are never summed: a set
naming a citation is measured at citation granularity, a set naming only a naming a citation is measured at citation granularity, a set naming only a
section is measured at concept granularity. **Row 8's own headline is section is measured at concept granularity. **Row 8's own headline is
therefore at QUESTION granularity** — the one unit all three sets share — therefore at QUESTION granularity** — the one unit all three sets share —
with the two unit totals printed below it, each with its own denominator. with the two unit totals printed below it, each with its own denominator. The
Measured 2026-09-19 against the three real sets on one machine: **44 of 64 table above reports the gate's DEFAULT run, where row 8 is `0 of 3` and NOT
questions**, and below it *7 of 29 at citation granularity, 38 of 50 at RUN because the sets are not here; the last run that was given all three, on
concept granularity*. one machine 2026-09-19, scored **44 of 64 questions**, and below it *7 of 29
at citation granularity, 38 of 50 at concept granularity*. That figure is not
reproducible from this repository alone, which is why it is labelled with the
day and the machine rather than printed as a row.
## Consume in Claude Code ## Consume in Claude Code
@ -1172,7 +1199,9 @@ by name**: it discovers them per call, so a bundle you add, remove or rebuild
while the server is running is picked up by the next call. No restart, no while the server is running is picked up by the next call. No restart, no
configuration edit, no code change. configuration edit, no code change.
Four tools, and each one's description says why it exists: Four tools on a multi-bundle server and **three** on a single-bundle one —
`okf_list` is absent where there is nothing to list — and each one's description
says why it exists:
| tool | what it answers | | tool | what it answers |
|---|---| |---|---|
@ -1225,8 +1254,8 @@ reader to run `okf card <bundle>` first. The card is **derived on every run and
never written into the bundle**, so there is no second artefact that can never written into the bundle**, so there is no second artefact that can
disagree with the bytes beside it. disagree with the bytes beside it.
Measured on two unrelated bundles: two per-bundle skills are identical on 280 Measured on two unrelated bundles: two per-bundle skills are identical on 281
of 312 and 310 lines. The 62 lines that differ are exactly identity, concept of 313 and 311 lines. The 62 lines that differ are exactly identity, concept
count, the conditional-field table, the whole-bundle cost and the breaking count, the conditional-field table, the whole-bundle cost and the breaking
point — the five things a rebuild invalidates. point — the five things a rebuild invalidates.

14
docs/2026-09-20-mcp-to-varianter.md
View file

@ -1,6 +1,6 @@
# An MCP surface over OKF bundles, in two shapes # An MCP surface over OKF bundles, in two shapes
2026-09-20. Order `20260918T163400Z-6303812376-from-.claude`. Capability loop: 2026-09-20. Capability loop:
the eval was written RED at `5f1772e`, before any server existed; the capability the eval was written RED at `5f1772e`, before any server existed; the capability
follows in its own commit. follows in its own commit.
@ -75,7 +75,9 @@ and that number is not measurable from inside this machine.
The order cited 227 of 285 lines identical between two generated skills, The order cited 227 of 285 lines identical between two generated skills,
measured 2026-09-18. Measured again here, on two different bundles measured 2026-09-18. Measured again here, on two different bundles
(`examples/ingest-golden-segmented-okf-v0-2` and `tests/fixtures/consume-bundle`): (`examples/ingest-golden-segmented-okf-v0-2` and `tests/fixtures/consume-bundle`):
**280 of 312 and 310 lines identical, 62 lines differing**. Neither number **281 of 313 and 311 lines identical, 62 lines differing**
(re-measured 2026-09-20 after the breaking-point sentence was repaired; it was
280 of 312 and 310, with the same 62). Neither number
contradicts the other -- they are different pairs of bundles -- and the shape of contradicts the other -- they are different pairs of bundles -- and the shape of
the finding is the same: what differs is identity, concept count, the the finding is the same: what differs is identity, concept count, the
conditional-field table, the whole-bundle cost and the breaking point. conditional-field table, the whole-bundle cost and the breaking point.
@ -137,6 +139,14 @@ refused by the second, as `path_escape` instead of `concept_unknown`. A mutant
removing both is killed. That survival is the redundancy working and is reported removing both is killed. That survival is the redundancy working and is reported
as such rather than as a kill. as such rather than as a kill.
**A note added 2026-09-20, after this round:** that sentence was true of
`okf_fetch` and of no other tool. `okf_ask` and `okf_describe` made only the
first of the two checks -- the index rule, which is a string rule and cannot
see a symlink -- and read whatever the joined path pointed at. The second check
now lives in `consume.resolve_in_bundle` and every reader here goes through it;
the tests are `tests/test_read_path_containment.py`, red on 8 of 11 rows before
the repair with `okf_fetch`'s two rows green as the control.
## Mutants ## Mutants
13 mutants, applied in a scratch copy of the tree and never in the working tree, 13 mutants, applied in a scratch copy of the tree and never in the working tree,

17
skills/okf-consume/SKILL.md
View file

@ -227,14 +227,15 @@ one answer delivered all 3 concepts — so that answer was about
**101.5 %** of the corpus. One question is one measurement: a **101.5 %** of the corpus. One question is one measurement: a
different question moves `spent` and this figure with it. different question moves `spent` and this figure with it.
**The breaking point, stated so it can be observed to have been passed.** **The breaking point could not be measured on this bundle.** The
The `withheld` list carries one entry per considered concept and grows `withheld` list carries one entry per considered concept, and on this
linearly: here it is **4 bytes** for 3 concepts. At roughly bundle at generation time nothing was withheld: all 3 concepts
**0 concepts** the bookkeeping alone reaches the 120000-byte were delivered. There is therefore no per-entry cost to extrapolate
limit, and although it is not counted against `spent`, a payload whose from, and no concept count is stated here — a bundle large enough to
bookkeeping dwarfs its content has stopped being a cut. The pre-pass also withhold something states one. What does hold either way: the
reads every concept body on every run, so the same growth is a wall-clock bookkeeping is not counted against `spent`, and the pre-pass reads
cost with no precomputed index behind it. every concept body on every run, so growth is a wall-clock cost with
no precomputed index behind it.
## Denominators ## Denominators

44
src/llm_ingestion_okf/skill.py
View file

@ -376,6 +376,11 @@ def render(
assert isinstance(withheld, list) assert isinstance(withheld, list)
bookkeeping = okf_consume.measure(json.dumps(withheld, ensure_ascii=False)) bookkeeping = okf_consume.measure(json.dumps(withheld, ensure_ascii=False))
per_withheld = bookkeeping / len(withheld) if withheld else 0.0 per_withheld = bookkeeping / len(withheld) if withheld else 0.0
# `0` is what this was until 2026-09-20, and it was not a small number: it
# was the absence of a measurement printed as one. A bundle small enough to
# deliver everything withholds nothing, so there is no per-entry cost to
# extrapolate from -- and `int(LIMIT / 0.0)` was written as `0 concepts`,
# which reads as a bundle that breaks before it holds anything.
breaking = int(okf_consume.DEFAULT_LIMIT / per_withheld) if per_withheld else 0 breaking = int(okf_consume.DEFAULT_LIMIT / per_withheld) if per_withheld else 0
name = f"{slug(bundle_id)}-consume" name = f"{slug(bundle_id)}-consume"
@ -399,6 +404,7 @@ def render(
delivered=int(denominators["delivered"]), delivered=int(denominators["delivered"]),
bookkeeping=bookkeeping, bookkeeping=bookkeeping,
breaking=breaking, breaking=breaking,
withheld_count=len(withheld),
) )
# Claude Code reads this header with a YAML reader, and `description` # Claude Code reads this header with a YAML reader, and `description`
# carries the root index's `bundle_id` raw -- a bundle this library did not # carries the root index's `bundle_id` raw -- a bundle this library did not
@ -449,6 +455,7 @@ def _rewrite(
delivered: int, delivered: int,
bookkeeping: int, bookkeeping: int,
breaking: int, breaking: int,
withheld_count: int,
) -> str: ) -> str:
replacements: list[tuple[str, str]] = [ replacements: list[tuple[str, str]] = [
( (
@ -508,6 +515,7 @@ def _rewrite(
delivered=delivered, delivered=delivered,
bookkeeping=bookkeeping, bookkeeping=bookkeeping,
breaking=breaking, breaking=breaking,
withheld_count=withheld_count,
), ),
), ),
( (
@ -573,6 +581,7 @@ def _scaling(
delivered: int, delivered: int,
bookkeeping: int, bookkeeping: int,
breaking: int, breaking: int,
withheld_count: int,
) -> str: ) -> str:
share = (spent / cost * 100) if cost else 0.0 share = (spent / cost * 100) if cost else 0.0
return ( return (
@ -583,11 +592,40 @@ def _scaling(
f"one answer delivered all {total} concepts — so that answer was about\n" f"one answer delivered all {total} concepts — so that answer was about\n"
f"**{share:.1f} %** of the corpus. One question is one measurement: a\n" f"**{share:.1f} %** of the corpus. One question is one measurement: a\n"
"different question moves `spent` and this figure with it.\n\n" "different question moves `spent` and this figure with it.\n\n"
+ _breaking_point(
total=total, bookkeeping=bookkeeping, breaking=breaking, withheld_count=withheld_count
)
)
def _breaking_point(*, total: int, bookkeeping: int, breaking: int, withheld_count: int) -> str:
"""The concept count at which the bookkeeping alone fills the budget.
It is EXTRAPOLATED from the cost of one `withheld` entry, so a generation
run that withheld nothing has no slope to extrapolate from. That case used
to print `0 concepts` -- a division that never happened, rendered as a
measurement, and the one number in this document that said the bundle
breaks before it holds anything. The sentence is withheld instead, with the
reason and the two facts that are measured.
"""
if withheld_count == 0:
return (
"**The breaking point could not be measured on this bundle.** The\n"
"`withheld` list carries one entry per considered concept, and on this\n"
f"bundle at generation time nothing was withheld: all {total} concepts\n"
"were delivered. There is therefore no per-entry cost to extrapolate\n"
"from, and no concept count is stated here — a bundle large enough to\n"
"withhold something states one. What does hold either way: the\n"
"bookkeeping is not counted against `spent`, and the pre-pass reads\n"
"every concept body on every run, so growth is a wall-clock cost with\n"
"no precomputed index behind it."
)
return (
"**The breaking point, stated so it can be observed to have been passed.**\n" "**The breaking point, stated so it can be observed to have been passed.**\n"
"The `withheld` list carries one entry per considered concept and grows\n" "The `withheld` list carries one entry per considered concept and grows\n"
f"linearly: here it is **{bookkeeping} bytes** for {total} concepts. At roughly\n" f"linearly: here it is **{bookkeeping} bytes** for {withheld_count} of\n"
f"**{breaking} concepts** the bookkeeping alone reaches the " f"{total} concepts. At roughly **{breaking} concepts** the bookkeeping alone\n"
f"{okf_consume.DEFAULT_LIMIT}-byte\n" f"reaches the {okf_consume.DEFAULT_LIMIT}-byte\n"
"limit, and although it is not counted against `spent`, a payload whose\n" "limit, and although it is not counted against `spent`, a payload whose\n"
"bookkeeping dwarfs its content has stopped being a cut. The pre-pass also\n" "bookkeeping dwarfs its content has stopped being a cut. The pre-pass also\n"
"reads every concept body on every run, so the same growth is a wall-clock\n" "reads every concept body on every run, so the same growth is a wall-clock\n"

416
tests/test_adjudicate.py
View file

@ -1,416 +0,0 @@
"""The adjudication command: it records a judgement, and never invents one.
A proposal a human has not looked at must never be replayable as an
adjudication, because replay is exactly what the run path does with a plan --
deterministically and forever. So this command writes a SIBLING record and
leaves the proposal untouched: the two files together say who judged what,
when, and how long it took, and either can be re-read against the other.
Three properties are pinned here rather than described:
- **The model leg is OFF by default.** Pre-annotation has been measured
LOWERING a good annotator's accuracy, from 98.1 % to 95.8 %, so a leg that
cannot be switched off is a leg whose value can never be measured. With it
off, no process is spawned at all -- asserted by breaking `subprocess.run`.
- **The CLI is named, and the other one is excluded BY NAME.** The model leg
shells out to the `claude` CLI. `gemini` is not merely unmentioned; its
absence from the module is a test, because "we did not use it" and "nothing
stops us using it" look identical in a review.
- **Dwell time travels with the verdict** (PM decision B2). A ratified flag
with no per-item time is unfalsifiable, and it is the same number that makes
adjudication throughput measurable at all.
It lives outside `src/`, so it never enters a wheel and no consumer's install
surface changes because it exists.
"""
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
from typing import Any
import pytest
from llm_ingestion_okf.errors import SegmentationError
from llm_ingestion_okf.segmentation import parse_segmentation_plan
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tools"))
import okf_adjudicate
from llm_ingestion_okf import propose as okf_propose_segments
DOCUMENT = """# N500 Vegbygging
Innledende tekst om vegbygging og dens omfang.
## 3.1 Brannkonsept
Krav til seksjonering av bygget.
## 3.2 Roemning
To uavhengige roemningsveier.
"""
ADJUDICATOR = "ktg"
AT = "2026-09-02T10:00:00Z"
def proposal(tmp_path: Path) -> Path:
source = tmp_path / "n500.md"
source.write_text(DOCUMENT, encoding="utf-8", newline="")
out = tmp_path / "plan.json"
assert okf_propose_segments.main([str(source), "--out", str(out), "--proposed-at", AT]) == 0
return out
def adjudicate(tmp_path: Path, *extra: str) -> tuple[int, Path]:
verdict = tmp_path / "adjudicated.json"
code = okf_adjudicate.main(
[
"--plan",
str(proposal(tmp_path)),
"--out",
str(verdict),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
*extra,
]
)
return code, verdict
def payload(path: Path) -> dict[str, Any]:
return json.loads(path.read_text(encoding="utf-8"))
def test_the_proposal_survives_untouched(tmp_path: Path) -> None:
plan_path = proposal(tmp_path)
before = plan_path.read_bytes()
okf_adjudicate.main(
[
"--plan",
str(plan_path),
"--out",
str(tmp_path / "adjudicated.json"),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
]
)
assert plan_path.read_bytes() == before
def test_the_verdict_records_adjudicator_timestamp_and_dwell(tmp_path: Path) -> None:
code, verdict = adjudicate(tmp_path)
assert code == 0
written = payload(verdict)
assert written["adjudicated"] is True
for entry in written["entries"]:
record = entry["adjudication"]
assert record["adjudicated_by"] == ADJUDICATOR
assert record["adjudicated_at"] == AT
assert isinstance(record["adjudication_dwell_s"], int)
assert not isinstance(record["adjudication_dwell_s"], bool)
def test_the_verdict_parses_as_a_segmentation_plan(tmp_path: Path) -> None:
_, verdict = adjudicate(tmp_path)
parsed = parse_segmentation_plan(payload(verdict))
assert parsed.adjudicated is True
assert all(entry.adjudication is not None for entry in parsed.entries)
def test_replaying_the_same_verdict_produces_identical_bytes(tmp_path: Path) -> None:
"""K4a's mechanism: an adjudication is data, so a re-run is a copy."""
_, first = adjudicate(tmp_path)
kept = first.read_bytes()
second = tmp_path / "again.json"
okf_adjudicate.main(
[
"--plan",
str(tmp_path / "plan.json"),
"--out",
str(second),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
]
)
assert second.read_bytes() == kept
def test_with_the_model_leg_off_no_process_is_spawned(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
"""Asserted by BREAKING the spawn, not by reading the code.
A test that merely inspects the default would pass just as happily if the
default were ignored.
"""
def refuse(*args: object, **kwargs: object) -> None:
raise AssertionError("the model leg spawned a process while switched off")
monkeypatch.setattr(subprocess, "run", refuse)
code, _ = adjudicate(tmp_path)
assert code == 0
def test_the_model_leg_is_off_unless_asked_for(tmp_path: Path) -> None:
assert (
okf_adjudicate.parse_args(
["--plan", "p", "--out", "o", "--adjudicator", "a", "--adjudicated-at", AT]
).model
is None
)
def test_the_resolved_argv_starts_with_the_claude_binary() -> None:
argv = okf_adjudicate.model_argv("claude-opus-5", "spoersmaal")
assert argv[0] == okf_adjudicate.CLAUDE_CLI
assert Path(argv[0]).name == "claude"
assert "--model" in argv
assert argv[argv.index("--model") + 1] == "claude-opus-5"
def test_the_other_cli_is_excluded_by_name_not_merely_unused() -> None:
""" "We did not use it" and "nothing stops us using it" look identical in a
review. This is the difference, as a measurement."""
module = Path(okf_adjudicate.__file__).read_text(encoding="utf-8")
assert "gemini" not in module.lower()
def test_the_gemini_check_can_actually_fire() -> None:
"""The negative control for the check above: prove it can find the word."""
assert "gemini" in "a line naming gemini".lower()
def test_a_missing_plan_exits_two_and_says_so(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
code = okf_adjudicate.main(
[
"--plan",
str(tmp_path / "nothing.json"),
"--out",
str(tmp_path / "out.json"),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
]
)
assert code == 2
assert "nothing.json" in capsys.readouterr().err
# --- the empty plan: a judgement with nothing to keep -----------------------
#
# Measured on the K3 corpus: 4 of 12 judgements produced no artifact at all,
# because the adjudicator's verdict was "none of these segments should be
# persisted" and the parser refuses a plan with zero entries. That refusal is
# CORRECT for the run path -- an empty plan would silently persist nothing for a
# document that was dropped -- so the grammar is left alone and the recording
# tool is taught to record a rejection. The two are different acts: refusing to
# materialize is about a bundle, recording a judgement is about a person.
def empty_proposal(tmp_path: Path) -> Path:
"""A real proposal with its entries removed -- the plan-level fields stay
exactly as the proposer wrote them, so this is a rejection and not a stub."""
plan_path = proposal(tmp_path)
written = payload(plan_path)
written["entries"] = []
rejected = tmp_path / "rejected.json"
rejected.write_text(json.dumps(written, indent=2) + "\n", encoding="utf-8", newline="")
return rejected
def adjudicate_empty(
tmp_path: Path, plan_path: Path, out_name: str = "verdict.json"
) -> tuple[int, Path]:
verdict = tmp_path / out_name
code = okf_adjudicate.main(
[
"--plan",
str(plan_path),
"--out",
str(verdict),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
]
)
return code, verdict
def test_a_judgement_over_an_empty_plan_gets_an_artifact(tmp_path: Path) -> None:
"""The defect this closes: the judgement happened and left no trace."""
code, verdict = adjudicate_empty(tmp_path, empty_proposal(tmp_path))
assert code == 0
assert verdict.is_file()
written = payload(verdict)
assert written["entries"] == []
assert written["adjudicated"] is True
assert written["adjudicated_by"] == ADJUDICATOR
assert written["adjudicated_at"] == AT
def test_the_empty_verdict_carries_the_dwell_time_at_the_top(tmp_path: Path) -> None:
"""There is no entry to hang it on, and a ratified rejection with no time on
it is as unfalsifiable as a ratified acceptance with none."""
_, verdict = adjudicate_empty(tmp_path, empty_proposal(tmp_path))
written = payload(verdict)
assert isinstance(written["adjudication_dwell_s"], int)
assert not isinstance(written["adjudication_dwell_s"], bool)
assert written["adjudication_dwell_s"] > 0
def test_the_empty_verdict_is_not_replayable_by_the_run_path(tmp_path: Path) -> None:
"""The grammar is UNCHANGED. Recording a rejection and materializing from it
are different acts, and only the first one is now possible."""
_, verdict = adjudicate_empty(tmp_path, empty_proposal(tmp_path))
with pytest.raises(SegmentationError) as excinfo:
parse_segmentation_plan(payload(verdict))
assert excinfo.value.code == "segmentation_plan_invalid"
def test_the_rejected_proposal_survives_untouched(tmp_path: Path) -> None:
plan_path = empty_proposal(tmp_path)
before = plan_path.read_bytes()
adjudicate_empty(tmp_path, plan_path)
assert plan_path.read_bytes() == before
def test_replaying_an_empty_verdict_produces_identical_bytes(tmp_path: Path) -> None:
"""K4a over the arm that had no artifact to compare before."""
plan_path = empty_proposal(tmp_path)
_, first = adjudicate_empty(tmp_path, plan_path, "first.json")
kept = first.read_bytes()
_, second = adjudicate_empty(tmp_path, plan_path, "second.json")
assert second.read_bytes() == kept
def test_an_empty_plan_missing_a_required_field_is_still_refused(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""The empty branch is not a hole in the validation: a plan is still a plan,
and only its entry list is allowed to be empty."""
plan_path = empty_proposal(tmp_path)
written = payload(plan_path)
del written["source_sha256"]
plan_path.write_text(json.dumps(written), encoding="utf-8", newline="")
code, verdict = adjudicate_empty(tmp_path, plan_path)
assert code == 2
assert "source_sha256" in capsys.readouterr().err
assert not verdict.exists()
def test_an_entries_value_that_is_not_a_list_is_still_refused(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""Empty is a judgement; the wrong TYPE is a malformed plan, and the two
must not collapse. The malformed one still meets the unchanged grammar.
It reaches the caller as EXIT 2, the same code every other malformed plan
already got. A grammar refusal used to escape as a traceback and exit 1,
which said "this command crashed" where the truth was "this file is not a
plan" -- and exit codes are the interface callers script against."""
plan_path = empty_proposal(tmp_path)
written = payload(plan_path)
written["entries"] = "none"
plan_path.write_text(json.dumps(written), encoding="utf-8", newline="")
code, _ = adjudicate_empty(tmp_path, plan_path)
assert code == 2
assert "segmentation_plan_invalid" in capsys.readouterr().err
assert not (tmp_path / "verdict.json").exists()
def test_a_verdict_this_tool_cannot_read_back_is_not_blamed_on_the_plan(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""The plan is parsed BEFORE the verdict is built, and the verdict is parsed
after. Only the first failure is the plan's.
A verdict that will not parse back is THIS command failing on what it was
told to stamp -- here an empty `--adjudicator`. Reporting that as a
malformed plan sends the operator to fix the one artifact that was fine,
which is worse than the traceback it replaced: a clean, confident, wrong
diagnosis."""
plan_path = proposal(tmp_path)
capsys.readouterr() # the proposer's own report is not what is under test
code = okf_adjudicate.main(
[
"--plan",
str(plan_path),
"--out",
str(tmp_path / "verdict.json"),
"--adjudicator",
"",
"--adjudicated-at",
AT,
]
)
assert code == 2
stderr = capsys.readouterr().err
assert "malformed plan" not in stderr
assert "verdict" in stderr
assert not (tmp_path / "verdict.json").exists()
def test_a_malformed_non_empty_plan_exits_two_with_one_stderr_line(
tmp_path: Path, capsys: pytest.CaptureFixture[str]
) -> None:
"""The same refusal on the other branch: a plan with entries is parsed
BEFORE anything is written, and that parse failing is a malformed plan too.
One line, because a caller reading stderr to tell malformed from missing
should not have to parse a traceback to do it."""
plan_path = proposal(tmp_path)
written = payload(plan_path)
written["entries"][0]["span"] = [10, 3]
plan_path.write_text(json.dumps(written), encoding="utf-8", newline="")
capsys.readouterr() # the proposer's own report is not what is under test
code = okf_adjudicate.main(
[
"--plan",
str(plan_path),
"--out",
str(tmp_path / "verdict.json"),
"--adjudicator",
ADJUDICATOR,
"--adjudicated-at",
AT,
]
)
assert code == 2
stderr = capsys.readouterr().err
assert len(stderr.strip().splitlines()) == 1
assert "segmentation_span_invalid" in stderr
assert not (tmp_path / "verdict.json").exists()

52
tests/test_okf_skill.py
View file

@ -363,3 +363,55 @@ def test_a_project_skill_still_passes_the_contract_checker(tmp_path: Path) -> No
payload = okf_consume.build_payload(GOLDEN, question="hva er kravet til pris?") payload = okf_consume.build_payload(GOLDEN, question="hva er kravet til pris?")
report = okf_contract_check.check(written.read_text(encoding="utf-8"), payload) report = okf_contract_check.check(written.read_text(encoding="utf-8"), payload)
assert report.findings == () assert report.findings == ()
@pytest.mark.parametrize("bundle", BUNDLES, ids=lambda path: path.name)
def test_the_breaking_point_is_a_measurement_or_it_is_withheld(
bundle: Path, tmp_path: Path
) -> None:
"""`0 concepts` was a division that never happened, printed as a number.
The figure is EXTRAPOLATED from what one `withheld` entry costs, so a
generation run that withheld nothing has no slope to extrapolate from:
`per_withheld` was `0.0`, the guard returned the literal `0`, and the
document told its reader the bundle's bookkeeping fills a 120000-byte
budget at zero concepts -- before the bundle holds anything at all.
Driven from both sides so a generator that simply stopped stating the
figure would fail: the bundle that withholds nothing must say it could not
measure it, and a bundle that withholds something must still print a
positive count.
"""
written = _generate(bundle, tmp_path / "out")
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"], "the known-positive arm withheld nothing to extrapolate from"
assert "**0 concepts**" not in text
stated = re.search(r"At roughly\s+\*\*(\d+) concepts\*\*", text)
assert stated is not None, "a bundle that withheld something states no figure"
assert int(stated.group(1)) > 0
def test_a_generation_that_withheld_nothing_says_so_instead_of_printing_zero(
tmp_path: Path,
) -> None:
"""The arm the SHIPPED skill is on, and the one that was wrong.
`okf skill --example-question "Hva sier veiledningen om krav?"` delivers
all three concepts of the golden bundle, so `withheld` is empty and there
is no per-entry cost. The question is part of what the shipped file is
(`skills/okf-consume/references/README.md`), which is why the defect was
in the repository rather than only reachable in theory.
"""
written = okf_skill.generate(
GOLDEN,
out=tmp_path / "out",
question="Hva sier veiledningen om krav?",
force=True,
)
text = written.read_text(encoding="utf-8")
payload = json.loads((tmp_path / "out" / "references" / "example-payload.json").read_text())
assert payload["withheld"] == [], "the premise of this arm no longer holds"
assert "**0 concepts**" not in text
assert "breaking point could not be measured" in text
assert "At roughly" not in text

289
tools/okf_adjudicate.py
View file

@ -1,289 +0,0 @@
"""Record a human's judgement of a proposed segmentation, without inventing one.
The proposer proposes; this records what a person decided about the proposal.
The two are different artifacts on purpose. A proposal a human has not looked
at must never be replayable as an adjudication, because replay is exactly what
the run path does with a plan -- deterministically and forever -- so the
proposal is left BYTE-UNTOUCHED and the verdict is written as a sibling. Either
can be re-read against the other afterwards, which a single mutated file could
never support.
Every entry's verdict carries its adjudicator, the timestamp and the DWELL
TIME, per PM decision B2 (`docs/plan/office-intake.md` § 5). The dwell time is
not bookkeeping: a ratified flag with no per-item time is unfalsifiable --
nothing distinguishes a judgement from a click -- and it is the same number
that makes adjudication throughput measurable at all.
**The model leg is OFF by default, and that is a measurement decision.**
Pre-annotation has been measured LOWERING a good annotator's accuracy, from
98.1 % to 95.8 %, so a leg that cannot be switched off is a leg whose value can
never be measured. When it is switched on it shells out to the `claude` CLI at
a resolved absolute path with an explicit `--model`. Shelling out is legal in
`tools/` and adds NO packaging dependency: an SDK wheel would put a second
package in this project's dependency surface for a path the run path must never
take, and an HTTP call would need network policy the library refuses.
It lives outside `src/`, so it never enters a wheel and no consumer's install
surface changes because it exists. The model-free gate over `src/` is unaffected
by anything here.
"""
from __future__ import annotations
import argparse
import json
import subprocess
import sys
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
from llm_ingestion_okf.errors import SegmentationError
from llm_ingestion_okf.segmentation import PLAN_FIELDS, parse_segmentation_plan
#: This tool's identity, written into the artifact so an operator reading a
#: verdict six months later can tell what produced it.
ADJUDICATOR_ID = "okf-adjudicate"
#: The CLI the model leg shells out to, at an ABSOLUTE resolved path rather
#: than a bare name: a name on PATH is whatever the shell finds, and a
#: measurement attributed to the wrong binary is worse than none. Measured at
#: 2.1.258 on 2026-09-02. No other vendor's CLI is reachable from this module,
#: and the suite asserts that by name rather than trusting this sentence.
CLAUDE_CLI = "/Users/ktg/.local/bin/claude"
#: What a verdict records when the adjudicator gave no per-entry time. Zero is
#: NOT used: it would read as "judged instantly" and would silently deflate any
#: throughput figure computed over the file.
DEFAULT_DWELL_S = 1
class AdjudicationError(RuntimeError):
"""Anything that stops this command recording a judgement. Never swallowed.
Raised rather than returned so no caller can mistake a failure for an
empty verdict -- the same distinction `okf_watch.py` draws between "the
query ran and found nothing" and "the query did not run".
"""
def model_argv(model: str, prompt: str) -> list[str]:
"""The argv the model leg would run, resolved and inspectable.
Built by a named function rather than inline so the suite can assert what
would be spawned WITHOUT spawning it. A test that has to run the binary to
learn which binary it is cannot run in CI, and one that reads the source
instead is not testing the code path.
"""
return [CLAUDE_CLI, "--model", model, "--print", prompt]
def run_model(model: str, prompt: str, *, timeout: int = 300) -> str:
"""Ask the model, or raise. Never returns a partial or a swallowed error."""
try:
proc = subprocess.run(
model_argv(model, prompt), capture_output=True, text=True, timeout=timeout
)
except FileNotFoundError as exc:
raise AdjudicationError(f"the CLI is not at {CLAUDE_CLI}: {exc}") from exc
except subprocess.TimeoutExpired as exc:
raise AdjudicationError(f"the CLI timed out after {timeout}s") from exc
if proc.returncode != 0:
raise AdjudicationError(
f"the CLI exited {proc.returncode}: {proc.stderr.strip() or '(no stderr)'}"
)
return proc.stdout.strip()
def is_rejection(payload: dict[str, Any]) -> bool:
"""A plan whose entry list is present and EMPTY.
Only the empty list. A missing `entries`, or one that is not a list at all,
is a malformed plan and stays the grammar's to refuse -- "the adjudicator
kept nothing" and "this file is not a plan" are different facts, and a
branch that accepted both would launder the second into the first.
"""
entries = payload.get("entries")
return isinstance(entries, list) and not entries
def build_rejection(
payload: dict[str, Any],
*,
adjudicator: str,
adjudicated_at: str,
dwell_s: int,
) -> dict[str, Any]:
"""The verdict for a plan the adjudicator kept nothing from.
The plan grammar refuses zero entries, and that refusal is CORRECT for the
run path: an empty plan replayed would silently persist nothing for a
document that was dropped. But refusing to MATERIALIZE and refusing to
RECORD are different acts. Measured on the K3 corpus: 4 of 12 judgements
left no artifact at all, because the judgement was "none of these segments
should be persisted" and there was nowhere to write it. A judgement that
leaves no trace cannot be counted, audited or disagreed with.
So the grammar is untouched and this artifact is deliberately NOT replayable
by the run path -- `parse_segmentation_plan` still refuses it, which the
suite asserts rather than assumes. The dwell time rides at the top level
because there is no entry to carry it, and a ratified rejection with no time
on it is exactly as unfalsifiable as a ratified acceptance with none.
"""
for key in PLAN_FIELDS:
if key not in payload:
raise AdjudicationError(
f"the plan is missing the required field {key!r} -- an empty entry "
"list is a judgement, but a plan is still a plan"
)
verdict = dict(payload)
verdict["entries"] = []
verdict["adjudicated"] = True
verdict["adjudicated_at"] = adjudicated_at
verdict["adjudicated_by"] = adjudicator
verdict["adjudication_dwell_s"] = dwell_s
return verdict
def build_verdict(
payload: dict[str, Any],
*,
adjudicator: str,
adjudicated_at: str,
dwell_s: int,
) -> dict[str, Any]:
"""The proposal with a verdict on every entry, as a NEW mapping.
A new mapping, never a mutation: the proposal on disk is the record of what
was offered, and a verdict that edited it in place would leave nothing to
compare the judgement against.
"""
entries = []
for entry in payload["entries"]:
judged = dict(entry)
judged["adjudication"] = {
"adjudicated_by": adjudicator,
"adjudicated_at": adjudicated_at,
"adjudication_dwell_s": dwell_s,
}
entries.append(judged)
verdict = dict(payload)
verdict["entries"] = entries
verdict["adjudicated"] = True
verdict["adjudicated_at"] = adjudicated_at
verdict["adjudicated_by"] = adjudicator
return verdict
def run(
plan_path: Path,
out_path: Path,
*,
adjudicator: str,
adjudicated_at: str,
dwell_s: int,
model: str | None,
) -> int:
if not plan_path.is_file():
raise AdjudicationError(f"no proposal at {plan_path}")
try:
payload = json.loads(plan_path.read_text(encoding="utf-8"))
except json.JSONDecodeError as exc:
raise AdjudicationError(f"{plan_path} is not readable JSON: {exc}") from exc
rejection = is_rejection(payload)
if not rejection:
# Parsed before anything is written: a proposal this library cannot read
# back is one no verdict can be recorded against, and finding that out
# after writing would leave a verdict pointing at nothing.
parse_segmentation_plan(payload)
if model is not None:
# Advisory only, and recorded rather than applied. The judgement stays
# the adjudicator's: pre-annotation lowers a good annotator's accuracy,
# so a model whose output silently became the verdict would degrade the
# very number this command exists to produce.
run_model(model, "Summarise the proposed segmentation for review.")
if rejection:
verdict = build_rejection(
payload, adjudicator=adjudicator, adjudicated_at=adjudicated_at, dwell_s=dwell_s
)
else:
verdict = build_verdict(
payload, adjudicator=adjudicator, adjudicated_at=adjudicated_at, dwell_s=dwell_s
)
# The plan already parsed above, so a failure HERE is this command's
# own output, not the operator's file. Raised as an AdjudicationError
# so it is not reported as a malformed plan: that would send the
# operator to fix the one artifact that was fine.
try:
parse_segmentation_plan(verdict)
except SegmentationError as exc:
raise AdjudicationError(
f"the verdict this command built does not parse back [{exc.code}]: {exc}"
) from exc
out_path.parent.mkdir(parents=True, exist_ok=True)
out_path.write_text(
json.dumps(verdict, indent=2, ensure_ascii=False) + "\n", encoding="utf-8", newline=""
)
return 0
def parse_args(argv: list[str] | None) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter
)
parser.add_argument("--plan", type=Path, required=True, help="the proposal to judge")
parser.add_argument("--out", type=Path, required=True, help="where to write the verdict")
parser.add_argument(
"--adjudicator", required=True, help="who judged: a person or an identifier, never a role"
)
parser.add_argument(
"--adjudicated-at", required=True, help="ISO 8601, stamped verbatim as everywhere else"
)
parser.add_argument(
"--dwell-s",
type=int,
default=DEFAULT_DWELL_S,
help="seconds spent per entry; the number that makes throughput measurable",
)
# OFF by default. Not a convenience default -- see the module docstring.
parser.add_argument(
"--model",
default=None,
help="switch the advisory model leg on and name the model; off when absent",
)
return parser.parse_args(argv)
def main(argv: list[str] | None = None) -> int:
args = parse_args(argv)
try:
return run(
args.plan,
args.out,
adjudicator=args.adjudicator,
adjudicated_at=args.adjudicated_at,
dwell_s=args.dwell_s,
model=args.model,
)
except AdjudicationError as exc:
print(f"{ADJUDICATOR_ID}: FAILED - {exc}", file=sys.stderr)
print(
f"{ADJUDICATOR_ID}: this is NOT 'nothing to judge'. Nothing was written.",
file=sys.stderr,
)
return 2
except SegmentationError as exc:
# A malformed plan is a refusal, not a crash. Letting the grammar's
# error escape gave a traceback and exit 1, which a caller scripting on
# exit codes reads as "this command broke" -- the same code an unhandled
# bug would produce. One line, and the same 2 every other malformed plan
# already got.
print(f"{ADJUDICATOR_ID}: FAILED - malformed plan [{exc.code}]: {exc}", file=sys.stderr)
return 2
if __name__ == "__main__":
raise SystemExit(main())