First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
10 KiB
P1 method trial (TRIAL, branch p1-metodeprove, not v1 code)
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
small real examples (PLAN.md § Rammer, order 20261002T094056Z-495685981, plan v3).
- K: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (
p1-compiler) writes short fragments per pack of five sources, and code resolves concept identity and assembles the pages. - K+: a copy of K plus one consolidation pass (
p1-consolidator) that rewrites pages with three or more fragments into flowing text, keeping every footnote label. - D: no model. A generated map of each source and a code-built concept register.
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
.claude/agents/. Every output lives outside the repository, under <wiki>/prove-p1/ (CLAUDE.md rule 3); this README
records numbers only.
Modules
Run each as PYTHONPATH=evals:. python3 -m p1.<module> ...; each module's docstring is its usage.
| Module | What it does |
|---|---|
split.py |
the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
dmap.py |
D: map pages and register pages |
kpack.py |
K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
assemble.py |
K's pages from accepted fragments; K+'s copy, tasks and apply |
measure.py |
the audit record, coverage of source blocks, structure counts |
cost.py |
tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
sidebyside.py |
the side-by-side page and its link check |
Output files under prove-p1/: rekkefolge.txt, blocks.json, begreper.txt, okter.txt, D/, K/, K+/,
resultat.json, struktur-<V>.json, dekning-<V>.json, kost.json, side-om-side.html, rapport.md.
Tests: python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'. Checks: bash tests/p1_repo.sh (repository)
and bash tests/p1_trial.sh (the trial's outputs; red until the runs are done).
Results
Numbers only, filled in by the run steps (plan steps 8-11).
Run 0 (step 8, 02.10.2026)
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
in all 599 files of
kilde/(the D12 projection base). - Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under
prove-p1/, so the agents' work directory isprove-p1/(noruntime/p1fallback).p1-compiler: startup 18604, 2 turns, 39142 tokens in all;p1-consolidator: startup 18167, 2 turns, 38270 tokens in all. - D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
source in a body footnote (D21 counts body-cited entries only; D's locators live in
sources[]). Duplicate pairs 9 (5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
Run K (step 9, 02.10.2026)
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
- The cost harvester was fixed twice during the run, each with a failing test first (
77b16ab,0a50286): a long streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth, less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on the load test's known short call, which keeps its recorded value. The recorded value stays beside it inkost.json. - K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
with a concept not among the lines sent / turns / tokens):
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
- No fragment was rejected for any reason (incl.
release-note-new0); no new concept was proposed, so all 650 accepted fragments enrich skeleton entries. Merge candidates 0.
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418); startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated); flat 21919125.
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424 (4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean 1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
Run K+ (step 10, 02.10.2026)
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its output back).
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with 8 of 76 pages consolidated.
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221, precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs, 63 islands, version numbers in running text 5 of 662.
Step 11: the side-by-side table (02.10.2026)
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
trimming (26f521f) and the concept pick counts a document only where the term stands as a term (09a50b2). The
first pick and D built before 26f521f are kept in prove-p1/forkastet/; D was rebuilt (252 pages: 19 map, 233
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
| L2b / L4 cost | measured 27-28.09, not measured again | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | L2b about 1.6-1.8 bn writing + 8.6 bn reading |
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost by block share. The projection is not validated (4 points).
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart): 10.9 M tokens up to step 11, almost all cache read.
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31, 1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9; D 0 for every source (its register pages cite nothing in a body).
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from several sources.
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
token as D7 is written;
cost.summarizeand the stop command have no test.