First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
125 lines
10 KiB
Markdown
125 lines
10 KiB
Markdown
# P1 method trial (TRIAL, branch `p1-metodeprove`, not v1 code)
|
|
|
|
The P1 method trial builds the 19 sources of the cut (the sources behind d01 d02 d03 d04 d37 d46 d53) three ways, in one
|
|
fixed order, and measures cost and quality side by side, so the operator can choose the build method for v1 by seeing
|
|
small real examples (PLAN.md § Rammer, order `20261002T094056Z-495685981`, plan v3).
|
|
|
|
- **K**: the fragment compiler. Code splits the sources into blocks, a narrow trial agent (`p1-compiler`) writes short
|
|
fragments per pack of five sources, and code resolves concept identity and assembles the pages.
|
|
- **K+**: a copy of K plus one consolidation pass (`p1-consolidator`) that rewrites pages with three or more fragments
|
|
into flowing text, keeping every footnote label.
|
|
- **D**: no model. A generated map of each source and a code-built concept register.
|
|
|
|
No module here calls a model. The model work is done in the operator's session by the two trial agent types under
|
|
`.claude/agents/`. Every output lives outside the repository, under `<wiki>/prove-p1/` (CLAUDE.md rule 3); this README
|
|
records numbers only.
|
|
|
|
## Modules
|
|
|
|
Run each as `PYTHONPATH=evals:. python3 -m p1.<module> ...`; each module's docstring is its usage.
|
|
|
|
| Module | What it does |
|
|
|---|---|
|
|
| `split.py` | the fixed source order, blocks with line spans, the block total, the three side-by-side concepts |
|
|
| `dmap.py` | D: map pages and register pages |
|
|
| `kpack.py` | K: the skeleton from every docs page, pack input parts, fragment validation, concept identity |
|
|
| `assemble.py` | K's pages from accepted fragments; K+'s copy, tasks and apply |
|
|
| `measure.py` | the audit record, coverage of source blocks, structure counts |
|
|
| `cost.py` | tokens from the run sessions' transcripts, tokens per source, projection, the stop rule |
|
|
| `sidebyside.py` | the side-by-side page and its link check |
|
|
|
|
Output files under `prove-p1/`: `rekkefolge.txt`, `blocks.json`, `begreper.txt`, `okter.txt`, `D/`, `K/`, `K+/`,
|
|
`resultat.json`, `struktur-<V>.json`, `dekning-<V>.json`, `kost.json`, `side-om-side.html`, `rapport.md`.
|
|
|
|
Tests: `python3 -m unittest discover -s evals/tests -p 'test_p1_*.py'`. Checks: `bash tests/p1_repo.sh` (repository)
|
|
and `bash tests/p1_trial.sh` (the trial's outputs; red until the runs are done).
|
|
|
|
## Results
|
|
|
|
Numbers only, filled in by the run steps (plan steps 8-11).
|
|
|
|
### Run 0 (step 8, 02.10.2026)
|
|
|
|
- Order: 19 sources (3 docs pages, then 16 release notes by version), equal to the files cited by the L4 cut.
|
|
- Blocks: 1221 in the 19 sources (docs pages 599, release notes 622; pack 1 = sources 1-5 = 602 blocks); 33727 blocks
|
|
in all 599 files of `kilde/` (the D12 projection base).
|
|
- Blocks per source in order: 59, 308, 232, 1, 2, 22, 35, 76, 21, 25, 57, 35, 34, 52, 33, 22, 46, 53, 108.
|
|
- Load test (the empty baseline run, not counted toward any cap): both trial types spawned and wrote under `prove-p1/`,
|
|
so the agents' work directory is `prove-p1/` (no `runtime/p1` fallback). `p1-compiler`: startup 18604, 2 turns,
|
|
39142 tokens in all; `p1-consolidator`: startup 18167, 2 turns, 38270 tokens in all.
|
|
- D: 254 pages (19 map, 235 register), 0 model tokens. Auditor exit 0, 0 findings. Criterion 1: 0 of 0 (bodies are
|
|
headings and link lists only, D8, by construction). Coverage 0 of 1221 blocks: by construction, because D cites no
|
|
source in a body footnote (D21 counts body-cited entries only; D's locators live in `sources[]`). Duplicate pairs 9
|
|
(5 by name, 4 by title Jaccard), merge candidates 0, islands 0, concept test 2 held by 254 pages, test 4 by 19.
|
|
|
|
### Run K (step 9, 02.10.2026)
|
|
|
|
- Skeleton: 3493 index entries from the titles and level 2-3 headings of all 198 docs pages (0 tokens).
|
|
- The cost harvester was fixed twice during the run, each with a failing test first (`77b16ab`, `0a50286`): a long
|
|
streamed message keeps only its first usage record in the transcript (output 2, no stop reason), so output was
|
|
undercounted (373 recorded for pack 1's 336 fragments). Output is now derived from the next message's input growth,
|
|
less the injected text counted high (2.5 characters per token), so every K output below is a lower bound. Checked on
|
|
the load test's known short call, which keeps its recorded value. The recorded value stays beside it in `kost.json`.
|
|
- K: per pack (blocks / index lines sent of 3493 / input parts / fragments accepted of proposed / rejected / accepted
|
|
with a concept not among the lines sent / turns / tokens):
|
|
- pack 1: 602 / 230 / 4 / 336 of 336 / 0 / 0 / 3 / 372996
|
|
- pack 2: 179 / 117 / 1 / 87 of 87 / 0 / 0 / 3 / 125716
|
|
- pack 3: 211 / 125 / 1 / 108 of 108 / 0 / 0 / 3 / 141320
|
|
- pack 4: 229 / 127 / 1 / 119 of 119 / 0 / 0 / 3 / 153494
|
|
- No fragment was rejected for any reason (incl. `release-note-new` 0); no new concept was proposed, so all 650
|
|
accepted fragments enrich skeleton entries. Merge candidates 0.
|
|
- Stop line after pack 1: projected per block 756525 (naive per source 1417385), cap 1000000: continue.
|
|
- K total: 793526 tokens in 4 runs and 12 turns (input 24, cache read 228534, cache write 429550, output 135418);
|
|
startup 74596 (4 x about 18.6 k). Tokens per source, mean of sources 1-5: 74599; of sources 15-19: 35119.
|
|
- Projected to all 599 sources (33727 blocks): fit 52842866 (4 points, growth term b = 0.0554, not validated);
|
|
flat 21919125.
|
|
- Auditor exit 0, 0 findings. Criterion 1: 664 of 664. Quotes: 650 entries, 160 supplied by code, 0 on a structural
|
|
line. Precision of body-cited entries: line 591, block 59, section 0, document 0.
|
|
- Coverage: 623 of 1221 blocks. Per position in the pack (covered of blocks, n packs): 1: 85 of 160 (4); 2: 241 of 424
|
|
(4); 3: 181 of 395 (4); 4: 83 of 182 (4); 5: 33 of 60 (3). A direction signal, n = 3-4.
|
|
- Structure: 212 concept pages, 3281 skeleton entries unused. Concepts per source in order: 11, 61, 31, 1, 0, 5, 13,
|
|
15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11. Duplicate pairs 3 (all title Jaccard), islands 63, documents per page mean
|
|
1.65 (max 8), version numbers in running text 5 of 664 claims, concept test 2 held by 212, test 4 by 73.
|
|
- Seen in the compiler's own reports: in one pack it wrote numbers as words to avoid the quote rule (D5 requires a
|
|
quote when the text holds a digit); the compilers' own fragment counts differed from the files (the files count).
|
|
|
|
### Run K+ (step 10, 02.10.2026)
|
|
|
|
- K+: a copy of K; 76 pages carry 3 or more fragments, which gives 10 consolidation runs of up to 8 pages.
|
|
- Run 1: 8 pages, applied 8, refused 0. 109651 tokens (input 8, cache read 72768, cache write 29909, output 6966, a lower
|
|
bound), 4 turns, startup 18176. The consolidator took one more turn than its instructions allow (it read its
|
|
output back).
|
|
- Stop line after run 1: 109651 x 76 / 8 = 1041685 > cap 1000000: STOP. Runs 2-10 were not run; K+ stands as K with
|
|
8 of 76 pages consolidated.
|
|
- K+ total = K + consolidation = 793526 + 109651 = 903177 tokens.
|
|
- Auditor exit 0, 0 findings. Criterion 1: 662 of 662 (2 fewer claims than K: sentences joined). Coverage 623 of 1221,
|
|
precision and quotes as K (line 591, block 59; 160 code quotes, 0 structural). 212 pages, 3 duplicate pairs,
|
|
63 islands, version numbers in running text 5 of 662.
|
|
|
|
### Step 11: the side-by-side table (02.10.2026)
|
|
|
|
Corrections after run 0, each its own commit with a failing test first: the backticked-term length is checked after
|
|
trimming (`26f521f`) and the concept pick counts a document only where the term stands as a term (`09a50b2`). The
|
|
first pick and D built before `26f521f` are kept in `prove-p1/forkastet/`; D was rebuilt (252 pages: 19 map, 233
|
|
register; 9 duplicate pairs; auditor exit 0) and the concepts were picked again by the same rule.
|
|
|
|
| variant | built to source | concepts (total / per source 1-5, 15-19 / test 2 / test 4) | skeleton unused | merge candidates | coverage | tokens per source (input / cache read / cache write / output / startup) | 1-5 | 15-19 | turns / subagents | projected 599 (fit / flat) | criterion 1 | fragments accepted / code quotes / structural quotes | precision (line / block / section / document) | pages / islands / duplicates |
|
|
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
|
| K | 19 of 19 | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | 1 / 12028 / 22608 / 7127 / 3926 | 74599 | 35119 | 12 / 4 | 52842866 / 21919125 | 664 of 664 | 650 of 650 / 160 / 0 | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
|
| K+ | 19 of 19, consolidation stopped after run 1 (8 of 76 pages) | 212 / 104, 45 / 212 / 73 | 3281 of 3493 | 0 | 623 of 1221 | consolidation 109651 for 8 pages; K+ total = K + consolidation = 903177 | - | - | 4 / 1 | not computed: stopped after 1 of 10 runs, and the pages to consolidate at 599 sources are not measured | 662 of 662 | as K | 591 / 59 / 0 / 0 | 212 / 63 / 3 |
|
|
| D | 19 of 19 | 0 concept pages (252 map and register) / 0, 0 by construction / 252 / 19 | - | 0 | 0 of 1221 (by construction) | 0 | 0 | 0 | 0 / 0 | 0 | 0 of 0 (by construction) | - | - | 252 / 0 / 9 |
|
|
| L4 (structure counted today) | 19 of 19 | 151 / 33, 33 / 151 / 68 | - | 0 | 685 of 1221 | see the cost row | - | - | - | - | 1048 of 1048 | - / 0 / 41 of 832 | 832 / 0 / 0 / 0 | 151 / 54 / 2 |
|
|
| L2b / L4 cost | measured 27-28.09, not measured again | | | | | L4 105.8 M for 19 sources (84 % cache read); L2b writing 30.93 M for 19 sources | | | | L2b about 1.6-1.8 bn writing + 8.6 bn reading | | | | |
|
|
|
|
- Every K and K+ output number is a lower bound (the harvester fix in run K). Tokens per source split each pack's cost
|
|
by block share. The projection is not validated (4 points).
|
|
- Side-by-side page: 3 concepts x 4 columns; 454 of 454 source links resolve. Main session (orchestration, apart):
|
|
10.9 M tokens up to step 11, almost all cache read.
|
|
- Concepts per source in order 1-19 (a page counts at the earliest source its body cites, D10): K and K+ 11, 61, 31,
|
|
1, 0, 5, 13, 15, 6, 3, 7, 4, 6, 4, 7, 5, 9, 13, 11; L4 2, 9, 19, 1, 2, 10, 14, 17, 5, 6, 14, 10, 7, 2, 9, 6, 3, 6, 9;
|
|
D 0 for every source (its register pages cite nothing in a body).
|
|
- K+ consolidation cost 13706 per page (lower bound) and is not split per source, because a page joins fragments from
|
|
several sources.
|
|
- Seen in the review (BLOCK, not fixed in this run): 45 of 1221 blocks are only MDX tag lines and cannot be covered
|
|
(K 623 of 1176 without them); the index-line filter sends a line when the whole name occurs as a phrase, not per
|
|
token as D7 is written; `cost.summarize` and the stop command have no test.
|