test(accounting): row 6 sees a refusal, a second real corpus, and 34 of 34 mutants
Row 6 was GREEN with R761 100 % rejected: every element of a refused document is booked as a coded rejection, so u = 0 and d = 0 and the bundle is empty. `refused_whole` asks that question on its own now -- the build order asked for an honest red there, and PM re-measured the green on 2026-09-18 with `okf build` exiting 1 unseen. A skipped row no longer leaves the verdict unqualified (`GATE GREEN (row 6 not run: ...)`), and the exit code is non-zero locally when a corpus source is on the machine and its row did not run. m-2. N200 Vegbygging:2024 joins R761 as a second real corpus. R761 holds 0 `fig`, 0 formulas and 0 references, so the only real corpus could not have found the hole in the STS role map; N200 carries 194 citations, 49 figures and 135 footnotes. A `.json` whose root holds an STS node tree is counted as STS rather than as keys and leaves -- the container is not the content. M-4: the review's 26 mutants, ported to the code as it stands, plus 8 for the new checks. 34 of 34 killed. `tools/okf_gate_mutants.py` runs on a copy of the tree, and a pattern that does not match is an ERROR and exit 2 -- a mutant that could not be applied was never measured. That fired once, on M13, after a refactor moved the line it edits. m-3: `APPROVED_EXCEPTIONS` was read by no row, so approving one changed nothing. Each pair is now checked against the witness's own vocabulary and the run says why it moves no denominator. The gate also prints its OWN limits beside the verdict, m-5 among them. The product's accounting tests state the new truth instead of the old one: `okf build --accounting` over the fixture corpus exits 1 with SIX unaccounted elements in its own vocabulary -- its first real finding, reachable only now that fixtures carry the constructs. Four shared element names disagree with the witness, each pinned with its cause; one of the four is a double count this package makes (a text box's paragraph, once inside the box and again in the paragraph carrying it). Three fixture defects were found and fixed while building them, each of which would have reported a loss the build never had: a shared string table not related to the workbook, a `graphicData` with no `uri`, and an odt `styles.xml` without `<office:styles/>`. Report: docs/2026-09-18-regnskapsgaten-herdet.md Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
e5dc21ec2f
commit
ed8d9d709f
SSH key fingerprint:
SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 1146 additions and 85 deletions
215
docs/2026-09-18-regnskapsgaten-herdet.md
Normal file
215
docs/2026-09-18-regnskapsgaten-herdet.md
Normal file
|
|
@ -0,0 +1,215 @@
|
|||
# The content-accounting gate, hardened: the judge opens the bundle itself
|
||||
|
||||
2026-09-18. Order `20260917T142837Z-445369650`, from an independent adversarial
|
||||
review of `0b00de4` commissioned after the accounting was built. The review's
|
||||
verdict was that the gate is "an honest RED today, but cannot be trusted as a
|
||||
judge once it turns green": it was a calculator over a report the judged party
|
||||
writes. This is what changed, measured.
|
||||
|
||||
Nothing in `src/` moved. The instrument was hardened; the thing it judges was
|
||||
not touched.
|
||||
|
||||
## 1. BLOCKER B-1 — the gate never opened a concept file
|
||||
|
||||
`okf_accounting_gate.account()` compared BOOKED NUMBERS with the witness's
|
||||
counts. Reproduced here before any code moved, as two tests over the real
|
||||
fixture bundle: a report that changes not one byte of the bundle and books
|
||||
every element as `carried` gave `GATE GREEN`, exit 0 — and so did booking
|
||||
every element as `rejected`.
|
||||
|
||||
**The witness now gives every element the PIECES OF TEXT IT IS MADE OF, and
|
||||
the gate looks for each of them in the concept bodies the run wrote.** Pieces
|
||||
rather than one joined string, and that distinction is load-bearing: a reader
|
||||
writes a heading's ATX marker and a picture's pointer block between the parts
|
||||
of a container, so an STS `<sec>` is never one contiguous run in the bundle
|
||||
even when every word of it is there. Measured — with one joined string per
|
||||
element, 2 of the 13 fixture documents reported false losses; with pieces, 0.
|
||||
|
||||
The normalisation is the one the build already uses and no looser (strip
|
||||
non-alphanumerics, casefold, remove a converter attribute block such as
|
||||
`{.mark}`), and it is implemented in the gate rather than imported from
|
||||
`accounting.py`: a judge borrowing the judged party's own comparison agrees
|
||||
with it by construction.
|
||||
|
||||
Five more refusals, each driven from both sides by a test:
|
||||
|
||||
- a negative booking (`carried: 25, rejected: {x: -15}` of 10 read as clean);
|
||||
- a document declared `persisted` that no concept in the bundle names;
|
||||
- a document declared `rejected` that the bundle holds a concept from;
|
||||
- a rejection code outside a closed list, at the document AND at the element;
|
||||
- an `accounting_version` the gate does not read.
|
||||
|
||||
And: **a document the build PERSISTED whose report carries nothing from it is
|
||||
never clean.** "Everything rejected" satisfied `u = 0, d = 0`.
|
||||
|
||||
**m-1:** an asset proved a carry by NAME, so a zero-byte file called
|
||||
`<sha12>-x.png` read as a carry. It is now checked on the BYTES and on the
|
||||
content address. The first attempt re-implemented the build's full naming rule
|
||||
and reported **50 of 50** carried R761 images as missing — the build
|
||||
lowercases the basename, folds separator runs and takes the suffix from the
|
||||
bytes. The readable tail is the build's cosmetic choice and is deliberately
|
||||
not reproduced here.
|
||||
|
||||
## 2. MAJOR M-1 and M-2 — what the witnesses could not see
|
||||
|
||||
Rows 2 and 3 require the build's inventory to EQUAL the witness's, so what the
|
||||
witness does not count, nothing can lose visibly. Thirteen classes were added,
|
||||
each with a red test first:
|
||||
|
||||
| format | now counted |
|
||||
|---|---|
|
||||
| `.docx` | header/footer, comment, endnote, text box |
|
||||
| `.pptx` | speaker note, hidden slide (`show="0"`, no longer an ordinary slide) |
|
||||
| `.xlsx` | formula, hidden sheet |
|
||||
| `.odt` | header/footer (they live in `styles.xml`), annotation |
|
||||
| `.xml` | `mixed-citation`, `mml:math`, `fig`, a figure's caption |
|
||||
| `.pdf` | the page's own TEXT (it counted pages and picture placements only) |
|
||||
|
||||
A text box's paragraphs are the box's own, or the text is booked twice. A
|
||||
hidden sheet's state lives in `workbook.xml` and is reached through the
|
||||
relationship id, so the sheet part itself says nothing about it. An annotation
|
||||
counted as prose makes the accounting demand that a reader carry a note the
|
||||
author wrote to themselves.
|
||||
|
||||
**M-2:** the two STS witnesses shared ONE role map, so row 5 — "two witnesses
|
||||
agree" — could not see a hole in it. They are written apart now. Measured on
|
||||
R761 2026-09-18: the XML delivery places a section's label at `sec/label`
|
||||
**7 714** times and inside a title **0** times; the JSON delivery puts
|
||||
**2 760** of them inside the title. The XML map was narrowed to the XML fact,
|
||||
and a JSON twin fixture carrying the nested placement holds the difference
|
||||
live — without it, swapping the maps changed nothing measurable.
|
||||
|
||||
## 3. MAJOR M-3 — 20 of 63 element types had a count of zero
|
||||
|
||||
A witness cannot be caught being wrong about something it never sees; that is
|
||||
why six of seven witness mutants survived the original suite. Seven hand-built
|
||||
documents close it (`tests/fixtures/accounting/make_accounting_fixtures.py`),
|
||||
every element type now occurs at least once, and **all twenty documents carry
|
||||
a hand count** read off the fixture's own bytes — four did before. The
|
||||
operator's own proposed `.xlsx image` exception could not be exercised at all
|
||||
until now.
|
||||
|
||||
Building them found three defects in the fixtures themselves, each of which
|
||||
would have reported a loss the build never had: a shared string table not
|
||||
related to the workbook (every `t="s"` cell converts empty), a `graphicData`
|
||||
with no `uri` (pandoc does not see a table), and an `odt` `styles.xml` without
|
||||
`<office:styles/>` (pandoc refuses the document).
|
||||
|
||||
## 4. Row 6 and the verdict line
|
||||
|
||||
- **R761 refused whole under the default gate is now RED.** It was GREEN with
|
||||
31 096 of 31 096 elements booked as coded rejections and `okf build` exiting
|
||||
1 unseen: every element had a fate, so `u = 0, d = 0`. The row asks the
|
||||
question separately now.
|
||||
- **A skipped row never leaves the verdict unqualified.** `GATE GREEN` now
|
||||
reads `GATE GREEN (row 6 not run: <reason>)`, and the exit code is non-zero
|
||||
locally when a corpus source exists on the machine and the row did not run.
|
||||
- **N200 Vegbygging:2024 is a second real corpus.** R761 holds **0** `fig`,
|
||||
**0** formulas and **0** references, so the gate's only real corpus could not
|
||||
have found the hole in the role map. N200 carries **194** citations, **49**
|
||||
figures and **135** footnotes. A `.json` file whose root carries an STS node
|
||||
tree is counted as STS rather than as keys and leaves — the container is not
|
||||
the content.
|
||||
|
||||
## 5. MAJOR M-4 — mutation
|
||||
|
||||
The review's 26 mutants, ported to the code as it stands, plus 8 written for
|
||||
the new checks: **34 of 34 killed** (21 of the review's 26 were killed before
|
||||
the last seven tests were added; all 26 are killed now, against the order's bar
|
||||
of 23). The harness is `tools/okf_gate_mutants.py`, it runs on a copy of the
|
||||
tree, and **a pattern that does not match is an ERROR and exit 2** — a mutant
|
||||
that could not be applied was never measured, and counting it as killed is the
|
||||
same mistake as reading an empty search as an absence. That fired once, on
|
||||
M13, after a refactor moved the line it edits.
|
||||
|
||||
## 6. The gate is REDDER than before, and that is the movement
|
||||
|
||||
At the shipped defaults, `python3 tools/okf_accounting_gate.py`, 3 m 51 s:
|
||||
|
||||
```
|
||||
1 file types with a fasit fixture | 13 of 13 | GREEN
|
||||
2 source inventory before build | 8 of 13 | RED
|
||||
3 accounting after build | 17 of 22 | RED
|
||||
4 a rejected document is reported | 1 of 1 | GREEN
|
||||
5 two witnesses agree | 4 of 4 | GREEN
|
||||
6 real corpora | 110 of 111| RED
|
||||
GATE RED: rows 2, 3, 6
|
||||
```
|
||||
|
||||
Row 3: **u = 19 unaccounted, d = 2 double-booked, 0 booked carried and not in
|
||||
the bundle, 0 declarations the gate refuses** — and **314 carried elements
|
||||
FOUND in the bundle**, 6 carrying no text the gate can check. The zero matters
|
||||
as much as the nineteen: nothing the build DOES book as carried failed the
|
||||
bundle check, so every red is the build's and none is the instrument's.
|
||||
|
||||
**The new "not carried" classes, per file type, with counts on the fixtures** —
|
||||
the raw material for the next capability order and for the operator's
|
||||
exception list:
|
||||
|
||||
| file type | not carried | on the fixtures |
|
||||
|---|---|---|
|
||||
| `.docx` | comment, endnote, footnote, header/footer, text box | 1, 1, 1, 2, 1 |
|
||||
| `.odt` | annotation, header/footer, image | 1, 2, 1 |
|
||||
| `.pptx` | hidden slide, speaker note | 1, 1 |
|
||||
| `.xlsx` | formula, hidden sheet, image | 1, 1, 1 |
|
||||
| `.xml` (STS) | citation, figure, figure caption, math | 1, 1, 1, 1 |
|
||||
|
||||
Two are DOUBLE-booked rather than lost: a hidden slide and a hidden sheet are
|
||||
carried into the bundle but booked as an ordinary slide and an ordinary sheet.
|
||||
A third is a genuine double count in this package: **a text box's paragraph is
|
||||
counted twice**, once inside the box and once in the paragraph carrying it.
|
||||
|
||||
On the real corpora: R761 under `--gate none` is **31 081 carried elements
|
||||
found in the bundle, 0 claimed and not found, 65 the gate cannot check**, over
|
||||
a document holding 31 096. N200 is **16 549 elements with no fate at all**,
|
||||
because `okf build` exits 2 on it (see § 7).
|
||||
|
||||
`okf build --accounting` over the fixture corpus now exits 1 with **6
|
||||
unaccounted** in the build's OWN vocabulary — its first real finding, and it
|
||||
only became reachable when the fixtures carried the constructs.
|
||||
|
||||
## 7. Two facts about `okf build` this run establishes
|
||||
|
||||
**R761 is felled 100 % by ONE rule, and it is not tier-dependent.** The
|
||||
document is refused whole under the default `guard-trusted-source` with
|
||||
`fail_secure`. Asked directly, guard 1.4.0 answers
|
||||
`any-tier: invisible carrier (output:zero-width-present)`, severity HIGH,
|
||||
assessment SEVERE — and `PRESET_USER_UPLOAD` gives the same disposition, so no
|
||||
tier this package offers persists it. The cause is measured rather than
|
||||
inferred: the document carries **71 U+00AD soft hyphens**, and removing them
|
||||
takes the same text to `WARN`, which persists. Whether that is intended is
|
||||
below.
|
||||
|
||||
**N200 gets 0 plans and exit 2.** The JSON delivery is fenced verbatim, so no
|
||||
line-anchored boundary grammar fires — the same behaviour a structureless
|
||||
`.txt` produces, documented in `CLAUDE.md`. It is not an accounting defect, and
|
||||
the gate reports it as what it is: 16 549 elements with no fate.
|
||||
|
||||
## 8. What this gate still cannot check
|
||||
|
||||
Printed on every run, beside the verdict, because a gate that reports only the
|
||||
build's gaps invites a green row to be read as a guarantee:
|
||||
|
||||
- a short element often stands elsewhere in the same document, so finding it
|
||||
proves it is present and not that THIS one is;
|
||||
- two pointed files with identical bytes are one content-addressed asset, so
|
||||
one of them losing its pointer is invisible here (m-5);
|
||||
- an image embedded in a binary container has no source file to hash;
|
||||
- absence is never verified: an element booked REJECTED is not looked for;
|
||||
- the witness is a second implementation of the same definitions, so a
|
||||
definition that is wrong for a format is wrong on both sides at once.
|
||||
|
||||
Every witness also prints, per file type, the named list of what it still does
|
||||
not count. `APPROVED_EXCEPTIONS` was read by no row at all (m-3); each pair is
|
||||
now checked against the witness's vocabulary and the run says, per pair, why it
|
||||
moves no denominator — and would say `WARNING` if one ever did.
|
||||
|
||||
## 9. Reproduction
|
||||
|
||||
```
|
||||
uv run python tools/okf_accounting_gate.py # 3 m 51 s, exit 1
|
||||
uv run python tools/okf_gate_mutants.py # 34 of 34 killed, exit 0
|
||||
uv run pytest tests/test_accounting_gate.py tests/test_content_accounting.py -q
|
||||
uv run python tools/okf_witness.py tests/fixtures/accounting/corpus
|
||||
python3 tests/fixtures/accounting/make_accounting_fixtures.py
|
||||
```
|
||||
Loading…
Add table
Add a link
Reference in a new issue