open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 4 Activity

test(accounting): row 6 sees a refusal, a second real corpus, and 34 of 34 mutants

Browse source 
Row 6 was GREEN with R761 100 % rejected: every element of a refused document
is booked as a coded rejection, so u = 0 and d = 0 and the bundle is empty.
`refused_whole` asks that question on its own now -- the build order asked for
an honest red there, and PM re-measured the green on 2026-09-18 with
`okf build` exiting 1 unseen.

A skipped row no longer leaves the verdict unqualified (`GATE GREEN (row 6 not
run: ...)`), and the exit code is non-zero locally when a corpus source is on
the machine and its row did not run. m-2.

N200 Vegbygging:2024 joins R761 as a second real corpus. R761 holds 0 `fig`,
0 formulas and 0 references, so the only real corpus could not have found the
hole in the STS role map; N200 carries 194 citations, 49 figures and 135
footnotes. A `.json` whose root holds an STS node tree is counted as STS
rather than as keys and leaves -- the container is not the content.

M-4: the review's 26 mutants, ported to the code as it stands, plus 8 for the
new checks. 34 of 34 killed. `tools/okf_gate_mutants.py` runs on a copy of the
tree, and a pattern that does not match is an ERROR and exit 2 -- a mutant
that could not be applied was never measured. That fired once, on M13, after a
refactor moved the line it edits.

m-3: `APPROVED_EXCEPTIONS` was read by no row, so approving one changed
nothing. Each pair is now checked against the witness's own vocabulary and the
run says why it moves no denominator. The gate also prints its OWN limits
beside the verdict, m-5 among them.

The product's accounting tests state the new truth instead of the old one:
`okf build --accounting` over the fixture corpus exits 1 with SIX unaccounted
elements in its own vocabulary -- its first real finding, reachable only now
that fixtures carry the constructs. Four shared element names disagree with
the witness, each pinned with its cause; one of the four is a double count
this package makes (a text box's paragraph, once inside the box and again in
the paragraph carrying it).

Three fixture defects were found and fixed while building them, each of which
would have reported a loss the build never had: a shared string table not
related to the workbook, a `graphicData` with no `uri`, and an odt
`styles.xml` without `<office:styles/>`.

Report: docs/2026-09-18-regnskapsgaten-herdet.md

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-18 02:48:34 +02:00
commit ed8d9d709f
Signed by: ktg
SSH key fingerprint: SHA256:JakMjO6FTBBzN0Bhfj9saOoEjaFxlSdYuZQQpM/lF9Q
11 changed files with 1146 additions and 85 deletions
Download patch file Download diff file Expand all files Collapse all files

215
docs/2026-09-18-regnskapsgaten-herdet.md Normal file
View file

@ -0,0 +1,215 @@
# The content-accounting gate, hardened: the judge opens the bundle itself
2026-09-18. Order `20260917T142837Z-445369650`, from an independent adversarial
review of `0b00de4` commissioned after the accounting was built. The review's
verdict was that the gate is "an honest RED today, but cannot be trusted as a
judge once it turns green": it was a calculator over a report the judged party
writes. This is what changed, measured.
Nothing in `src/` moved. The instrument was hardened; the thing it judges was
not touched.
## 1. BLOCKER B-1 — the gate never opened a concept file
`okf_accounting_gate.account()` compared BOOKED NUMBERS with the witness's
counts. Reproduced here before any code moved, as two tests over the real
fixture bundle: a report that changes not one byte of the bundle and books
every element as `carried` gave `GATE GREEN`, exit 0 — and so did booking
every element as `rejected`.
**The witness now gives every element the PIECES OF TEXT IT IS MADE OF, and
the gate looks for each of them in the concept bodies the run wrote.** Pieces
rather than one joined string, and that distinction is load-bearing: a reader
writes a heading's ATX marker and a picture's pointer block between the parts
of a container, so an STS `<sec>` is never one contiguous run in the bundle
even when every word of it is there. Measured — with one joined string per
element, 2 of the 13 fixture documents reported false losses; with pieces, 0.
The normalisation is the one the build already uses and no looser (strip
non-alphanumerics, casefold, remove a converter attribute block such as
`{.mark}`), and it is implemented in the gate rather than imported from
`accounting.py`: a judge borrowing the judged party's own comparison agrees
with it by construction.
Five more refusals, each driven from both sides by a test:
- a negative booking (`carried: 25, rejected: {x: -15}` of 10 read as clean);
- a document declared `persisted` that no concept in the bundle names;
- a document declared `rejected` that the bundle holds a concept from;
- a rejection code outside a closed list, at the document AND at the element;
- an `accounting_version` the gate does not read.
And: **a document the build PERSISTED whose report carries nothing from it is
never clean.** "Everything rejected" satisfied `u = 0, d = 0`.
**m-1:** an asset proved a carry by NAME, so a zero-byte file called
`<sha12>-x.png` read as a carry. It is now checked on the BYTES and on the
content address. The first attempt re-implemented the build's full naming rule
and reported **50 of 50** carried R761 images as missing — the build
lowercases the basename, folds separator runs and takes the suffix from the
bytes. The readable tail is the build's cosmetic choice and is deliberately
not reproduced here.
## 2. MAJOR M-1 and M-2 — what the witnesses could not see
Rows 2 and 3 require the build's inventory to EQUAL the witness's, so what the
witness does not count, nothing can lose visibly. Thirteen classes were added,
each with a red test first:
| format | now counted |
|---|---|
| `.docx` | header/footer, comment, endnote, text box |
| `.pptx` | speaker note, hidden slide (`show="0"`, no longer an ordinary slide) |
| `.xlsx` | formula, hidden sheet |
| `.odt` | header/footer (they live in `styles.xml`), annotation |
| `.xml` | `mixed-citation`, `mml:math`, `fig`, a figure's caption |
| `.pdf` | the page's own TEXT (it counted pages and picture placements only) |
A text box's paragraphs are the box's own, or the text is booked twice. A
hidden sheet's state lives in `workbook.xml` and is reached through the
relationship id, so the sheet part itself says nothing about it. An annotation
counted as prose makes the accounting demand that a reader carry a note the
author wrote to themselves.
**M-2:** the two STS witnesses shared ONE role map, so row 5 — "two witnesses
agree" — could not see a hole in it. They are written apart now. Measured on
R761 2026-09-18: the XML delivery places a section's label at `sec/label`
**7 714** times and inside a title **0** times; the JSON delivery puts
**2 760** of them inside the title. The XML map was narrowed to the XML fact,
and a JSON twin fixture carrying the nested placement holds the difference
live — without it, swapping the maps changed nothing measurable.
## 3. MAJOR M-3 — 20 of 63 element types had a count of zero
A witness cannot be caught being wrong about something it never sees; that is
why six of seven witness mutants survived the original suite. Seven hand-built
documents close it (`tests/fixtures/accounting/make_accounting_fixtures.py`),
every element type now occurs at least once, and **all twenty documents carry
a hand count** read off the fixture's own bytes — four did before. The
operator's own proposed `.xlsx image` exception could not be exercised at all
until now.
Building them found three defects in the fixtures themselves, each of which
would have reported a loss the build never had: a shared string table not
related to the workbook (every `t="s"` cell converts empty), a `graphicData`
with no `uri` (pandoc does not see a table), and an `odt` `styles.xml` without
`<office:styles/>` (pandoc refuses the document).
## 4. Row 6 and the verdict line
- **R761 refused whole under the default gate is now RED.** It was GREEN with
31 096 of 31 096 elements booked as coded rejections and `okf build` exiting
1 unseen: every element had a fate, so `u = 0, d = 0`. The row asks the
question separately now.
- **A skipped row never leaves the verdict unqualified.** `GATE GREEN` now
reads `GATE GREEN (row 6 not run: <reason>)`, and the exit code is non-zero
locally when a corpus source exists on the machine and the row did not run.
- **N200 Vegbygging:2024 is a second real corpus.** R761 holds **0** `fig`,
**0** formulas and **0** references, so the gate's only real corpus could not
have found the hole in the role map. N200 carries **194** citations, **49**
figures and **135** footnotes. A `.json` file whose root carries an STS node
tree is counted as STS rather than as keys and leaves — the container is not
the content.
## 5. MAJOR M-4 — mutation
The review's 26 mutants, ported to the code as it stands, plus 8 written for
the new checks: **34 of 34 killed** (21 of the review's 26 were killed before
the last seven tests were added; all 26 are killed now, against the order's bar
of 23). The harness is `tools/okf_gate_mutants.py`, it runs on a copy of the
tree, and **a pattern that does not match is an ERROR and exit 2** — a mutant
that could not be applied was never measured, and counting it as killed is the
same mistake as reading an empty search as an absence. That fired once, on
M13, after a refactor moved the line it edits.
## 6. The gate is REDDER than before, and that is the movement
At the shipped defaults, `python3 tools/okf_accounting_gate.py`, 3 m 51 s:
```
1 file types with a fasit fixture | 13 of 13 | GREEN
2 source inventory before build | 8 of 13 | RED
3 accounting after build | 17 of 22 | RED
4 a rejected document is reported | 1 of 1 | GREEN
5 two witnesses agree | 4 of 4 | GREEN
6 real corpora | 110 of 111| RED
GATE RED: rows 2, 3, 6
```
Row 3: **u = 19 unaccounted, d = 2 double-booked, 0 booked carried and not in
the bundle, 0 declarations the gate refuses** — and **314 carried elements
FOUND in the bundle**, 6 carrying no text the gate can check. The zero matters
as much as the nineteen: nothing the build DOES book as carried failed the
bundle check, so every red is the build's and none is the instrument's.
**The new "not carried" classes, per file type, with counts on the fixtures** —
the raw material for the next capability order and for the operator's
exception list:
| file type | not carried | on the fixtures |
|---|---|---|
| `.docx` | comment, endnote, footnote, header/footer, text box | 1, 1, 1, 2, 1 |
| `.odt` | annotation, header/footer, image | 1, 2, 1 |
| `.pptx` | hidden slide, speaker note | 1, 1 |
| `.xlsx` | formula, hidden sheet, image | 1, 1, 1 |
| `.xml` (STS) | citation, figure, figure caption, math | 1, 1, 1, 1 |
Two are DOUBLE-booked rather than lost: a hidden slide and a hidden sheet are
carried into the bundle but booked as an ordinary slide and an ordinary sheet.
A third is a genuine double count in this package: **a text box's paragraph is
counted twice**, once inside the box and once in the paragraph carrying it.
On the real corpora: R761 under `--gate none` is **31 081 carried elements
found in the bundle, 0 claimed and not found, 65 the gate cannot check**, over
a document holding 31 096. N200 is **16 549 elements with no fate at all**,
because `okf build` exits 2 on it (see § 7).
`okf build --accounting` over the fixture corpus now exits 1 with **6
unaccounted** in the build's OWN vocabulary — its first real finding, and it
only became reachable when the fixtures carried the constructs.
## 7. Two facts about `okf build` this run establishes
**R761 is felled 100 % by ONE rule, and it is not tier-dependent.** The
document is refused whole under the default `guard-trusted-source` with
`fail_secure`. Asked directly, guard 1.4.0 answers
`any-tier: invisible carrier (output:zero-width-present)`, severity HIGH,
assessment SEVERE — and `PRESET_USER_UPLOAD` gives the same disposition, so no
tier this package offers persists it. The cause is measured rather than
inferred: the document carries **71 U+00AD soft hyphens**, and removing them
takes the same text to `WARN`, which persists. Whether that is intended is
below.
**N200 gets 0 plans and exit 2.** The JSON delivery is fenced verbatim, so no
line-anchored boundary grammar fires — the same behaviour a structureless
`.txt` produces, documented in `CLAUDE.md`. It is not an accounting defect, and
the gate reports it as what it is: 16 549 elements with no fate.
## 8. What this gate still cannot check
Printed on every run, beside the verdict, because a gate that reports only the
build's gaps invites a green row to be read as a guarantee:
- a short element often stands elsewhere in the same document, so finding it
proves it is present and not that THIS one is;
- two pointed files with identical bytes are one content-addressed asset, so
one of them losing its pointer is invisible here (m-5);
- an image embedded in a binary container has no source file to hash;
- absence is never verified: an element booked REJECTED is not looked for;
- the witness is a second implementation of the same definitions, so a
definition that is wrong for a format is wrong on both sides at once.
Every witness also prints, per file type, the named list of what it still does
not count. `APPROVED_EXCEPTIONS` was read by no row at all (m-3); each pair is
now checked against the witness's vocabulary and the run says, per pair, why it
moves no denominator — and would say `WARNING` if one ever did.
## 9. Reproduction
```
uv run python tools/okf_accounting_gate.py # 3 m 51 s, exit 1
uv run python tools/okf_gate_mutants.py # 34 of 34 killed, exit 0
uv run pytest tests/test_accounting_gate.py tests/test_content_accounting.py -q
uv run python tools/okf_witness.py tests/fixtures/accounting/corpus
python3 tests/fixtures/accounting/make_accounting_fixtures.py
```

BIN
tests/fixtures/accounting/corpus/liste-og-bilde.odt vendored
View file

Binary file not shown.

BIN
tests/fixtures/accounting/corpus/notater-og-skjult.pptx vendored
View file

Binary file not shown.

BIN
tests/fixtures/accounting/corpus/skjult-ark-og-formel.xlsx vendored
View file

Binary file not shown.

38
tests/fixtures/accounting/make_accounting_fixtures.py vendored
View file

@ -120,7 +120,9 @@ def _slide(title: str, body: str, *, hidden: bool = False) -> str:
+ f"<p:txBody><a:p><a:r><a:t>{title}</a:t></a:r></a:p></p:txBody></p:sp>"
+ "<p:sp><p:nvSpPr><p:nvPr/></p:nvSpPr>"
+ f"<p:txBody><a:p><a:r><a:t>{body}</a:t></a:r></a:p></p:txBody></p:sp>"
+ "<p:graphicFrame><a:graphic><a:graphicData><a:tbl><a:tr>"
+ "<p:graphicFrame><a:graphic><a:graphicData "
+ 'uri="http://schemas.openxmlformats.org/drawingml/2006/table">'
+ "<a:tbl><a:tr>"
+ "<a:tc><a:txBody><a:p><a:r><a:t>Post</a:t></a:r></a:p></a:txBody></a:tc>"
+ "<a:tc><a:txBody><a:p><a:r><a:t>84.1</a:t></a:r></a:p></a:txBody></a:tc>"
+ "</a:tr></a:tbl></a:graphicData></a:graphic></p:graphicFrame>"
@ -141,11 +143,23 @@ _PPTX_PARTS: dict[str, str | bytes] = {
f'openxmlformats-officedocument.presentationml.slide+xml"/>'
for n in (1, 2)
)
+ '<Override PartName="/ppt/notesSlides/notesSlide1.xml" ContentType="application/vnd.'
+ 'openxmlformats-officedocument.presentationml.notesSlide+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/officeDocument" Target="ppt/presentation.xml"/></Relationships>',
"ppt/_rels/presentation.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/slide" Target="slides/slide1.xml"/>'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/slide" Target="slides/slide2.xml"/></Relationships>',
"ppt/slides/_rels/slide1.xml.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/notesSlide" Target="../notesSlides/notesSlide1.xml"/></Relationships>',
"ppt/presentation.xml": _XML
+ f"<p:presentation {_P}><p:sldIdLst>"
+ '<p:sldId id="256" r:id="rId1"/><p:sldId id="257" r:id="rId2"/>'
@ -171,6 +185,15 @@ _XLSX_PARTS: dict[str, str | bytes] = {
+ 'relationships+xml"/>'
+ '<Override PartName="/xl/workbook.xml" ContentType="application/vnd.openxmlformats-'
+ 'officedocument.spreadsheetml.sheet.main+xml"/>'
# Declaring the parts is not decoration. Without the sharedStrings
# override every `t="s"` cell converts to an EMPTY cell -- the fixture
# would have reported four cells lost that the build never lost.
+ '<Override PartName="/xl/worksheets/sheet1.xml" ContentType="application/vnd.'
+ 'openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
+ '<Override PartName="/xl/worksheets/sheet2.xml" ContentType="application/vnd.'
+ 'openxmlformats-officedocument.spreadsheetml.worksheet+xml"/>'
+ '<Override PartName="/xl/sharedStrings.xml" ContentType="application/vnd.'
+ 'openxmlformats-officedocument.spreadsheetml.sharedStrings+xml"/>'
+ "</Types>",
"_rels/.rels": _XML
+ '<Relationships xmlns="http://schemas.openxmlformats.org/package/2006/relationships">'
@ -187,7 +210,12 @@ _XLSX_PARTS: dict[str, str | bytes] = {
+ '<Relationship Id="rId1" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/worksheet" Target="worksheets/sheet1.xml"/>'
+ '<Relationship Id="rId2" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/worksheet" Target="worksheets/sheet2.xml"/></Relationships>',
+ 'relationships/worksheet" Target="worksheets/sheet2.xml"/>'
# The shared string table is reached through the WORKBOOK's relationship,
# not by its path: without this line every `t="s"` cell converts empty and
# the fixture reports four cells lost that the build never lost.
+ '<Relationship Id="rId3" Type="http://schemas.openxmlformats.org/officeDocument/2006/'
+ 'relationships/sharedStrings" Target="sharedStrings.xml"/></Relationships>',
"xl/sharedStrings.xml": _XML
+ f'<sst {_S} count="{len(_XLSX_STRINGS)}" uniqueCount="{len(_XLSX_STRINGS)}">'
+ "".join(f"<si><t>{value}</t></si>" for value in _XLSX_STRINGS)
@ -237,7 +265,8 @@ _ODT_PARTS: dict[str, str | bytes] = {
+ '<manifest:file-entry manifest:full-path="content.xml" manifest:media-type="text/xml"/>'
+ "</manifest:manifest>",
"content.xml": _XML
+ f"<office:document-content {_ODT_NS}><office:body><office:text>"
+ f'<office:document-content {_ODT_NS} office:version="1.3">'
+ "<office:body><office:text>"
+ '<text:h text:outline-level="1">Drift av gangbruer</text:h>'
+ "<text:p>Gangbruer inspiseres hvert aar.</text:p>"
+ "<text:list><text:list-item><text:p>Rekkverk</text:p></text:list-item>"
@ -252,7 +281,8 @@ _ODT_PARTS: dict[str, str | bytes] = {
+ "</table:table-row></table:table>"
+ "</office:text></office:body></office:document-content>",
"styles.xml": _XML
+ f"<office:document-styles {_ODT_NS}><office:master-styles>"
+ f'<office:document-styles {_ODT_NS} office:version="1.3">'
+ "<office:styles/><office:master-styles>"
+ '<style:master-page style:name="Standard">'
+ "<style:header><text:p>Intern arbeidsversjon</text:p></style:header>"
+ "<style:footer><text:p>Vegdirektoratet</text:p></style:footer>"

23
tests/fixtures/accounting/witness/sts-label-in-title.twin.json vendored Normal file
View file

@ -0,0 +1,23 @@
{
"productInfo": {"meta": {"productTitle": "Label inside title"}},
"standardContent": {
"id": null, "tag": "#document", "a": {},
"c": [
{"e": 1, "t": null, "x": {"id": null, "tag": "standard", "a": {}, "c": [
{"e": 1, "t": null, "x": {"id": null, "tag": "body", "a": {}, "c": [
{"e": 1, "t": null, "x": {"id": null, "tag": "sec", "a": {}, "c": [
{"e": 1, "t": null, "x": {"id": null, "tag": "title", "a": {}, "c": [
{"e": 1, "t": null, "x": {"id": null, "tag": "label", "a": {}, "c": [
{"e": 2, "t": "84.1", "x": null}
]}},
{"e": 2, "t": "Toleranser", "x": null}
]}},
{"e": 1, "t": null, "x": {"id": null, "tag": "p", "a": {}, "c": [
{"e": 2, "t": "Toleranseklasse er gitt i tabell 84-2.", "x": null}
]}}
]}}
]}}
]}}
]
}
}

170
tests/test_accounting_gate.py
View file

@ -614,10 +614,16 @@ def test_an_image_booked_carried_without_its_bytes_is_unverified(tmp_path: Path)
def test_a_negative_booking_is_never_clean(tmp_path: Path) -> None:
declared = _all_carried()
declared["documents"][0]["fates"]["heading"] = {"carried": 25, "rejected": {"x": -15}}
declared["documents"][0]["fates"]["heading"] = {
"carried": 25,
"rejected": {"extractor_unknown": -15},
}
unit = gate.account(_inventory(), _build(accounting=declared), _corpus(tmp_path))[0]
assert unit.invalid >= 1
assert not unit.clean
# 25 + (-15) = 10 booked against a source holding 2, so eight are booked
# twice. Absorbing the sign would read 40 and report thirty-eight.
assert unit.double == 8
def test_a_document_declared_persisted_that_is_not_in_the_bundle_is_never_clean(
@ -686,6 +692,29 @@ def test_an_asset_with_the_right_name_and_the_wrong_bytes_is_not_carried(tmp_pat
assert gate.account(_inventory(), build, corpus)[1].unaccounted == 1
def test_an_asset_under_a_reduced_name_still_proves_the_carry(tmp_path: Path) -> None:
"""The build lowercases and folds the source's basename and sniffs the
suffix from the bytes. Measured on R761, a judge checking the FULL name
called 50 of 50 carried images missing -- the judge's defect, not the
build's, so the content address is the check and the readable tail is not.
"""
corpus = _corpus(tmp_path)
source = corpus / "graphics" / "x.png"
digest = gate._sha256(source)
reduced = {f"{digest[:12]}-25-0143-tabeller-r761-r762.jpeg": digest}
build = _build(accounting=_declared(fate="carried"), assets=reduced)
assert gate.account(_inventory(), build, corpus)[1].clean
def test_an_asset_holding_the_bytes_under_a_foreign_address_is_not_a_carry(
tmp_path: Path,
) -> None:
corpus = _corpus(tmp_path)
source = corpus / "graphics" / "x.png"
build = _build(accounting=_declared(fate="carried"), assets={"x.png": gate._sha256(source)})
assert gate.account(_inventory(), build, corpus)[1].unaccounted == 1
def test_the_cheat_that_books_everything_carried_makes_row3_red(tmp_path: Path) -> None:
"""The review's `MODE=carried`: a report that changes not one byte of the
bundle and books every element as carried."""
@ -747,13 +776,146 @@ def test_row5_is_red_when_a_witness_is_missing() -> None:
def test_row6_without_its_source_is_red_locally_and_skipped_in_ci(tmp_path: Path) -> None:
missing = tmp_path / "absent"
local = gate.row6(missing, ci=False)
ci = gate.row6(missing, ci=True)
local = gate.row6(missing, None, ci=False)
ci = gate.row6(missing, None, ci=True)
assert (local.status, local.fails) == (gate.RED, True)
assert (ci.status, ci.fails) == (gate.SKIPPED, False)
assert "source missing" in ci.reason
def test_a_skipped_row_never_leaves_the_verdict_unqualified() -> None:
"""m-2: `CI=1` with a missing source printed `GATE GREEN` with nothing
beside it, so the one line most readers stop at said the corpus passed."""
skipped = gate.Row(6, "real corpora", 0, 0, gate.SKIPPED, "not measured, source missing: x")
rendered = gate.render([skipped])
assert "GATE GREEN (row 6 not run: not measured, source missing: x)" in rendered
def test_a_corpus_refused_whole_under_the_default_gate_is_red(tmp_path: Path) -> None:
"""Row 6 was GREEN with R761 100 % rejected: every element booked as a
coded rejection satisfies u = 0 and d = 0. The build order asked for an
honest red there, so the row says this on its own."""
inventory = {
"documents": {
"a.md": {
"suffix": ".md",
"elements": {"heading": 2},
"texts": {"heading": [["A"], ["B"]]},
"images": [],
}
},
"files": {},
}
declared = {
"accounting_version": 1,
"documents": [
{
"source_file": "a.md",
"status": "rejected",
"code": "fail_secure",
"inventory": {"heading": 2},
"fates": {"heading": {"rejected": {"fail_secure": 2}}},
}
],
"files": [],
}
build = _build(accounting=declared, sources=set(), exit_code=1)
units = gate.account(inventory, build, tmp_path)
assert all(u.clean for u in units), "every element has a coded fate"
assert gate.refused_whole(inventory["documents"], build) is not None
def test_the_two_real_corpora_are_named_and_the_second_is_not_r761() -> None:
"""R761 holds 0 `fig`, 0 formulas and 0 references, so the gate's only
real corpus could not see the hole in the STS role map."""
corpora = gate.real_corpora(Path("/r761"), Path("/n200.json"))
assert [c.label.split()[0] for c in corpora] == ["R761", "N200"]
def test_a_unit_clean_in_only_one_of_the_two_builds_is_not_clean() -> None:
"""M13: the two gates see different things, so either build could cover
for the other."""
clean = gate.Unit("a", "document", 0, 0)
dirty = gate.Unit("a", "document", 1, 0)
assert gate.clean_in_every_run([[clean], [clean]]) == 1
assert gate.clean_in_every_run([[clean], [dirty]]) == 0
def test_the_gate_exits_one_when_a_row_is_red(monkeypatch: pytest.MonkeyPatch) -> None:
"""M14: nothing checked the real command's exit code, so `main` could
return 0 over a red table and no test would notice."""
pytest.importorskip("pdfplumber")
pytest.importorskip("pypandoc")
monkeypatch.delenv("CI", raising=False)
code = gate.main(["--r761", "/no/such/corpus", "--n200", "/no/such/file.json"])
assert code == 1
def test_a_file_with_no_declaration_is_unaccounted_when_conservation_failed(
tmp_path: Path,
) -> None:
"""M17: without the door, a file that is not merged counts as a coded
rejection ONLY because the build's own conservation identity held. A run
that exited non-zero has not earned that inference."""
corpus = _corpus(tmp_path)
failed = _build(exit_code=1, log="K1b FAILED")
assert gate.account(_inventory(), failed, corpus)[1].unaccounted == 1
assert gate.account(_inventory(), _build(), corpus)[1].clean
def test_a_merged_file_declared_carried_without_its_bytes_is_still_a_false_claim(
tmp_path: Path,
) -> None:
"""M05: with `fates` summing to one anyway, the false claim is the only
thing that can see it."""
corpus = _corpus(tmp_path)
build = _build(accounting=_declared(fate="carried"), sources={"a.md", "graphics/x.png"})
unit = gate.account(_inventory(), build, corpus)[1]
assert (unit.unaccounted, unit.clean) == (1, False)
def test_a_rejection_code_inside_an_element_fate_must_also_be_known(tmp_path: Path) -> None:
"""M24: the document's own `code` was checked and the per-element ones
were not, so a report could name any reason it liked for an element."""
declared = _all_carried()
declared["documents"][0]["fates"]["heading"] = {"rejected": {"because_i_said_so": 2}}
unit = gate.account(_inventory(), _build(accounting=declared), _corpus(tmp_path))[0]
assert unit.invalid >= 1
assert "because_i_said_so" in "; ".join(unit.notes)
def test_the_json_twin_is_read_with_the_json_role_map(tmp_path: Path) -> None:
"""M-2, measured on R761 2026-09-18: the XML delivery places a section's
label at `sec/label` (7 714, and 0 inside a title); the JSON delivery puts
2 760 of them inside the title. Read with the XML map, the twin loses
every one of those."""
twin = gate.FIXTURES / "witness" / "sts-label-in-title.twin.json"
assert witness.count_sts_json(twin.read_bytes()).counts["section_label"] == 1
assert witness._sts_role_xml("label", "title", "sec") is None
assert witness._sts_role_json("label", "title", "sec") == "section_label"
def test_an_approved_exception_is_read_and_says_what_it_does() -> None:
"""m-3: `APPROVED_EXCEPTIONS` was read by no row, so approving one changed
nothing and the list could have said anything."""
for suffix, element in gate.APPROVED_EXCEPTIONS:
assert "no denominator moves" in gate.exception_effect(suffix, element)
assert "WARNING" in gate.exception_effect(".pdf", "page")
assert "names nothing" in gate.exception_effect(".doc", "heading")
rendered = gate.render([])
assert "no denominator moves" in rendered
def test_the_gate_states_its_own_limits() -> None:
rendered = gate.render([])
assert "what this gate cannot check" in rendered
assert len(gate.LIMITS) >= 5
def test_every_witnessed_type_has_a_vocabulary() -> None:
assert set(gate.FORMAT_VOCABULARY) == set(witness.WITNESSED_SUFFIXES)
def test_the_proposed_exceptions_are_not_applied() -> None:
assert "NOT APPROVED" in gate.render([])
assert not {suffix for suffix, _ in gate.APPROVED_EXCEPTIONS} & {
@ -789,7 +951,7 @@ def test_bad_usage_exits_two() -> None:
def real_rows() -> list[gate.Row]:
pytest.importorskip("pdfplumber")
pytest.importorskip("pypandoc")
return gate.evaluate(r761=None, ci=True, consume=False)
return gate.evaluate(r761=None, n200=None, ci=True, consume=False)
def _cheating_report(inventory: dict[str, Any], mode: str) -> dict[str, Any]:

109
tests/test_content_accounting.py
View file

@ -77,7 +77,7 @@ def test_the_log_separates_carried_files_from_rejections(tmp_path: Path) -> None
code, bundle, _ = _build(CORPUS, tmp_path)
log = (bundle / "log.md").read_text(encoding="utf-8")
assert code == 0
assert "merged + files carried through a document + coded rejections = 13 + 2 + 0 = 15" in log
assert "merged + files carried through a document + coded rejections = 20 + 2 + 0 = 22" in log
assert "`extractor_unknown`" not in log
@ -123,12 +123,75 @@ def _witness(path: Path) -> dict[str, Any]:
return data
#: What the witness counts and this package does not, per file type. Added
#: 2026-09-18 when the gate's witnesses were widened: rows 2 and 3 of the gate
#: are RED on exactly these, and that is the finding rather than a regression.
#: Closing one here turns this test red, which is the point -- the list is the
#: standing statement of what a bundle built by this package leaves behind.
NOT_IN_THIS_PACKAGES_VOCABULARY = {
"comment",
"endnote",
"header_footer",
"text_box",
"note",
"hidden_slide",
"formula",
"hidden_sheet",
"annotation",
"citation",
"math",
"figure",
"figure_caption",
}
#: Shared element names on which the two now DISAGREE, with the cause of each.
#: Three are reclassifications -- the witness gives the element its own role
#: and this package still calls it the ordinary one -- and the fourth is a
#: double count this package makes. Measured 2026-09-18.
VOCABULARY_SHIFTS: dict[tuple[str, str], tuple[int, int, str]] = {
("notater-og-skjult.pptx", "slide"): (2, 1, "a hidden slide counts as an ordinary slide"),
("skjult-ark-og-formel.xlsx", "sheet"): (2, 1, "a hidden sheet counts as an ordinary sheet"),
("liste-og-bilde.odt", "paragraph"): (5, 4, "an annotation counts as prose"),
(
"topptekst-og-kommentar.docx",
"paragraph",
): (
5,
3,
"a text box's paragraph is counted TWICE: inside the box, and again in the "
"paragraph that carries the box",
),
}
@pytest.mark.parametrize("name", sorted(_witness(FIXTURES / "inventory.json")["documents"]))
def test_the_inventory_equals_the_witness(name: str) -> None:
def test_the_inventory_equals_the_witness_on_every_shared_element(name: str) -> None:
"""The coupling is unchanged where the two vocabularies meet: one number
off on a shared element is a red test unless it is in the table above,
with its cause written down."""
pytest.importorskip("pdfplumber")
want = _witness(FIXTURES / "inventory.json")["documents"][name]["elements"]
got = accounting.inventory(CORPUS, CORPUS / name)
assert got.counts() == want
got = accounting.inventory(CORPUS, CORPUS / name).counts()
shared = {k: v for k, v in want.items() if k not in NOT_IN_THIS_PACKAGES_VOCABULARY}
differences = {k: (got[k], v) for k, v in shared.items() if got.get(k) != v}
expected = {
element: (mine, theirs)
for (document, element), (mine, theirs, _) in VOCABULARY_SHIFTS.items()
if document == name
}
assert differences == expected
@pytest.mark.parametrize("name", sorted(_witness(FIXTURES / "inventory.json")["documents"]))
def test_the_classes_this_package_does_not_count_are_named(name: str) -> None:
"""A known-negative: every element the witness counts and this package
does not is on the list above, by name. Nothing is missing quietly."""
pytest.importorskip("pdfplumber")
want = _witness(FIXTURES / "inventory.json")["documents"][name]["elements"]
got = accounting.inventory(CORPUS, CORPUS / name).counts()
assert set(want) - set(got) <= NOT_IN_THIS_PACKAGES_VOCABULARY
assert not set(got) - set(want), "this package counts something no witness does"
def test_the_inventory_resolves_the_files_a_document_points_at() -> None:
@ -144,15 +207,43 @@ def test_the_inventory_of_an_unread_type_is_empty() -> None:
# --- the door ----------------------------------------------------------------
def test_the_door_books_every_element_of_the_fixture_corpus_once(tmp_path: Path) -> None:
#: The elements this package loses from the fixture corpus, in its OWN
#: vocabulary, measured 2026-09-18 on the widened corpus. Each is a real loss
#: with a named cause, and the door exits 1 because of them -- which is the
#: behaviour the flag was built for, exercised for the first time by fixtures
#: that actually carry the constructs.
KNOWN_LOSSES = {
("liste-og-bilde.odt", "image"): (0, 1),
("liste-og-bilde.odt", "paragraph"): (4, 5),
("skjult-ark-og-formel.xlsx", "image"): (0, 1),
("topptekst-og-kommentar.docx", "footnote"): (0, 1),
("topptekst-og-kommentar.docx", "paragraph"): (3, 5),
}
def test_the_door_books_every_element_of_the_fixture_corpus_but_the_known_losses(
tmp_path: Path,
) -> None:
pytest.importorskip("pdfplumber")
path = tmp_path / "accounting.json"
code, _, err = _build(CORPUS, tmp_path, "--accounting", str(path))
assert code == 0, err
data = json.loads(path.read_text(encoding="utf-8"))
assert data["accounting_version"] == 1
assert data["unaccounted"] == 0
assert data["double_booked"] == 0
assert data["unaccounted"] == 6, err
assert code == 1, "a run that loses content does not exit zero"
found = {}
for document in data["documents"]:
for kind, number in document["inventory"].items():
fate = document.get("fates", {}).get(kind, {})
booked = (
fate.get("carried", 0)
+ fate.get("pointer", 0)
+ sum(fate.get("rejected", {}).values())
)
if booked != number:
found[(document["source_file"], kind)] = (booked, number)
assert found == KNOWN_LOSSES
files = {entry["source_file"]: entry["fate"] for entry in data["files"]}
assert files == {"graphics/figur-84-1.png": "carried", "graphics/tabell-84-2.png": "carried"}
web = next(d for d in data["documents"] if d["source_file"] == "prosess-84-web.html")
@ -163,8 +254,8 @@ def test_the_door_writes_the_accounting_into_the_log(tmp_path: Path) -> None:
pytest.importorskip("pdfplumber")
_, bundle, _ = _build(CORPUS, tmp_path, "--accounting", str(tmp_path / "a.json"))
log = (bundle / "log.md").read_text(encoding="utf-8")
assert "* **Accounting**: 13 document(s) and 2 other file(s);" in log
assert "0 unaccounted, 0 double-booked." in log
assert "* **Accounting**: 20 document(s) and 2 other file(s);" in log
assert "6 unaccounted, 0 double-booked." in log
def test_a_rejected_document_is_logged_with_what_its_source_held(tmp_path: Path) -> None:

331
tools/okf_accounting_gate.py
View file

@ -58,11 +58,12 @@ import io
import json
import os
import re
import shutil
import sys
import tempfile
import warnings
import zipfile
from collections.abc import Iterable, Mapping
from collections.abc import Iterable, Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path, PurePosixPath
from typing import Any
@ -85,6 +86,7 @@ PDF_FIXTURE = CORPUS / "prosess-84-tabell.pdf"
README = REPO / "README.md"
R761_DEFAULT = Path.home() / "repos" / "vegnormal-okf" / "data" / "raw" / "860019"
N200_DEFAULT = R761_DEFAULT.parent / "N200-2024-860015.json"
R761_ZIP = "14ce59dc-2150-480b-b661-6ea605fe3b24.zip"
R761_JSON = "R761-2025-860019.json"
R761_PDF = "R761-prosesskoden-2025.pdf"
@ -130,6 +132,60 @@ PROPOSED_EXCEPTIONS: tuple[dict[str, str], ...] = (
)
#: Limits of the instrument itself, printed beside the verdict. A gate that
#: only reports the build's gaps and none of its own invites the reader to
#: take a green row for a guarantee.
LIMITS: tuple[str, ...] = (
"a short element (a label, a one-word title) often stands elsewhere in the "
"same document, so finding it proves it is present and not that THIS one is",
"two pointed files with identical bytes are one content-addressed asset, so "
"one of them losing its pointer is invisible here (m-5)",
"an image embedded in a binary container has no source file to hash, so a "
"carry of it is counted as one the gate cannot check",
"absence is never verified: an element booked as REJECTED is not looked for "
"in the bundle, only one booked as carried",
"the witness is a second implementation of the same definitions, so a "
"definition that is wrong for a format is wrong on both sides at once",
)
def exception_effect(suffix: str, element: str) -> str:
"""What an approved exception actually does to the numbers.
m-3: `APPROVED_EXCEPTIONS` was read by no row at all, so approving one
changed nothing and the list could say anything. It still moves no
denominator -- and now the run SAYS why, per pair, from the witness's own
vocabulary rather than from the sentence next to the list.
"""
vocabulary = FORMAT_VOCABULARY.get(suffix)
if vocabulary is None:
return f"no witness reads {suffix}, so the pair names nothing"
if element in vocabulary:
return (
f"WARNING: the witness DOES count `{element}` for {suffix}, so this "
"approval would lower a denominator"
)
return f"the witness counts no `{element}` for {suffix}: no denominator moves"
#: Every element name a witness can produce, per file type.
FORMAT_VOCABULARY: dict[str, tuple[str, ...]] = {
".csv": witness.CSV,
".docx": witness.DOCX,
".htm": witness.HTML,
".html": witness.HTML,
".json": witness.JSON,
".md": witness.MARKDOWN,
".odt": witness.ODT,
".pdf": witness.PDF,
".pptx": witness.PPTX,
".rtf": witness.RTF,
".txt": witness.TEXT,
".xlsx": witness.XLSX,
".xml": witness.STS_ROLES,
}
@dataclass
class Row:
number: int
@ -305,15 +361,26 @@ def _sha12(path: Path) -> str:
def asset_holds(build: Build, source: Path) -> bool:
"""Did the run carry THESE bytes, under the name the layout gives them?
"""Did the run carry THESE bytes, placed under their own content address?
Both halves are load-bearing. The name alone was the check until an
independent review wrote a zero-byte file called `<sha12>-x.png` and the
gate read it as a carry (m-1); the bytes alone would credit a file the
layout would have named something else.
Both halves are load-bearing and neither is the build's naming rule. The
NAME alone was the check until an independent review wrote a zero-byte
file called `<sha12>-x.png` and the gate read it as a carry (m-1). The
BYTES alone would credit an asset standing under any name at all, which
is the property the layout exists to guarantee.
What the gate deliberately does NOT reproduce is the readable tail: the
build lowercases the source's basename, folds its separator runs, cuts it
to a maximum and takes the suffix from the BYTES rather than from the
name. Re-implementing that here would make the judge agree with the judged
by construction -- and it would be wrong: measured 2026-09-18 on R761,
whose own hrefs carry spaces, capitals and parentheses, a judge checking
the full name reported 50 of 50 carried images as missing.
"""
digest = _sha256(source)
return build.assets.get(f"{digest[:12]}-{source.name}") == digest
return any(
found == digest and name.startswith(digest[:12]) for name, found in build.assets.items()
)
# --- the judge's own reading of the bundle ------------------------------------
@ -526,7 +593,10 @@ def _document_unit(
elif booked > have:
double += booked - have
notes.append(f"{element}: {booked} booked, source has {have}")
want = max(carried, 0) + max(pointer, 0)
# Capped at what the source holds: a booking ABOVE that is already
# reported as double, and counting the excess as "claimed and not
# found" would report one defect under two names.
want = min(max(carried, 0) + max(pointer, 0), have) if have else 0
if want == 0:
continue
if not persisted:
@ -858,60 +928,156 @@ def witness_pairs(r761: Path | None) -> tuple[list[tuple[str, list[str]]], list[
return pairs, notes
#: The two builds row 6 runs: the default gate is what a user gets; `none`
#: persists the document, which is the only way its pictures are carried and
#: the double booking of the files beside it becomes visible.
#: The two builds row 6 runs per corpus: the default gate is what a user gets;
#: `none` persists the document, which is the only way its pictures are
#: carried and the double booking of the files beside it becomes visible.
R761_GATES: tuple[str | None, ...] = (None, "none")
def row6(r761: Path | None, ci: bool) -> Row:
"""R761 through two builds; a unit is clean only if it is clean in both."""
name = "real corpus: R761 Prosesskoden:2025"
if r761 is None or not r761.is_dir():
@dataclass(frozen=True)
class RealCorpus:
"""A corpus of real documents, read only, outside this repository."""
label: str
kind: str # "zip" or "file"
path: Path
@property
def available(self) -> bool:
return self.path.exists()
def real_corpora(r761: Path | None, n200: Path | None) -> list[RealCorpus]:
"""R761 is the gate's original corpus; N200 was added 2026-09-18 because
R761 holds NONE of the STS classes the role map was missing -- 0 `fig`, 0
formulas, 0 references -- so the only real corpus could not have found the
hole. N200 carries 194 citations, 49 figures and 135 footnotes."""
corpora = []
if r761 is not None:
corpora.append(RealCorpus("R761 Prosesskoden:2025", "zip", r761 / R761_ZIP))
if n200 is not None:
corpora.append(RealCorpus("N200 Vegbygging:2024", "file", n200))
return corpora
def _corpus_inbox(corpus: RealCorpus, root: Path) -> Path:
inbox = root / "inbox"
inbox.mkdir(parents=True)
if corpus.kind == "zip":
with zipfile.ZipFile(corpus.path) as archive:
archive.extractall(inbox)
else:
shutil.copy2(corpus.path, inbox / corpus.path.name)
return inbox
def clean_in_every_run(runs: Sequence[Sequence[Unit]]) -> int:
"""Units clean in EVERY build, never in any of them.
The two gates see different things -- the default one refuses, `none`
persists and carries the pictures -- so a unit that is clean in one and
dirty in the other has a fate the run does not agree on, and calling that
clean would let either build cover for the other.
"""
return sum(1 for parts in zip(*runs) if all(unit.clean for unit in parts))
def refused_whole(documents: Mapping[str, Any], build: Build) -> str | None:
"""Did this run persist NOTHING of a corpus that holds documents?
Booking every element of a refused document as a coded rejection gives
u = 0 and d = 0, so the numbers are clean and the bundle is empty. Row 6
read GREEN on R761 with 110 of 110 elements rejected and `okf build`
exiting 1 unseen. The build order asked for an honest red there, so the
row asks this question on its own.
"""
if not documents:
return None
persisted = sum(1 for name in documents if name in build.source_files)
if persisted:
return None
elements = sum(sum(entry["elements"].values()) for entry in documents.values())
codes = sorted(
{str(d.get("code")) for d in (build.accounting or {}).get("documents", []) if d.get("code")}
)
return (
f"the default gate persisted 0 of {len(documents)} document(s) "
f"({', '.join(codes) or 'no code declared'}), {elements} element(s) rejected whole; "
f"okf build exited {build.exit_code}"
)
def row6(r761: Path | None, n200: Path | None, ci: bool) -> Row:
"""Every real corpus through two builds; a unit is clean only in both."""
name = "real corpora"
corpora = [c for c in real_corpora(r761, n200) if c.available]
missing = [c for c in real_corpora(r761, n200) if not c.available]
if not corpora:
# SKIPPED is only free when there is nothing to measure. A source that
# EXISTS and was not measured is a row that did not run, and a row
# that did not run is not a row that passed.
status = SKIPPED if ci else RED
return Row(6, name, 0, 0, status, f"not measured, source missing: {r761}")
names = ", ".join(str(c.path) for c in missing) or "no corpus configured"
return Row(6, name, 0, 0, status, f"not measured, source missing: {names}")
door = door_available()
runs: list[tuple[str, Build, list[Unit]]] = []
with zipfile.ZipFile(r761 / R761_ZIP) as archive, tempfile.TemporaryDirectory() as tmp:
inbox = Path(tmp) / "inbox"
archive.extractall(inbox)
inventory = witness.witness_inbox(inbox)
json_counts = dict(witness.count_sts_json((r761 / R761_JSON).read_bytes()).counts)
for index, gate_name in enumerate(R761_GATES):
build = run_build(inbox, Path(tmp) / f"work{index}", door=door, gate=gate_name)
runs.append((gate_name or "default", build, account(inventory, build, inbox)))
files = inventory["files"]
pointed = sum(1 for f in files.values() if f["pointed_at_by"])
documents = inventory["documents"]
details = [
f"witness (json): {json.dumps(json_counts, sort_keys=True)}",
f"zip: {len(documents)} document(s), {len(files)} other files "
f"({pointed} pointed at, {len(files) - pointed} not)",
]
reasons = []
for label, build, units in runs:
persisted = sum(1 for d in documents if d in build.source_files)
u_total = sum(u.unaccounted for u in units)
d_total = sum(u.double for u in units)
reasons.append(f"gate {label}: u = {u_total}, d = {d_total}")
details: list[str] = [f"{c.path} not measured: source missing" for c in missing]
reasons: list[str] = []
clean = total = 0
refused: list[str] = []
for corpus in corpora:
with tempfile.TemporaryDirectory() as tmp:
inbox = _corpus_inbox(corpus, Path(tmp))
inventory = witness.witness_inbox(inbox)
runs: list[tuple[str, Build, list[Unit]]] = []
for index, gate_name in enumerate(R761_GATES):
build = run_build(inbox, Path(tmp) / f"work{index}", door=door, gate=gate_name)
runs.append((gate_name or "default", build, account(inventory, build, inbox)))
documents = inventory["documents"]
files = inventory["files"]
pointed = sum(1 for f in files.values() if f["pointed_at_by"])
elements = sum(sum(e["elements"].values()) for e in documents.values())
details.append(
f"gate {label}: exit {build.exit_code}, {persisted} of {len(documents)} "
f"document(s) persisted, {len(build.asset_prefixes)} asset file(s)"
f"{corpus.label}: {len(documents)} document(s), {elements} element(s), "
f"{len(files)} other file(s) ({pointed} pointed at, {len(files) - pointed} not)"
)
for unit in units:
if unit.kind == "document":
details.append(
f" {unit.name}: u={unit.unaccounted} d={unit.double} ({'; '.join(unit.notes)})"
)
doubled = [u for u in units if u.kind == "file" and u.double]
if doubled:
details.append(
f" {len(doubled)} file(s) carried through the document AND rejected, "
f"e.g. {doubled[0].name}"
for label, build, units in runs:
persisted = sum(1 for d in documents if d in build.source_files)
u_total = sum(u.unaccounted for u in units)
d_total = sum(u.double for u in units)
unverified = sum(u.unverified for u in units)
reasons.append(
f"{corpus.label} gate {label}: u = {u_total}, d = {d_total}, "
f"{unverified} claimed and not found"
)
clean = sum(1 for parts in zip(*(units for _, _, units in runs)) if all(u.clean for u in parts))
total = len(runs[0][2])
return _row(6, name, clean, total, f"{'; '.join(reasons)} over {total} units", details)
details.append(
f" gate {label}: exit {build.exit_code}, {persisted} of {len(documents)} "
f"document(s) persisted, {len(build.assets)} asset file(s); {_tally(units)}"
)
for unit in units:
if unit.kind == "document" and not unit.clean:
details.append(
f" {unit.name}: u={unit.unaccounted} d={unit.double} "
f"unverified={unit.unverified} invalid={unit.invalid} "
f"({'; '.join(unit.notes)})"
)
doubled = [u for u in units if u.kind == "file" and u.double]
if doubled:
details.append(
f" {len(doubled)} file(s) carried through the document AND rejected, "
f"e.g. {doubled[0].name}"
)
if label == "default":
whole = refused_whole(documents, build)
if whole is not None:
refused.append(f"{corpus.label}: {whole}")
clean += clean_in_every_run([units for _, _, units in runs])
total += len(runs[0][2])
details.extend(refused)
status = GREEN if total > 0 and clean == total and not refused else RED
reason = "; ".join(reasons) + f" over {total} unit(s)"
if refused:
reason = "a real corpus is refused whole under the default gate; " + reason
return Row(6, name, clean, total, status, reason, details)
def row7(workdir: Path) -> Row:
@ -944,7 +1110,7 @@ def row7(workdir: Path) -> Row:
# --- the run -----------------------------------------------------------------
def evaluate(*, r761: Path | None, ci: bool, consume: bool) -> list[Row]:
def evaluate(*, r761: Path | None, n200: Path | None, ci: bool, consume: bool) -> list[Row]:
table = readme_types()
inventory = load_inventory(INVENTORY)
rejected_inventory = load_inventory(REJECTED_INVENTORY)
@ -959,7 +1125,7 @@ def evaluate(*, r761: Path | None, ci: bool, consume: bool) -> list[Row]:
rejected_build = run_build(REJECTED, Path(tmp) / "rejected", door=door)
rows.append(row4(rejected_inventory, rejected_build))
rows.append(row5(*witness_pairs(r761)))
rows.append(row6(r761, ci))
rows.append(row6(r761, n200, ci))
if consume:
rows.append(row7(work))
return rows
@ -979,7 +1145,10 @@ def render(rows: list[Row]) -> str:
)
lines.append(f"exceptions approved ({APPROVED_ON}), and none moves a denominator:")
for suffix, element in sorted(APPROVED_EXCEPTIONS) or [("(none)", "")]:
lines.append(f" - {suffix} {element}")
lines.append(f" - {suffix} {element}: {exception_effect(suffix, element)}")
lines += ["", "what this gate cannot check, whatever the rows say:"] + [
f" - {limit}" for limit in LIMITS
]
lines += [
"",
"not counted by any witness -- what no row here can see (per file type):",
@ -988,11 +1157,17 @@ def render(rows: list[Row]) -> str:
for item in witness.NOT_COUNTED[suffix]:
lines.append(f" - {suffix}: {item}")
failing = [str(r.number) for r in rows if r.fails]
skipped = [r for r in rows if r.status == SKIPPED]
lines.append("")
lines.append(
f"GATE {'RED' if failing else 'GREEN'}"
+ (f": rows {', '.join(failing)}" if failing else "")
)
verdict = f"GATE {'RED' if failing else 'GREEN'}"
if failing:
verdict += f": rows {', '.join(failing)}"
# A row that did not run is not a row that passed, and the one line most
# readers stop at is this one: `GATE GREEN` with a silent SKIPPED behind
# it is the shape the review reproduced with `CI=1` and a missing source.
for row in skipped:
verdict += f" (row {row.number} not run: {row.reason})"
lines.append(verdict)
return "\n".join(lines) + "\n"
@ -1005,14 +1180,29 @@ def main(argv: list[str] | None = None) -> int:
default=R761_DEFAULT,
help="directory holding the R761 zip, JSON and PDF (read only)",
)
parser.add_argument(
"--n200",
type=Path,
default=N200_DEFAULT,
help="the N200 JSON delivery, the second real corpus (read only)",
)
parser.add_argument(
"--consume", action="store_true", help="also run row 7 (diagnostic, never fails)"
)
args = parser.parse_args(argv)
try:
rows = evaluate(r761=args.r761, ci=bool(os.environ.get("CI")), consume=args.consume)
except (OSError, ValueError, KeyError) as exc:
print(f"okf-accounting-gate: did not run: {exc}", file=sys.stderr)
rows = evaluate(
r761=args.r761,
n200=args.n200,
ci=bool(os.environ.get("CI")),
consume=args.consume,
)
# Broad on purpose: a gate that dies with a traceback exits 1, which is
# the code it uses for RED, so a reader cannot tell a finding from a
# crash. `ET.ParseError`, `BadZipFile` and `CalledProcessError` all
# reached that path (m-4).
except Exception as exc:
print(f"okf-accounting-gate: did not run: {type(exc).__name__}: {exc}", file=sys.stderr)
return 2
if args.json:
payload = {
@ -1026,7 +1216,18 @@ def main(argv: list[str] | None = None) -> int:
print(json.dumps(payload, indent=2, ensure_ascii=False))
else:
print(render(rows), end="")
return 1 if any(r.fails for r in rows) else 0
if any(r.fails for r in rows):
return 1
# A row skipped while its source is on this machine did not run, and a
# zero here would report that as a pass.
for row in rows:
if row.status == SKIPPED and any(c.available for c in real_corpora(args.r761, args.n200)):
print(
f"okf-accounting-gate: row {row.number} was skipped while its source exists",
file=sys.stderr,
)
return 1
return 0
if __name__ == "__main__":

318
tools/okf_gate_mutants.py Normal file
View file

@ -0,0 +1,318 @@
"""Mutation run over the content-accounting gate and its witness.
The gate judges `okf build`; this asks what judges the gate. Each mutant is ONE
textual edit that makes the instrument weaker in a way a reader would call a
defect, and the question is whether the suite goes red. A mutant that survives
names a check nothing holds.
The twenty-six mutants are an independent review's, ported to the code as it
stands rather than re-invented: at `0b00de4` twelve of them survived the
forty-two tests, among them `main` always returning 0 and six of the seven
witness mutants -- which had no fixture that could exercise the element they
removed.
python3 tools/okf_gate_mutants.py
Runs on a COPY of the tree in a temporary directory: the working tree is never
edited, so an interrupted run cannot leave a mutant behind. A pattern that does
not match the expected number of times is reported as an ERROR and the run
exits 2 -- a mutant that could not be applied was never measured, and counting
it as killed is the same mistake as reading an empty search as an absence.
"""
from __future__ import annotations
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path
REPO = Path(__file__).resolve().parent.parent
GATE = "tools/okf_accounting_gate.py"
WITNESS = "tools/okf_witness.py"
SUITE = "tests/test_accounting_gate.py"
@dataclass(frozen=True)
class Mutant:
label: str
file: str
old: str
new: str
#: How many times `old` must occur. A role map written twice on purpose
#: (`_sts_role_xml`, `_sts_role_json`) has two, and the mutant edits the
#: first -- the XML witness -- so the JSON one stays honest and row 5 has
#: a chance to see the disagreement.
occurrences: int = 1
first_only: bool = False
MUTANTS: tuple[Mutant, ...] = (
Mutant(
"M01 m=0 counts as green ('not measured' -> GREEN)",
GATE,
"GREEN if m > 0 and k == m else RED",
"GREEN if k == m else RED",
),
Mutant(
"M02 row 6 never fails the gate",
GATE,
"self.status == RED and self.number <= 6",
"self.status == RED and self.number <= 5",
),
Mutant(
"M03 a missing corpus is SKIPPED locally too",
GATE,
"status = SKIPPED if ci else RED",
"status = SKIPPED",
),
Mutant(
"M04 double booking of elements never counted",
GATE,
"double += booked - have",
"double += 0",
),
Mutant(
"M05 'declared carried, bytes absent' never a false claim",
GATE,
'false_claim = fate == "carried" and not carried',
"false_claim = False",
),
Mutant(
"M06 an unpointed duplicate counts as carried",
GATE,
"carried = bool(pointed_by) and asset_holds(build, corpus / name)",
"carried = asset_holds(build, corpus / name)",
),
Mutant(
"M07 row 2 accepts ANY declared inventory",
GATE,
"if got != want:",
"if got is None:",
),
Mutant(
"M08 row 1 ignores a stale committed fasit",
GATE,
'if fresh["documents"].get(name) == entry:',
"if True:",
),
Mutant(
"M09 row 4 ignores the Images 'found' count",
GATE,
"elif int(found.group(2)) != declared_images:",
"elif False:",
),
Mutant(
"M10 row 4 accepts any element total in the log line",
GATE,
'rf"{re.escape(doc)}: {total} elements found in the source, 0 carried: "',
'rf"{re.escape(doc)}: \\d+ elements found in the source, 0 carried: "',
),
Mutant(
"M11 an unavailable witness is agreement",
GATE,
'return ["a witness is unavailable"]',
"return []",
),
Mutant(
"M12 row 5 counts every pair as agreeing",
GATE,
"good = sum(1 for _, problems in pairs if not problems)",
"good = len(pairs)",
),
Mutant(
"M13 row 6: clean in ANY of the two builds is enough",
GATE,
"if all(unit.clean for unit in parts)",
"if any(unit.clean for unit in parts)",
),
Mutant(
"M14 main always exits 0",
GATE,
" if any(r.fails for r in rows):\n return 1",
" if False:\n return 1",
),
Mutant(
"M15 T silently drops .rtf",
GATE,
'return sorted(set(re.findall(r"^\\| `(\\.[a-z0-9]+)` \\|", section, re.MULTILINE)))',
'return sorted(\n set(re.findall(r"^\\| `(\\.[a-z0-9]+)` \\|", section, re.MULTILINE)) - {".rtf"}\n )',
),
Mutant(
"M16 an element with no declared fate is ignored",
GATE,
"for element in sorted(set(elements) | set(fates)):",
"for element in sorted(set(fates)):",
),
Mutant(
"M17 conservation always 'held'",
GATE,
'return build.exit_code == 0 and "K1b FAILED" not in build.log',
"return True",
),
Mutant(
"M18 row 3 counts every unit as clean",
GATE,
"clean = sum(1 for u in units if u.clean)",
"clean = len(units)",
),
Mutant(
"M19 a negative rejection is absorbed into the total",
GATE,
"booked = carried + pointer + sum(rejected.values())",
"booked = carried + pointer + abs(sum(rejected.values()))",
),
# --- the judge's own reading of the bundle, added 2026-09-18 -------------
Mutant(
"M20 a booked carry is never looked for in the bundle",
GATE,
"found = sum(1 for pieces in sayable if all(finder(piece) for piece in pieces))",
"found = len(elements_pieces)",
),
Mutant(
"M21 an unverified booking is clean",
GATE,
"return not (self.unaccounted or self.double or self.unverified or self.invalid)",
"return not (self.unaccounted or self.double)",
),
Mutant(
"M22 any asset file at all proves a carry",
GATE,
" found == digest and name.startswith(digest[:12])"
" for name, found in build.assets.items()",
" True for name, found in build.assets.items()",
),
Mutant(
"M23 a corpus refused whole is not reported",
GATE,
" if persisted:\n return None",
" if True:\n return None",
),
Mutant(
"M24 a rejection code outside the closed list is accepted",
GATE,
" unknown = sorted(c for c in rejected if c not in REJECTION_CODES)",
" unknown = []",
),
Mutant(
"M25 a skipped row leaves the verdict unqualified",
GATE,
' verdict += f" (row {row.number} not run: {row.reason})"',
' verdict += ""',
),
Mutant(
"W01 docx: separator footnotes counted",
WITNESS,
'if role != "comment" and int(note.get(f"{_W}id", "0")) <= 0:',
'if role != "comment" and int(note.get(f"{_W}id", "0")) <= -1:',
),
Mutant(
"W02 pptx: pictures never counted",
WITNESS,
' for _ in root.iter(f"{_P}pic"):\n count.add("image")',
' for _ in ():\n count.add("image")',
),
Mutant(
"W03 sts (xml witness): footnotes have no role",
WITNESS,
' if tag == "fn":\n return "footnote"',
' if tag == "fn":\n return None',
occurrences=2,
first_only=True,
),
Mutant(
"W04 xlsx: images never counted",
WITNESS,
' for _ in root.iter(f"{_XDR}pic"):\n count.add("image")',
' for _ in ():\n count.add("image")',
),
Mutant(
"W05 odt: list items never counted",
WITNESS,
' for item in root.iter(f"{_TEXT}list-item"):',
" for item in ():",
),
Mutant(
"W06 docx: tables never counted",
WITNESS,
' for table in root.iter(f"{_W}tbl"):',
" for table in ():",
),
Mutant(
"W07 sts (xml witness): table cells have no role",
WITNESS,
' if tag in ("td", "th"):\n return "cell"',
' if tag in ("td", "th"):\n return None',
occurrences=2,
first_only=True,
),
Mutant(
"W08 docx: a header or footer is never counted",
WITNESS,
' count.add("header_footer", *lines)',
" pass",
),
Mutant(
"W09 sts json: the twin reuses the XML map",
WITNESS,
" role = _sts_role_json(tag, parent, grandparent)",
" role = _sts_role_xml(tag, parent, grandparent)",
),
)
def _apply(text: str, mutant: Mutant) -> str:
return text.replace(mutant.old, mutant.new, 1 if mutant.first_only else -1)
def main(argv: list[str] | None = None) -> int:
del argv
survived: list[str] = []
killed = 0
errors: list[str] = []
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp) / "tree"
shutil.copytree(
REPO,
root,
ignore=shutil.ignore_patterns(".git", ".venv", "__pycache__", "*.egg-info"),
)
for mutant in MUTANTS:
path = root / mutant.file
original = path.read_text(encoding="utf-8")
seen = original.count(mutant.old)
if seen != mutant.occurrences:
errors.append(
f"{mutant.label}: pattern occurs {seen} times, expected "
f"{mutant.occurrences} -- NOT MEASURED"
)
continue
path.write_text(_apply(original, mutant), encoding="utf-8")
try:
run = subprocess.run(
[sys.executable, "-m", "pytest", SUITE, "-q", "-x", "-p", "no:cacheprovider"],
cwd=root,
capture_output=True,
text=True,
)
finally:
path.write_text(original, encoding="utf-8")
tail = (run.stdout.strip().splitlines() or [run.stderr.strip()[-160:]])[-1]
if run.returncode != 0:
killed += 1
print(f"killed {mutant.label} [{tail}]")
else:
survived.append(mutant.label)
print(f"SURVIVED {mutant.label} [{tail}]")
print(f"\nkilled {killed} of {len(MUTANTS)}")
for label in survived:
print(f" survived: {label}")
for problem in errors:
print(f" ERROR: {problem}")
return 2 if errors else 0
if __name__ == "__main__":
raise SystemExit(main())

27
tools/okf_witness.py
View file

@ -453,13 +453,15 @@ def _sts_role_xml(tag: str, parent: str | None, grandparent: str | None) -> str
number neither of them should have produced (independent review, M-2).
In this delivery a section's label is `sec/label` and a table's label is
`table-wrap/label`.
`table-wrap/label`. Measured on R761 2026-09-18: `sec/label` 7 714,
`sec/title/label` **0** -- the nested placement is a fact about the JSON
delivery and does not belong in this map.
"""
if tag == "sec":
return "section"
if tag == "title" and parent == "sec":
return "title"
if tag == "label" and (parent == "sec" or (parent == "title" and grandparent == "sec")):
if tag == "label" and parent == "sec":
return "section_label"
if tag == "label" and parent == "table-wrap":
return "table_label"
@ -578,6 +580,22 @@ def count_sts_xml(data: bytes) -> tuple[Count, list[str], bool]:
return count, refs, True
def is_sts_json(data: bytes) -> bool:
"""Is this the publisher's JSON delivery of an STS document?
A standard shipped as JSON holds sections, titles, citations and tables.
Counted as generic JSON it holds keys and leaves: the container, not the
content -- and then no row can see that a citation left the bundle."""
if b'"standardContent"' not in data:
return False
try:
document = json.loads(data)
except ValueError:
return False
content = document.get("standardContent") if isinstance(document, dict) else None
return isinstance(content, dict) and isinstance(content.get("c"), list)
def count_sts_json(data: bytes) -> Count:
"""The same roles, read from the publisher's JSON node tree."""
document = json.loads(data)
@ -1226,7 +1244,10 @@ def witness_file(inbox: Path, path: Path) -> Inventory:
elif suffix == ".csv":
count, name = count_csv(data.decode("utf-8-sig")), "csv"
elif suffix == ".json":
count, name = count_json(data.decode("utf-8-sig")), "json"
if is_sts_json(data):
count, name = count_sts_json(data), "sts json node tree"
else:
count, name = count_json(data.decode("utf-8-sig")), "json"
elif suffix in (".html", ".htm"):
parser = _HtmlCounter()
parser.feed(data.decode("utf-8-sig"))