test(accounting): the witnesses see what the formats actually hold (M-1..M-3)

Rows 2 and 3 require the build's inventory to EQUAL the witness's, so what
the witness does not count, nothing can lose visibly. An independent review
put a header and a comment in a docx, measured 0 of either in the bundle,
and the accounting still read "2 of 2 carried".

Thirteen classes are now counted, each with a red test written first:
docx header/footer, comment, endnote and text box (a box's paragraphs are
its own, or the text is booked twice) - pptx speaker note and hidden slide
(`show="0"`, no longer counted as an ordinary slide) - xlsx formula and
hidden sheet (the state lives in `workbook.xml` and is reached through the
relationship id, so the sheet part itself says nothing about it) - odt
header/footer from `styles.xml` and annotation (counted as prose, it made
the accounting demand a reader carry a note the author wrote to themselves)
- STS `mixed-citation`, `mml:math`, `fig` and its caption, measured by the
review at 4.1 % of N200's source text and 3.9 % of N100's.

M-2: the two STS witnesses had ONE role map between them, so row 5 -- "two
witnesses agree" -- could not see a hole in it. `_sts_role_xml` and
`_sts_role_json` are written apart, each for its own delivery, and a test
holds them apart.

M-3: 20 of 63 element types had a count of ZERO in their only fixture. Seven
hand-built documents close it, every element type now occurs at least once
(a test asserts it), and ALL TWENTY documents carry a hand count read off
the fixture's own bytes (four did before). `.xlsx image` -- the operator's
own proposed exception -- could not be exercised at all until now.

Every witness also states WHAT IT STILL DOES NOT COUNT, per file type, and
the gate prints that list on every run.

THE FIXTURE ROWS ARE RED NOW, AND THAT IS THE POINT. Row 2 red on .docx,
.odt, .pptx, .xlsx and .xml; row 3 at u = 25, d = 2 over the new classes,
including a footnote and four spreadsheet cells the build genuinely drops.
`0 claimed and not found` on the same run: nothing the build DOES book as
carried failed the bundle check, so the red is the build's and not the
instrument's.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

tools/okf_accounting_gate.py

@@ -980,6 +980,13 @@ def render(rows: list[Row]) -> str:
     lines.append(f"exceptions approved ({APPROVED_ON}), and none moves a denominator:")
     for suffix, element in sorted(APPROVED_EXCEPTIONS) or [("(none)", "")]:
         lines.append(f"    - {suffix} {element}")
+    lines += [
+        "",
+        "not counted by any witness -- what no row here can see (per file type):",
+    ]
+    for suffix in sorted(witness.NOT_COUNTED):
+        for item in witness.NOT_COUNTED[suffix]:
+            lines.append(f"    - {suffix}: {item}")
     failing = [str(r.number) for r in rows if r.fails]
     lines.append("")
     lines.append(

tools/okf_witness.py

@@ -141,6 +141,12 @@ class Inventory:
         }
 
 
+def _normal(text: str) -> str:
+    """One space between words: a reader re-wraps, and a piece has to survive
+    that to be looked for at all."""
+    return " ".join(text.split())
+
+
 def _local(tag: str) -> str:
     return tag.rsplit("}", 1)[-1] if "}" in tag else tag.split(":")[-1]
 
@@ -421,33 +427,33 @@ class _HtmlCounter(HTMLParser):
 # --- xml / sts ---------------------------------------------------------------
 
 STS_ROLES = (
-    "section",
-    "title",
-    "section_label",
+    "cell",
+    "citation",
+    "figure",
+    "figure_caption",
+    "footnote",
+    "image",
+    "list_item",
+    "math",
     "paragraph",
+    "section",
+    "section_label",
     "table",
     "table_label",
-    "cell",
-    "list_item",
-    "image",
-    "footnote",
+    "title",
 )
 
 
-def _sts_role(tag: str, parent: str | None, grandparent: str | None) -> str | None:
-    """The one mapping from an STS element to its accounting role.
+def _sts_role_xml(tag: str, parent: str | None, grandparent: str | None) -> str | None:
+    """An STS XML element's accounting role.
 
-    Used by BOTH STS witnesses, and the role is the unit, not the tag, because
-    the publisher's two deliveries of one document place the same text
-    differently (measured on R761 Prosesskoden:2025, 2026-09-17):
+    Written for the XML delivery ALONE. Until 2026-09-18 one function served
+    both deliveries, so row 5 -- "two witnesses agree" -- could not see a hole
+    in it: a role missing here was missing there, and the two agreed on a
+    number neither of them should have produced (independent review, M-2).
 
-    - a section's label: XML `sec/label` on 7 714 sections; JSON `sec/label`
-      on 4 954 and `sec/title/label` on the 2 760 that carry a title.
-    - a table's label: XML `table-wrap/label` (10); JSON
-      `table-wrap/table/caption` (10).
-
-    Counted by tag, the two witnesses disagree by 2 760 and by 10 on text
-    both of them carry.
+    In this delivery a section's label is `sec/label` and a table's label is
+    `table-wrap/label`.
     """
     if tag == "sec":
         return "section"
@@ -459,6 +465,14 @@ def _sts_role(tag: str, parent: str | None, grandparent: str | None) -> str | No
         return "table_label"
     if tag == "caption" and parent == "table" and grandparent == "table-wrap":
         return "table_label"
+    if tag == "caption" and parent == "fig":
+        return "figure_caption"
+    if tag == "fig":
+        return "figure"
+    if tag == "mixed-citation":
+        return "citation"
+    if tag == "math":
+        return "math"
     if tag == "p":
         return "paragraph"
     if tag == "table-wrap":
@@ -474,8 +488,54 @@ def _sts_role(tag: str, parent: str | None, grandparent: str | None) -> str | No
     return None
 
 
-def _normal(text: str) -> str:
-    return " ".join(text.split())
+def _sts_role_json(tag: str, parent: str | None, grandparent: str | None) -> str | None:
+    """The same roles, read from the publisher's JSON node tree.
+
+    Written apart from the XML map, because the publisher's two deliveries of
+    ONE document place the same text differently (measured on R761
+    Prosesskoden:2025, 2026-09-17):
+
+    - a section's label: XML `sec/label` on 7 714 sections; JSON `sec/label`
+      on 4 954 and `sec/title/label` on the 2 760 that carry a title.
+    - a table's label: XML `table-wrap/label` (10); JSON
+      `table-wrap/table/caption` (10).
+
+    Counted by tag alone, the two witnesses disagree by 2 760 and by 10 on
+    text both of them carry. The role, not the tag, is the unit.
+    """
+    if tag == "sec":
+        return "section"
+    if tag == "title" and parent == "sec":
+        return "title"
+    if tag == "label" and parent == "sec":
+        return "section_label"
+    if tag == "label" and parent == "title" and grandparent == "sec":
+        return "section_label"
+    if tag == "label" and parent == "table-wrap":
+        return "table_label"
+    if tag == "caption" and parent == "table" and grandparent == "table-wrap":
+        return "table_label"
+    if tag == "caption" and parent == "fig":
+        return "figure_caption"
+    if tag == "fig":
+        return "figure"
+    if tag == "mixed-citation":
+        return "citation"
+    if tag == "math":
+        return "math"
+    if tag == "p":
+        return "paragraph"
+    if tag == "table-wrap":
+        return "table"
+    if tag in ("td", "th"):
+        return "cell"
+    if tag == "list-item":
+        return "list_item"
+    if tag in ("graphic", "inline-graphic"):
+        return "image"
+    if tag == "fn":
+        return "footnote"
+    return None
 
 
 def count_sts_xml(data: bytes) -> tuple[Count, list[str], bool]:
@@ -499,7 +559,7 @@ def count_sts_xml(data: bytes) -> tuple[Count, list[str], bool]:
 
     def walk(node: ET.Element, parent: str | None, grandparent: str | None) -> list[str]:
         tag = _local(node.tag)
-        role = _sts_role(tag, parent, grandparent)
+        role = _sts_role_xml(tag, parent, grandparent)
         pieces: list[str] = []
         if node.text and node.text.strip():
             pieces.append(_normal(node.text))
@@ -531,7 +591,7 @@ def count_sts_json(data: bytes) -> Count:
         if not isinstance(body, dict):
             return pieces
         tag = str(body.get("tag"))
-        role = _sts_role(tag, parent, grandparent)
+        role = _sts_role_json(tag, parent, grandparent)
         for child in body.get("c") or []:
             pieces.extend(walk(child, tag, parent))
         if role is not None:
@@ -560,7 +620,20 @@ def _text_of(node: ET.Element, tag: str) -> str:
     return "".join(t.text or "" for t in node.iter(tag))
 
 
-DOCX = ("cell", "footnote", "heading", "image", "paragraph", "table")
+DOCX = (
+    "cell",
+    "comment",
+    "endnote",
+    "footnote",
+    "header_footer",
+    "heading",
+    "image",
+    "paragraph",
+    "table",
+    "text_box",
+)
+
+_DOCX_HEADER_FOOTER = re.compile(r"word/(header|footer)\d*\.xml")
 
 
 def _docx_lines(para: ET.Element) -> list[str]:
@@ -570,11 +643,20 @@ def _docx_lines(para: ET.Element) -> list[str]:
     a broken paragraph can land in different places -- inside a grid table
     they land on different rows, with other cells' text between them."""
     lines = [""]
-    for node in para.iter():
-        if node.tag == f"{_W}t":
-            lines[-1] += node.text or ""
-        elif node.tag in (f"{_W}br", f"{_W}cr"):
-            lines.append("")
+
+    def walk(node: ET.Element) -> None:
+        for child in node:
+            # A text box holds its own paragraphs. Read as part of the
+            # paragraph that carries the box, its text is counted twice.
+            if child.tag == f"{_W}txbxContent":
+                continue
+            if child.tag == f"{_W}t":
+                lines[-1] += child.text or ""
+            elif child.tag in (f"{_W}br", f"{_W}cr"):
+                lines.append("")
+            walk(child)
+
+    walk(para)
     return [_normal(line) for line in lines if line.strip()]
 
 
@@ -583,12 +665,23 @@ def count_docx(data: bytes) -> Count:
     other w:p with text. table: w:tbl. cell: w:tc. image: a:blip.
     footnote: a w:footnote with a positive id."""
     count = Count(DOCX)
+    parts: dict[str, ET.Element] = {}
     with zipfile.ZipFile(io.BytesIO(data)) as archive:
         root = ET.fromstring(archive.read("word/document.xml"))
-        notes = None
-        if "word/footnotes.xml" in archive.namelist():
-            notes = ET.fromstring(archive.read("word/footnotes.xml"))
+        for name in sorted(archive.namelist()):
+            if name in ("word/footnotes.xml", "word/endnotes.xml", "word/comments.xml") or (
+                _DOCX_HEADER_FOOTER.fullmatch(name)
+            ):
+                parts[name] = ET.fromstring(archive.read(name))
+    # A text box's paragraphs are `w:p` in the body too: counted as prose they
+    # would be booked twice, so the box owns them and they are its pieces.
+    boxed: set[int] = set()
+    for box in root.iter(f"{_W}txbxContent"):
+        boxed.update(id(p) for p in box.iter(f"{_W}p"))
+        count.add("text_box", *[line for p in box.iter(f"{_W}p") for line in _docx_lines(p)])
     for para in root.iter(f"{_W}p"):
+        if id(para) in boxed:
+            continue
         style = para.find(f"{_W}pPr/{_W}pStyle")
         lines = _docx_lines(para)
         if style is not None and _HEADING_STYLE.match(style.get(f"{_W}val", "")):
@@ -601,16 +694,29 @@ def count_docx(data: bytes) -> Count:
         count.add("cell", *[line for p in cell.iter(f"{_W}p") for line in _docx_lines(p)])
     for _ in root.iter(f"{_A}blip"):
         count.add("image")
-    if notes is not None:
-        for note in notes.iter(f"{_W}footnote"):
-            if int(note.get(f"{_W}id", "0")) > 0:
-                count.add(
-                    "footnote", *[line for p in note.iter(f"{_W}p") for line in _docx_lines(p)]
-                )
+    for name, part in parts.items():
+        if _DOCX_HEADER_FOOTER.fullmatch(name):
+            for para in part.iter(f"{_W}p"):
+                lines = _docx_lines(para)
+                if lines:
+                    count.add("header_footer", *lines)
+            continue
+        role, tag = (
+            ("footnote", f"{_W}footnote")
+            if name.endswith("footnotes.xml")
+            else ("endnote", f"{_W}endnote")
+            if name.endswith("endnotes.xml")
+            else ("comment", f"{_W}comment")
+        )
+        for note in part.iter(tag):
+            # A separator note carries id 0 and no document text.
+            if role != "comment" and int(note.get(f"{_W}id", "0")) <= 0:
+                continue
+            count.add(role, *[line for p in note.iter(f"{_W}p") for line in _docx_lines(p)])
     return count
 
 
-PPTX = ("cell", "image", "paragraph", "slide", "table", "title")
+PPTX = ("cell", "hidden_slide", "image", "note", "paragraph", "slide", "table", "title")
 
 
 def count_pptx(data: bytes) -> Count:
@@ -620,9 +726,15 @@ def count_pptx(data: bytes) -> Count:
     count = Count(PPTX)
     with zipfile.ZipFile(io.BytesIO(data)) as archive:
         slides = [n for n in archive.namelist() if re.fullmatch(r"ppt/slides/slide\d+\.xml", n)]
+        notes = [
+            n for n in archive.namelist() if re.fullmatch(r"ppt/notesSlides/notesSlide\d+\.xml", n)
+        ]
         for name in sorted(slides, key=_slide_order):
             root = ET.fromstring(archive.read(name))
-            count.add("slide", *_pptx_lines(root))
+            # `show="0"` is the deck saying this slide is not shown. Counted as
+            # an ordinary slide it is indistinguishable from one that is.
+            hidden = root.get("show") == "0"
+            count.add("hidden_slide" if hidden else "slide", *_pptx_lines(root))
             for table in root.iter(f"{_A}tbl"):
                 count.add("table", *_pptx_lines(table))
             for cell in root.iter(f"{_A}tc"):
@@ -645,6 +757,10 @@ def count_pptx(data: bytes) -> Count:
                 else:
                     for text in texts:
                         count.add("paragraph", text)
+        for name in sorted(notes, key=_slide_order):
+            root = ET.fromstring(archive.read(name))
+            for line in _pptx_lines(root):
+                count.add("note", line)
     return count
 
 
@@ -660,7 +776,7 @@ def _slide_order(name: str) -> tuple[int, str]:
     return (int(match.group(1)) if match else 0, name)
 
 
-XLSX = ("cell", "image", "row", "sheet")
+XLSX = ("cell", "formula", "hidden_sheet", "image", "row", "sheet")
 
 
 def _shared_strings(archive: zipfile.ZipFile) -> list[str]:
@@ -684,6 +800,31 @@ def _cell_text(cell: ET.Element, shared: list[str]) -> str:
     return _normal(raw)
 
 
+def _hidden_sheets(archive: zipfile.ZipFile) -> set[str]:
+    """The worksheet PARTS the workbook marks hidden.
+
+    The sheet file says nothing about it: the state lives in `workbook.xml`
+    and the part is reached through the relationship id."""
+    names = archive.namelist()
+    if "xl/workbook.xml" not in names or "xl/_rels/workbook.xml.rels" not in names:
+        return set()
+    relationships = ET.fromstring(archive.read("xl/_rels/workbook.xml.rels"))
+    targets = {
+        node.get("Id", ""): str(node.get("Target", ""))
+        for node in relationships
+        if _local(node.tag) == "Relationship"
+    }
+    hidden: set[str] = set()
+    workbook = ET.fromstring(archive.read("xl/workbook.xml"))
+    for sheet in workbook.iter(f"{_S}sheet"):
+        if sheet.get("state") in ("hidden", "veryHidden"):
+            rid = next((v for k, v in sheet.attrib.items() if _local(k) == "id"), "")
+            target = targets.get(rid, "")
+            if target:
+                hidden.add(f"xl/{target.lstrip('/')}" if not target.startswith("xl/") else target)
+    return hidden
+
+
 def count_xlsx(data: bytes) -> Count:
     """sheet: xl/worksheets/sheetN.xml. row: a row holding a value. cell: a c
     with a value. image: an xdr:pic in a drawing.
@@ -693,6 +834,7 @@ def count_xlsx(data: bytes) -> Count:
     count = Count(XLSX)
     with zipfile.ZipFile(io.BytesIO(data)) as archive:
         shared = _shared_strings(archive)
+        hidden = _hidden_sheets(archive)
         for name in sorted(archive.namelist()):
             if re.fullmatch(r"xl/worksheets/sheet\d+\.xml", name):
                 root = ET.fromstring(archive.read(name))
@@ -706,10 +848,14 @@ def count_xlsx(data: bytes) -> Count:
                     values = [_cell_text(c, shared) for c in valued]
                     for value in values:
                         count.add("cell", value)
+                    for cell in valued:
+                        formula = cell.find(f"{_S}f")
+                        if formula is not None:
+                            count.add("formula", _normal(formula.text or ""))
                     if valued:
                         count.add("row", *values)
                     sheet_pieces.extend(values)
-                count.add("sheet", *sheet_pieces)
+                count.add("hidden_sheet" if name in hidden else "sheet", *sheet_pieces)
             elif re.fullmatch(r"xl/drawings/drawing\d+\.xml", name):
                 root = ET.fromstring(archive.read(name))
                 for _ in root.iter(f"{_XDR}pic"):
@@ -717,19 +863,43 @@ def count_xlsx(data: bytes) -> Count:
     return count
 
 
-ODT = ("cell", "heading", "image", "list_item", "paragraph", "table")
+ODT = (
+    "annotation",
+    "cell",
+    "header_footer",
+    "heading",
+    "image",
+    "list_item",
+    "paragraph",
+    "table",
+)
+
+_OFFICE = "{urn:oasis:names:tc:opendocument:xmlns:office:1.0}"
+_STYLE = "{urn:oasis:names:tc:opendocument:xmlns:style:1.0}"
 
 
 def count_odt(data: bytes) -> Count:
     """heading: text:h. paragraph: a text:p with text outside a table cell.
     table: table:table. cell: table:table-cell. list_item: text:list-item.
     image: draw:image."""
+    styles = None
     with zipfile.ZipFile(io.BytesIO(data)) as archive:
         root = ET.fromstring(archive.read("content.xml"))
+        if "styles.xml" in archive.namelist():
+            styles = ET.fromstring(archive.read("styles.xml"))
     count = Count(ODT)
     in_cell: set[int] = set()
     for cell in root.iter(f"{_TABLE}table-cell"):
         in_cell.update(id(p) for p in cell.iter(f"{_TEXT}p"))
+    # A comment is not prose. Counted as a paragraph it makes the accounting
+    # demand that a reader carry a note the author wrote to themselves.
+    annotated: set[int] = set()
+    for note in root.iter(f"{_OFFICE}annotation"):
+        annotated.update(id(p) for p in note.iter(f"{_TEXT}p"))
+        count.add(
+            "annotation",
+            *[_normal("".join(p.itertext())) for p in note.iter(f"{_TEXT}p")],
+        )
 
     def lines(node: ET.Element) -> list[str]:
         pieces = [_normal("".join(p.itertext())) for p in node.iter(f"{_TEXT}p")]
@@ -740,7 +910,7 @@ def count_odt(data: bytes) -> Count:
         count.add("heading", _normal("".join(heading.itertext())))
     for para in root.iter(f"{_TEXT}p"):
         text = _normal("".join(para.itertext()))
-        if id(para) not in in_cell and text:
+        if id(para) not in in_cell and id(para) not in annotated and text:
             count.add("paragraph", text)
     for table in root.iter(f"{_TABLE}table"):
         count.add("table", *lines(table))
@@ -750,6 +920,15 @@ def count_odt(data: bytes) -> Count:
         count.add("list_item", *lines(item))
     for _ in root.iter(f"{_DRAW}image"):
         count.add("image")
+    # The header and the footer live in `styles.xml`, which is why no reader
+    # looking only at `content.xml` can see them at all.
+    if styles is not None:
+        for place in (f"{_STYLE}header", f"{_STYLE}footer"):
+            for region in styles.iter(place):
+                for para in region.iter(f"{_TEXT}p"):
+                    text = _normal("".join(para.itertext()))
+                    if text:
+                        count.add("header_footer", text)
     return count
 
 
@@ -759,10 +938,11 @@ _RTF_CONTROL = re.compile(r"\\([a-zA-Z]+)(-?\d+)? ?|\\'([0-9a-fA-F]{2})|\\([^a-z
 
 #: Groups that hold no document text. `{\*...}` says so in the format itself;
 #: these say it by name, and without them a fixture's font table reads as the
-#: first paragraph of its prose.
+#: first paragraph of its prose and a picture's hex payload as the next one.
 _RTF_SILENT = frozenset(
     {
         "fonttbl",
+        "pict",
         "colortbl",
         "stylesheet",
         "info",
@@ -957,6 +1137,79 @@ WITNESSED_SUFFIXES = (
 )
 
 
+#: What each witness STILL does not count, by name and per file type.
+#:
+#: An accounting can only lose visibly what something counts, so this list is
+#: the gate's own statement of its blind spots -- printed on every run, never
+#: inferred, and the raw material for the next capability order. Written
+#: 2026-09-18 from an independent review's per-format reading of this file.
+NOT_COUNTED: dict[str, tuple[str, ...]] = {
+    ".csv": (
+        "a semicolon-separated file (the Norwegian default) reads as one cell per row",
+        "quoting and encoding errors, which arrive as text",
+    ),
+    ".docx": (
+        "SmartArt, charts and embedded OLE objects",
+        "tracked deletions",
+        "hyperlink targets (the link text counts, the address does not)",
+        "an equation written as `m:oMath` (it holds `m:t`, not `w:t`)",
+        "a picture's alt text",
+    ),
+    ".htm": (
+        "text in `div`, `blockquote`, `pre`, `dd`, `figcaption`, `caption` and bare text",
+        "`alt` and `title` attributes",
+        "`details`/`summary`, `picture`/`source`, `svg`, `object`, `iframe`",
+    ),
+    ".html": (
+        "text in `div`, `blockquote`, `pre`, `dd`, `figcaption`, `caption` and bare text",
+        "`alt` and `title` attributes",
+        "`details`/`summary`, `picture`/`source`, `svg`, `object`, `iframe`",
+    ),
+    ".json": ("the order of members, and comments a JSON superset would allow",),
+    ".md": (
+        "Setext headings (`===`, `---`)",
+        "reference images `![a][r]` and raw `<img>`",
+        "footnotes, indented code blocks and front matter (they count as paragraphs)",
+    ),
+    ".odt": (
+        "tracked changes",
+        "`draw:object` (an embedded chart or formula)",
+        "a picture's `xlink:href`: an odt image counts as embedded, so its bytes "
+        "cannot be traced to an inbox file",
+    ),
+    ".pdf": (
+        "headings, paragraphs and tables as such (operator-approved exception, "
+        "2026-09-17): the page's text is counted, its structure is not",
+        "form fields, annotations, attachments and bookmarks",
+    ),
+    ".pptx": (
+        "SmartArt, charts and comments",
+        "alt text",
+        "a slide layout's and master's own text",
+    ),
+    ".rtf": (
+        "text inside `\\header`, `\\footer` and `\\footnote` counts as body prose",
+        "a picture Word writes twice (`\\shppict` and `\\nonshppict`) counts twice",
+        "a picture's payload is binary, so the paragraph holding it has no text "
+        "the gate can look for",
+        "an empty `\\par` counts as a paragraph, where docx counts only one with text",
+    ),
+    ".txt": ("nothing beyond lines and paragraphs: the format declares no more",),
+    ".xlsx": (
+        "merged cells, cell comments, defined names and charts",
+        "a sheet's NAME",
+        "number formats (a date reads as its serial number)",
+    ),
+    ".xml": (
+        "`ref` and `element-citation` outside `mixed-citation`",
+        "`def-list`, `term-sec` and `app`",
+        "a `non-normative-note`'s label",
+        "attributes, and all text of an XML document that is not NISO-STS beyond "
+        "the element's own text",
+    ),
+}
+
+
 def witness_file(inbox: Path, path: Path) -> Inventory:
     """Count one file under `inbox`. Raises WitnessRefused for a file the
     witness does not read."""