test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote

`docs/2026-09-04-k2-pptx-odt-rtf.md` measured the corpus denominator for
these three rows at zero, so K2 for them was unmeasured rather than
passing. This measures them on the synthetic set committed in `a7b050b`,
whose hand count landed BEFORE any of these numbers existed.

Coverage 59/59 and pairing 20/20 for all three, standalone and product
alike, where "product" is scored off the persisted concept in the bundle
rather than an intermediate. The instrument's own denominator, derived
from each file's XML or stream, agrees with the hand count on all three:
59 strings, 20 pairable. Door B over the fixture directory persists 3/3
with nothing quarantined, rejected, failed or skipped, so K1b holds at
3 = N. There is no sub-100 % cell to explain, and that is a result about
three documents we wrote, not about the formats.

THE NEGATIVE CONTROL IS WHAT MAKES THOSE FIGURES MEAN ANYTHING. Removing
one label and its value from each converted text moves every row to
57/59 and 19/20 -- exactly the two strings and the one pair taken away.
`tests/test_fidelity.py` carries that control per reader, so it cannot
rot into a claim.

The order names `okf_fidelity` as the instrument and the instrument
refused all three types (`no source-string reader for '.pptx'`), so step
2 could not run without readers for them. Added here, stated in the
report as sitting outside the order's enumerated fence: this is the
measuring instrument, not extraction code. `extract.py` is untouched and
every `_EVIDENCE` row still reads `unmeasured` -- three synthetic
documents in one house style are exercise, not coverage, and the suite
asserts the rows are not promoted.

Two converter findings, reported and not fixed, both exit code 0 with no
warning. RTF cell paragraphs without `\pard\intbl` are read as each row
nested inside the previous one, 2076 characters where 117 were expected.
And the `\uN?` unicode escape -- THE FORM WORD EMITS -- loses the
character after it: `A\u248?BC` reads back as `AoC`. Any Word-produced
RTF carrying Norwegian letters loses one character after each of them,
silently. That is now a named limit of the `rtf` row rather than an
unknown.

Pandoc 3.9 and the host's 3.10.2 produce byte-identical output for all
three fixtures, so the numbers are not sitting on a version-fragile edge.
`~/corpora/` was neither read nor written; N there is still 43.

Suite 1141 passed, `ruff` clean, `mypy --strict src/ tools/` clean over
25 files, all after `git add`. Regenerating the fixtures leaves
`git diff --quiet` clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-07 05:22:12 +02:00
commit bd44929c89
3 changed files with 443 additions and 4 deletions

View file

@ -0,0 +1,186 @@
# K2 for pptx/odt/rtf on a synthetic denominator, N = 3, 2026-09-07
Order `20260906T220349Z-652851369-from-.claude`. Closes the gap the previous
report opened rather than filled: `docs/2026-09-04-k2-pptx-odt-rtf.md` measured
the corpus denominator for these three office rows and found it **zero**
`~/corpora/okf-telling-20260829/K2/trinn1` holds 43 files and not one is a
`pptx`, an `odt` or an `rtf`, with a known-positive control (`docx` → 5/5)
proving the query could find. So K2 for those rows was *unmeasured*, which is a
different fact from *passes*, and the corpus could not be made to say otherwise.
This report measures them on a **synthetic** denominator instead: three
documents written here, with a hand count committed before the measurement ran.
That is a weaker instrument than a corpus and the limits are stated in full at
the bottom. It is not weaker than nothing, which is what the row had.
**Nothing in `~/corpora/` was read or written.** N there is still 43.
## What was measured, and against what
`tools/okf_fidelity.py` asks the source document's own two questions:
- **coverage** — of the strings the FILE stores, how many appear in the
converted text. Read from the package's own XML (or, for RTF, from the
stream), never from a converter's output: a fasit derived from one converter
would score that converter on its own homework.
- **pairing** — of the rows carrying a label and a value, how many keep both on
one output line. That is the criterion a requirement table is read by.
The fixture set is `tests/fixtures/k2-office/`: **one authored document laid out
three times, in three containers**, so the container and its reader are the only
variable between the three measurements. A title, an intro, a 20-row
label/value table, a caption, and a 4×4 grid. Norwegian throughout, with the
diacritics the pipeline has to survive.
**The fasit was hand-counted from the authored content and committed first**, in
`a7b050b`, before any of the numbers below existed:
| | count | how it was counted |
|---|---|---|
| cells | **56** | 20 rows × 2 columns + 4 rows × 4 columns |
| pairs | **20** | the 20 label rows; every label ends in a colon |
| distinct strings | **59** | title + intro + caption + 56 cells, all distinct after normalisation |
The instrument derives its own denominator from each file's XML or stream,
independently of that hand count. **The two agree on all three documents:
59 strings, 20 pairable.** A disagreement would have been the report; there
isn't one.
## The result
Converter: the **vendored pandoc 3.9**, resolved by path and version-asserted by
`src/llm_ingestion_okf/_pandoc.py`, with `-t markdown --eol=lf --wrap=none`.
"Product" is the text scored off the **persisted concept in the bundle** — the
artifact Door B actually writes, not an intermediate.
| format | coverage | paired | denominator |
|---|---|---|---|
| `pptx` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
| `odt` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
| `rtf` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
Standalone (converter alone) and product (through `extract_text`, and through
the written concept) are equal for all three — no fall from one to the other, so
there is nothing here to attribute to this library.
**There is no sub-100 % cell to explain.** The order asks that every fall below
100 % be named with a cause; there are none, and that absence is a result about
a document set we wrote, not about the format.
**Door B, product path, N = 3:** `process_inbox` over the fixture directory
persisted **3/3**, with 0 quarantined, 0 rejected, 0 failed and 0 skipped.
**K1b: 3 + 0 = 3 = N.**
### The negative control
100 % means nothing from an instrument that cannot see a loss. Removing one
label and its value from each converted text, and re-scoring the same shipped
fixtures:
| format | coverage | paired |
|---|---|---|
| `pptx` | 57/59 (96.6 %) | 19/20 (95.0 %) |
| `odt` | 57/59 (96.6 %) | 19/20 (95.0 %) |
| `rtf` | 57/59 (96.6 %) | 19/20 (95.0 %) |
The instrument moves by exactly the two strings and the one pair that were
taken away. `tests/test_fidelity.py` carries the same control per reader as a
test, so it cannot rot.
### Converter version
The order asks for a comparison against the host's newer converter if the
numbers fell. They did not, and the comparison was run anyway because it is
cheap and it bounds a different risk: **pandoc 3.9 and the host's pandoc 3.10.2
produce byte-identical output for all three fixtures** (`cmp`, three files,
no differences). These numbers are therefore not sitting on a version-fragile
edge.
## Two converter findings, neither of them fixed
Both were measured while laying the RTF out, both are structurally plausible
input read silently wrong, and both come back **exit code 0 with no warning**
the same shape as the missing `word/styles.xml` and the `inlineStr` xlsx already
recorded in `tests/fixtures/README.md`. Per the order they are reported, not
patched; nothing in `src/` works around either.
**1. RTF cell paragraphs need `\pard\intbl`.** Without it, consecutive
`\trowd…\row` rows are read as each row NESTED inside the previous one. Five
label/value rows came back as five levels of nested table — **2076 characters
where 117 were expected**. The fixture writes `\intbl`, which is correct RTF; a
document that omits it is not this library's problem to fix, but it is a shape a
real drop directory can contain.
**2. The `\uN?` unicode escape loses the character after it.** This is the form
**Word emits**, so it is the common case rather than an edge one. Measured
directly against the vendored 3.9:
```
A\u248?BC -> AøC the ring letter arrives, the B is GONE
A\u248?xBC -> AøBC the same rule seen from the other side
A\u248 ?BC -> AøBC an explicit delimiter, and the text survives
```
The `?` is taken as the control word's delimiter, and `\uc1` then skips a *real*
character. `\uc0` does not change it. The fixture is written in the form that
round-trips (`\uN ?`), so this finding is recorded here rather than baked
invisibly into a fixture.
**What this means for a real `rtf` drop.** Any Word-produced RTF containing
Norwegian letters will lose one character after each of them, silently. That is
a defect in the converter, not in this library, and it is now a named limit of
the `rtf` row rather than an unknown.
## Honesty limits
Read these as part of the table, not as a footnote to it.
- **Synthetic, N = 3, one house style.** Three documents we wrote are not a
corpus. They are internally consistent by construction: one author, one
layout convention, one set of table shapes. A real `pptx` deck with text
boxes over images, an `odt` with tracked changes, or an RTF from a different
producer are all outside what was measured.
- **We authored the fixtures to be readable, and they read.** The 100 % says
the path works end to end on documents built to exercise it. It does not say
the format is covered. `extract._EVIDENCE` therefore still reads
`unmeasured` for all three rows, and `tests/test_k2_office_fixtures.py`
asserts that it does — so a later reader cannot mistake this report for
promotion.
- **The `rtf` row is the weakest of the three.** For `pptx` and `odt` the fasit
is read out of a package format that neither we nor the converter defined.
For `rtf` there is no part to read, so the fasit comes from a reader written
here against a stream written here. Both halves are ours, and the
independence the other two rows have is missing.
- **Drawn content is out of scope entirely.** Conversion recovers text; a
figure, a diagram or a drawn shape has no text to recover, only a caption.
Every office extraction already warns about this on every call. The fixtures
contain no drawn content, so this report measures nothing about it.
- **`pptx` speaker notes, `odt` footnotes and RTF headers/footers** carry text
in real documents and appear in none of these fixtures. Unmeasured, and named
here so the gap has a denominator of its own.
## What changed in the tree
- `tests/fixtures/k2-office/` — the three documents, hand-laid by
`tests/fixtures/make_k2_office.py`. **No converter wrote any of them**: a file
written by the converter and read by the converter proves only that the
converter agrees with itself. The commissioning order offered pandoc as one
generator option; the committed fixture policy in `tests/fixtures/README.md`
forbids it, and the policy wins.
- `tests/fixtures/k2-office-fasit.json` — the hand count, committed first.
- `tests/test_k2_office_fixtures.py` — pins N = 3, 3/3 through Door B offline,
the concept contents, the fasit, and that the rows stay `unmeasured`.
- `tools/okf_fidelity.py` — **source-string readers for `pptx`, `odt` and
`rtf`.** Stated plainly because it sits outside the order's enumerated fence:
the instrument the order names refused all three types
(`ValueError: no source-string reader for '.pptx'`), so step 2 could not run
without it. It is the measuring instrument, not extraction code —
`src/llm_ingestion_okf/extract.py` is untouched, and so is every `_EVIDENCE`
row.
**Suite: 1141 passed** (1127 before this work, 1132 after the fixture commit,
1141 with the reader tests). `ruff check`, `ruff format --check` and
`mypy --strict src/ tools/` (25 files) clean, all measured after `git add`.
Re-running `make_k2_office.py` leaves `git diff --quiet` clean on
`tests/fixtures/k2-office`, so the three documents are reproducible from that
one file. Nothing pushed.

View file

@ -17,6 +17,7 @@ from __future__ import annotations
import sys
import zipfile
from collections.abc import Callable
from pathlib import Path
import pytest
@ -113,9 +114,124 @@ def test_a_workbooks_shared_strings_are_read(tmp_path: Path) -> None:
def test_a_type_with_no_reader_is_refused_rather_than_scored_zero(tmp_path: Path) -> None:
"""Zero coverage and "no instrument" are different facts."""
path = tmp_path / "x.pptx"
"""Zero coverage and "no instrument" are different facts.
The type here USED to be `.pptx`, which now has a reader. Replaced in place
rather than deleted: the property is about the refusal, not about which
suffix happens to lack a reader this week, and dropping the test with the
reader would have removed the guarantee along with its example.
"""
path = tmp_path / "x.epub"
with zipfile.ZipFile(path, "w") as archive:
archive.writestr("a", "b")
with pytest.raises(ValueError):
okf_fidelity.source_strings(path)
# --- the three office rows the corpus has none of --------------------------
#
# `pptx`, `odt` and `rtf` had no source-string reader here, so the instrument
# refused them and K2 for those rows could not be run at all -- see
# `docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md`. Each reader is checked the
# same way as the two above: it finds the document's own strings, and it is
# able to see a loss.
SLIDE_XML = """<?xml version="1.0"?>
<p:sld xmlns:a="a" xmlns:p="p"><p:cSld><p:spTree>
<p:sp><p:txBody><a:p><a:r><a:t>Kravspesifikasjon for tunnel</a:t></a:r></a:p></p:txBody></p:sp>
<p:graphicFrame><a:tbl><a:tr>
<a:tc><a:txBody><a:p><a:r><a:t>Navn tilbyder:</a:t></a:r></a:p></a:txBody></a:tc>
<a:tc><a:txBody><a:p><a:r><a:t>Entreprenoer AS</a:t></a:r></a:p></a:txBody></a:tc>
</a:tr></a:tbl></p:graphicFrame>
</p:spTree></p:cSld></p:sld>
"""
CONTENT_XML = """<?xml version="1.0"?>
<office:document-content xmlns:office="o" xmlns:text="t" xmlns:table="tb">
<office:body><office:text>
<text:h text:outline-level="1">Kravspesifikasjon for tunnel</text:h>
<table:table><table:table-row>
<table:table-cell><text:p>Navn <text:span>tilbyder</text:span>:</text:p></table:table-cell>
<table:table-cell><text:p>Entreprenoer AS</text:p></table:table-cell>
</table:table-row></table:table>
</office:text></office:body></office:document-content>
"""
RTF_STREAM = (
"{\\rtf1\\ansi\\deff0{\\fonttbl{\\f0\\froman Times New Roman;}}\n"
"\\pard Kravspesifikasjon for tunnel\\par\n"
"\\trowd\\cellx3000\\cellx6000\n"
"\\pard\\intbl Navn tilbyder:\\cell \\pard\\intbl Entrepren\\u248 ?r AS\\cell \\row\n"
"}"
)
def pptx(tmp_path: Path) -> Path:
path = tmp_path / "p.pptx"
with zipfile.ZipFile(path, "w") as archive:
archive.writestr("ppt/slides/slide1.xml", SLIDE_XML)
return path
def odt(tmp_path: Path) -> Path:
path = tmp_path / "o.odt"
with zipfile.ZipFile(path, "w") as archive:
archive.writestr("content.xml", CONTENT_XML)
return path
def rtf(tmp_path: Path) -> Path:
path = tmp_path / "r.rtf"
path.write_text(RTF_STREAM, encoding="ascii")
return path
def test_a_presentations_slide_text_is_read(tmp_path: Path) -> None:
"""One string per `a:p`, table cells included -- a cell is a paragraph."""
assert set(okf_fidelity.source_strings(pptx(tmp_path))) == {
"kravspesifikasjon for tunnel",
"navn tilbyder:",
"entreprenoer as",
}
def test_an_odf_bodys_paragraphs_are_read(tmp_path: Path) -> None:
"""Headings count, and a run split across a `text:span` is still one string."""
assert set(okf_fidelity.source_strings(odt(tmp_path))) == {
"kravspesifikasjon for tunnel",
"navn tilbyder:",
"entreprenoer as",
}
def test_an_rtf_streams_paragraphs_and_cells_are_read(tmp_path: Path) -> None:
"""The control words are markup; the text between them is the document."""
assert set(okf_fidelity.source_strings(rtf(tmp_path))) == {
"kravspesifikasjon for tunnel",
"navn tilbyder:",
"entreprenør as",
}
@pytest.mark.parametrize("build", [pptx, odt, rtf])
def test_each_new_reader_can_see_a_loss(build: Callable[[Path], Path], tmp_path: Path) -> None:
"""The negative control, once per reader.
A reader that returns its strings and an instrument that always says yes
are indistinguishable from a green number, so every reader added here owes
the same proof as the first two.
"""
path = build(tmp_path)
strings = list(okf_fidelity.source_strings(path))
result = okf_fidelity.score(path, "\n".join(strings[:-1]))
assert result.covered == len(strings) - 1 < result.total
@pytest.mark.parametrize("build", [pptx, odt, rtf])
def test_each_new_reader_pairs_a_label_only_when_a_value_sits_beside_it(
build: Callable[[Path], Path], tmp_path: Path
) -> None:
path = build(tmp_path)
label = next(item for item in okf_fidelity.source_strings(path) if item.endswith(":"))
assert okf_fidelity.score(path, label).paired == 0
assert okf_fidelity.score(path, f"{label} en verdi").paired == 1

View file

@ -8,7 +8,8 @@ converter version, against the product path, or at all.
Two questions, both of them the source document's own:
- **coverage** -- of the strings the file itself stores (a `docx` paragraph, an
`xlsx` cell), how many appear in the converted text. Not character count: a
`xlsx` cell, a `pptx` or `odt` paragraph, an `rtf` cell), how many appear in
the converted text. Not character count: a
converter emitting 2.7x more characters can carry FEWER source strings, which
arm A measured directly.
- **pairing** -- of the source rows carrying a label and a value, how many keep
@ -42,9 +43,20 @@ from llm_ingestion_okf.extract import extract_text # noqa: E402
MIN_STRING = 3
_WS = re.compile(r"\s+")
_XML_TEXT = re.compile(r"<(?:w:t|t)(?:\s[^>]*)?>(.*?)</(?:w:t|t)>", re.DOTALL)
_XML_TEXT = re.compile(r"<(?:w:t|a:t|t)(?:\s[^>]*)?>(.*?)</(?:w:t|a:t|t)>", re.DOTALL)
_XML_PARA = re.compile(r"<w:p[\s>].*?</w:p>", re.DOTALL)
_XML_SI = re.compile(r"<si>(.*?)</si>", re.DOTALL)
#: A presentation's paragraph. A table cell holds one, so cells and body text
#: are read by the same rule rather than by two that could disagree.
_PPTX_PARA = re.compile(r"<a:p[\s>].*?</a:p>", re.DOTALL)
#: A slide part. Sorted NUMERICALLY on the trailing index, because the
#: lexicographic order of `slide2` and `slide10` is not the deck's order.
_PPTX_SLIDE = re.compile(r"^ppt/slides/slide(\d+)\.xml$")
#: An ODF paragraph or heading. A table cell holds a `text:p`, so the same rule
#: covers cells; the inner markup (`text:span` and friends) is stripped rather
#: than parsed, since the question is which STRING the file stores.
_ODF_PARA = re.compile(r"<text:(p|h)(?:\s[^>]*)?>(.*?)</text:\1>", re.DOTALL)
_TAGS = re.compile(r"<[^>]*>")
_ENTITIES = (("&amp;", "&"), ("&lt;", "<"), ("&gt;", ">"), ("&quot;", '"'), ("&apos;", "'"))
@ -70,6 +82,112 @@ def _runs(fragment: str) -> str:
return "".join(match.group(1) for match in _XML_TEXT.finditer(fragment))
# --- the rtf stream ---------------------------------------------------------
#
# RTF is not a package, so there is no XML part to read the document's own
# strings out of: the stream itself is the format. What follows reads exactly
# the constructs the fixture set uses -- groups, destinations, control words,
# the `\'hh` byte escape and the `\uN` unicode escape -- and it is stated as a
# limit rather than sold as an RTF parser. That limit matters for what the
# measurement can claim: for `pptx` and `odt` the fasit comes from a package
# format neither we nor the converter defined, and for `rtf` it comes from a
# reader written here against a stream written here. The `rtf` row is the
# weakest of the three on that axis and the report says so.
#: Control words that END a paragraph or a cell. Everything the document is
#: divided into for the purpose of "did this string survive".
_RTF_BOUNDARIES = frozenset({"par", "cell", "row", "line", "sect", "nestcell", "nestrow"})
#: Groups whose contents are the file's own bookkeeping rather than its text.
#: Reading them would count a font name as a document string.
_RTF_DESTINATIONS = frozenset(
{
"fonttbl",
"colortbl",
"stylesheet",
"info",
"pict",
"listtable",
"generator",
"filetbl",
}
)
#: A control word plus its optional numeric parameter. The trailing space is
#: the word's DELIMITER and is consumed with it -- that single rule is what
#: makes `\u248 ?` read as one character followed by a discardable `?`, and it
#: is the rule the vendored converter applies differently (see the fixture
#: README).
_RTF_CONTROL = re.compile(r"\\([a-z]+)(-?[0-9]+)? ?")
def rtf_paragraphs(raw: bytes) -> list[str]:
"""The strings an RTF stream stores, one per paragraph or table cell."""
text = raw.decode("cp1252", "replace")
end = len(text)
out: list[str] = []
buf: list[str] = []
position = 0
depth = 0
skip_depth: int | None = None
pending_replacements = 0
while position < end:
char = text[position]
if char == "{":
depth += 1
position += 1
continue
if char == "}":
if skip_depth is not None and depth <= skip_depth:
skip_depth = None
depth -= 1
position += 1
continue
if char == "\\":
control = _RTF_CONTROL.match(text, position)
if control is not None:
word, parameter = control.group(1), control.group(2)
position = control.end()
if skip_depth is not None:
continue
if word == "u" and parameter is not None:
buf.append(chr(int(parameter)))
pending_replacements = 1
continue
pending_replacements = 0
if word in _RTF_BOUNDARIES:
out.append("".join(buf))
buf = []
elif word in _RTF_DESTINATIONS:
skip_depth = depth
continue
symbol = text[position + 1] if position + 1 < end else ""
if symbol == "'":
if skip_depth is None:
buf.append(chr(int(text[position + 2 : position + 4] or "3f", 16)))
position += 4
continue
if symbol == "*":
skip_depth = depth
position += 2
continue
if skip_depth is None:
buf.append(symbol)
position += 2
continue
position += 1
if char in "\r\n" or skip_depth is not None:
continue
if pending_replacements:
pending_replacements -= 1
continue
buf.append(char)
out.append("".join(buf))
return out
def source_strings(path: Path) -> tuple[str, ...]:
"""The strings the FILE itself stores, deduplicated and in document order.
@ -78,6 +196,8 @@ def source_strings(path: Path) -> tuple[str, ...]:
homework.
"""
suffix = path.suffix.lower()
if suffix == ".rtf":
return _dedupe(rtf_paragraphs(path.read_bytes()))
with zipfile.ZipFile(path) as archive:
names = set(archive.namelist())
if suffix == ".docx":
@ -88,9 +208,26 @@ def source_strings(path: Path) -> tuple[str, ...]:
if "xl/sharedStrings.xml" in names:
shared = archive.read("xl/sharedStrings.xml").decode("utf-8", "replace")
found = [_runs(match.group(1)) for match in _XML_SI.finditer(shared)]
elif suffix == ".pptx":
found = []
slides = [
(int(match.group(1)), match.group(0))
for match in (_PPTX_SLIDE.match(name) for name in names)
if match is not None
]
for _, name in sorted(slides):
slide = archive.read(name).decode("utf-8", "replace")
found.extend(_runs(match.group(0)) for match in _PPTX_PARA.finditer(slide))
elif suffix == ".odt":
body = archive.read("content.xml").decode("utf-8", "replace")
found = [_TAGS.sub("", match.group(2)) for match in _ODF_PARA.finditer(body)]
else:
raise ValueError(f"no source-string reader for {suffix!r}")
return _dedupe(found)
def _dedupe(found: list[str]) -> tuple[str, ...]:
seen: dict[str, None] = {}
for item in found:
value = normalise(item)