test(fidelity): K2 on pptx, odt and rtf -- 59/59 and 20/20, on a set we wrote
`docs/2026-09-04-k2-pptx-odt-rtf.md` measured the corpus denominator for
these three rows at zero, so K2 for them was unmeasured rather than
passing. This measures them on the synthetic set committed in `a7b050b`,
whose hand count landed BEFORE any of these numbers existed.
Coverage 59/59 and pairing 20/20 for all three, standalone and product
alike, where "product" is scored off the persisted concept in the bundle
rather than an intermediate. The instrument's own denominator, derived
from each file's XML or stream, agrees with the hand count on all three:
59 strings, 20 pairable. Door B over the fixture directory persists 3/3
with nothing quarantined, rejected, failed or skipped, so K1b holds at
3 = N. There is no sub-100 % cell to explain, and that is a result about
three documents we wrote, not about the formats.
THE NEGATIVE CONTROL IS WHAT MAKES THOSE FIGURES MEAN ANYTHING. Removing
one label and its value from each converted text moves every row to
57/59 and 19/20 -- exactly the two strings and the one pair taken away.
`tests/test_fidelity.py` carries that control per reader, so it cannot
rot into a claim.
The order names `okf_fidelity` as the instrument and the instrument
refused all three types (`no source-string reader for '.pptx'`), so step
2 could not run without readers for them. Added here, stated in the
report as sitting outside the order's enumerated fence: this is the
measuring instrument, not extraction code. `extract.py` is untouched and
every `_EVIDENCE` row still reads `unmeasured` -- three synthetic
documents in one house style are exercise, not coverage, and the suite
asserts the rows are not promoted.
Two converter findings, reported and not fixed, both exit code 0 with no
warning. RTF cell paragraphs without `\pard\intbl` are read as each row
nested inside the previous one, 2076 characters where 117 were expected.
And the `\uN?` unicode escape -- THE FORM WORD EMITS -- loses the
character after it: `A\u248?BC` reads back as `AoC`. Any Word-produced
RTF carrying Norwegian letters loses one character after each of them,
silently. That is now a named limit of the `rtf` row rather than an
unknown.
Pandoc 3.9 and the host's 3.10.2 produce byte-identical output for all
three fixtures, so the numbers are not sitting on a version-fragile edge.
`~/corpora/` was neither read nor written; N there is still 43.
Suite 1141 passed, `ruff` clean, `mypy --strict src/ tools/` clean over
25 files, all after `git add`. Regenerating the fixtures leaves
`git diff --quiet` clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
a7b050b569
commit
bd44929c89
3 changed files with 443 additions and 4 deletions
186
docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md
Normal file
186
docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md
Normal file
|
|
@ -0,0 +1,186 @@
|
|||
# K2 for pptx/odt/rtf on a synthetic denominator, N = 3, 2026-09-07
|
||||
|
||||
Order `20260906T220349Z-652851369-from-.claude`. Closes the gap the previous
|
||||
report opened rather than filled: `docs/2026-09-04-k2-pptx-odt-rtf.md` measured
|
||||
the corpus denominator for these three office rows and found it **zero** —
|
||||
`~/corpora/okf-telling-20260829/K2/trinn1` holds 43 files and not one is a
|
||||
`pptx`, an `odt` or an `rtf`, with a known-positive control (`docx` → 5/5)
|
||||
proving the query could find. So K2 for those rows was *unmeasured*, which is a
|
||||
different fact from *passes*, and the corpus could not be made to say otherwise.
|
||||
|
||||
This report measures them on a **synthetic** denominator instead: three
|
||||
documents written here, with a hand count committed before the measurement ran.
|
||||
That is a weaker instrument than a corpus and the limits are stated in full at
|
||||
the bottom. It is not weaker than nothing, which is what the row had.
|
||||
|
||||
**Nothing in `~/corpora/` was read or written.** N there is still 43.
|
||||
|
||||
## What was measured, and against what
|
||||
|
||||
`tools/okf_fidelity.py` asks the source document's own two questions:
|
||||
|
||||
- **coverage** — of the strings the FILE stores, how many appear in the
|
||||
converted text. Read from the package's own XML (or, for RTF, from the
|
||||
stream), never from a converter's output: a fasit derived from one converter
|
||||
would score that converter on its own homework.
|
||||
- **pairing** — of the rows carrying a label and a value, how many keep both on
|
||||
one output line. That is the criterion a requirement table is read by.
|
||||
|
||||
The fixture set is `tests/fixtures/k2-office/`: **one authored document laid out
|
||||
three times, in three containers**, so the container and its reader are the only
|
||||
variable between the three measurements. A title, an intro, a 20-row
|
||||
label/value table, a caption, and a 4×4 grid. Norwegian throughout, with the
|
||||
diacritics the pipeline has to survive.
|
||||
|
||||
**The fasit was hand-counted from the authored content and committed first**, in
|
||||
`a7b050b`, before any of the numbers below existed:
|
||||
|
||||
| | count | how it was counted |
|
||||
|---|---|---|
|
||||
| cells | **56** | 20 rows × 2 columns + 4 rows × 4 columns |
|
||||
| pairs | **20** | the 20 label rows; every label ends in a colon |
|
||||
| distinct strings | **59** | title + intro + caption + 56 cells, all distinct after normalisation |
|
||||
|
||||
The instrument derives its own denominator from each file's XML or stream,
|
||||
independently of that hand count. **The two agree on all three documents:
|
||||
59 strings, 20 pairable.** A disagreement would have been the report; there
|
||||
isn't one.
|
||||
|
||||
## The result
|
||||
|
||||
Converter: the **vendored pandoc 3.9**, resolved by path and version-asserted by
|
||||
`src/llm_ingestion_okf/_pandoc.py`, with `-t markdown --eol=lf --wrap=none`.
|
||||
"Product" is the text scored off the **persisted concept in the bundle** — the
|
||||
artifact Door B actually writes, not an intermediate.
|
||||
|
||||
| format | coverage | paired | denominator |
|
||||
|---|---|---|---|
|
||||
| `pptx` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
|
||||
| `odt` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
|
||||
| `rtf` | **59/59 (100 %)** | **20/20 (100 %)** | 59 strings, 20 pairs |
|
||||
|
||||
Standalone (converter alone) and product (through `extract_text`, and through
|
||||
the written concept) are equal for all three — no fall from one to the other, so
|
||||
there is nothing here to attribute to this library.
|
||||
|
||||
**There is no sub-100 % cell to explain.** The order asks that every fall below
|
||||
100 % be named with a cause; there are none, and that absence is a result about
|
||||
a document set we wrote, not about the format.
|
||||
|
||||
**Door B, product path, N = 3:** `process_inbox` over the fixture directory
|
||||
persisted **3/3**, with 0 quarantined, 0 rejected, 0 failed and 0 skipped.
|
||||
**K1b: 3 + 0 = 3 = N.**
|
||||
|
||||
### The negative control
|
||||
|
||||
100 % means nothing from an instrument that cannot see a loss. Removing one
|
||||
label and its value from each converted text, and re-scoring the same shipped
|
||||
fixtures:
|
||||
|
||||
| format | coverage | paired |
|
||||
|---|---|---|
|
||||
| `pptx` | 57/59 (96.6 %) | 19/20 (95.0 %) |
|
||||
| `odt` | 57/59 (96.6 %) | 19/20 (95.0 %) |
|
||||
| `rtf` | 57/59 (96.6 %) | 19/20 (95.0 %) |
|
||||
|
||||
The instrument moves by exactly the two strings and the one pair that were
|
||||
taken away. `tests/test_fidelity.py` carries the same control per reader as a
|
||||
test, so it cannot rot.
|
||||
|
||||
### Converter version
|
||||
|
||||
The order asks for a comparison against the host's newer converter if the
|
||||
numbers fell. They did not, and the comparison was run anyway because it is
|
||||
cheap and it bounds a different risk: **pandoc 3.9 and the host's pandoc 3.10.2
|
||||
produce byte-identical output for all three fixtures** (`cmp`, three files,
|
||||
no differences). These numbers are therefore not sitting on a version-fragile
|
||||
edge.
|
||||
|
||||
## Two converter findings, neither of them fixed
|
||||
|
||||
Both were measured while laying the RTF out, both are structurally plausible
|
||||
input read silently wrong, and both come back **exit code 0 with no warning** —
|
||||
the same shape as the missing `word/styles.xml` and the `inlineStr` xlsx already
|
||||
recorded in `tests/fixtures/README.md`. Per the order they are reported, not
|
||||
patched; nothing in `src/` works around either.
|
||||
|
||||
**1. RTF cell paragraphs need `\pard\intbl`.** Without it, consecutive
|
||||
`\trowd…\row` rows are read as each row NESTED inside the previous one. Five
|
||||
label/value rows came back as five levels of nested table — **2076 characters
|
||||
where 117 were expected**. The fixture writes `\intbl`, which is correct RTF; a
|
||||
document that omits it is not this library's problem to fix, but it is a shape a
|
||||
real drop directory can contain.
|
||||
|
||||
**2. The `\uN?` unicode escape loses the character after it.** This is the form
|
||||
**Word emits**, so it is the common case rather than an edge one. Measured
|
||||
directly against the vendored 3.9:
|
||||
|
||||
```
|
||||
A\u248?BC -> AøC the ring letter arrives, the B is GONE
|
||||
A\u248?xBC -> AøBC the same rule seen from the other side
|
||||
A\u248 ?BC -> AøBC an explicit delimiter, and the text survives
|
||||
```
|
||||
|
||||
The `?` is taken as the control word's delimiter, and `\uc1` then skips a *real*
|
||||
character. `\uc0` does not change it. The fixture is written in the form that
|
||||
round-trips (`\uN ?`), so this finding is recorded here rather than baked
|
||||
invisibly into a fixture.
|
||||
|
||||
**What this means for a real `rtf` drop.** Any Word-produced RTF containing
|
||||
Norwegian letters will lose one character after each of them, silently. That is
|
||||
a defect in the converter, not in this library, and it is now a named limit of
|
||||
the `rtf` row rather than an unknown.
|
||||
|
||||
## Honesty limits
|
||||
|
||||
Read these as part of the table, not as a footnote to it.
|
||||
|
||||
- **Synthetic, N = 3, one house style.** Three documents we wrote are not a
|
||||
corpus. They are internally consistent by construction: one author, one
|
||||
layout convention, one set of table shapes. A real `pptx` deck with text
|
||||
boxes over images, an `odt` with tracked changes, or an RTF from a different
|
||||
producer are all outside what was measured.
|
||||
- **We authored the fixtures to be readable, and they read.** The 100 % says
|
||||
the path works end to end on documents built to exercise it. It does not say
|
||||
the format is covered. `extract._EVIDENCE` therefore still reads
|
||||
`unmeasured` for all three rows, and `tests/test_k2_office_fixtures.py`
|
||||
asserts that it does — so a later reader cannot mistake this report for
|
||||
promotion.
|
||||
- **The `rtf` row is the weakest of the three.** For `pptx` and `odt` the fasit
|
||||
is read out of a package format that neither we nor the converter defined.
|
||||
For `rtf` there is no part to read, so the fasit comes from a reader written
|
||||
here against a stream written here. Both halves are ours, and the
|
||||
independence the other two rows have is missing.
|
||||
- **Drawn content is out of scope entirely.** Conversion recovers text; a
|
||||
figure, a diagram or a drawn shape has no text to recover, only a caption.
|
||||
Every office extraction already warns about this on every call. The fixtures
|
||||
contain no drawn content, so this report measures nothing about it.
|
||||
- **`pptx` speaker notes, `odt` footnotes and RTF headers/footers** carry text
|
||||
in real documents and appear in none of these fixtures. Unmeasured, and named
|
||||
here so the gap has a denominator of its own.
|
||||
|
||||
## What changed in the tree
|
||||
|
||||
- `tests/fixtures/k2-office/` — the three documents, hand-laid by
|
||||
`tests/fixtures/make_k2_office.py`. **No converter wrote any of them**: a file
|
||||
written by the converter and read by the converter proves only that the
|
||||
converter agrees with itself. The commissioning order offered pandoc as one
|
||||
generator option; the committed fixture policy in `tests/fixtures/README.md`
|
||||
forbids it, and the policy wins.
|
||||
- `tests/fixtures/k2-office-fasit.json` — the hand count, committed first.
|
||||
- `tests/test_k2_office_fixtures.py` — pins N = 3, 3/3 through Door B offline,
|
||||
the concept contents, the fasit, and that the rows stay `unmeasured`.
|
||||
- `tools/okf_fidelity.py` — **source-string readers for `pptx`, `odt` and
|
||||
`rtf`.** Stated plainly because it sits outside the order's enumerated fence:
|
||||
the instrument the order names refused all three types
|
||||
(`ValueError: no source-string reader for '.pptx'`), so step 2 could not run
|
||||
without it. It is the measuring instrument, not extraction code —
|
||||
`src/llm_ingestion_okf/extract.py` is untouched, and so is every `_EVIDENCE`
|
||||
row.
|
||||
|
||||
**Suite: 1141 passed** (1127 before this work, 1132 after the fixture commit,
|
||||
1141 with the reader tests). `ruff check`, `ruff format --check` and
|
||||
`mypy --strict src/ tools/` (25 files) clean, all measured after `git add`.
|
||||
Re-running `make_k2_office.py` leaves `git diff --quiet` clean on
|
||||
`tests/fixtures/k2-office`, so the three documents are reproducible from that
|
||||
one file. Nothing pushed.
|
||||
|
|
@ -17,6 +17,7 @@ from __future__ import annotations
|
|||
|
||||
import sys
|
||||
import zipfile
|
||||
from collections.abc import Callable
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
|
@ -113,9 +114,124 @@ def test_a_workbooks_shared_strings_are_read(tmp_path: Path) -> None:
|
|||
|
||||
|
||||
def test_a_type_with_no_reader_is_refused_rather_than_scored_zero(tmp_path: Path) -> None:
|
||||
"""Zero coverage and "no instrument" are different facts."""
|
||||
path = tmp_path / "x.pptx"
|
||||
"""Zero coverage and "no instrument" are different facts.
|
||||
|
||||
The type here USED to be `.pptx`, which now has a reader. Replaced in place
|
||||
rather than deleted: the property is about the refusal, not about which
|
||||
suffix happens to lack a reader this week, and dropping the test with the
|
||||
reader would have removed the guarantee along with its example.
|
||||
"""
|
||||
path = tmp_path / "x.epub"
|
||||
with zipfile.ZipFile(path, "w") as archive:
|
||||
archive.writestr("a", "b")
|
||||
with pytest.raises(ValueError):
|
||||
okf_fidelity.source_strings(path)
|
||||
|
||||
|
||||
# --- the three office rows the corpus has none of --------------------------
|
||||
#
|
||||
# `pptx`, `odt` and `rtf` had no source-string reader here, so the instrument
|
||||
# refused them and K2 for those rows could not be run at all -- see
|
||||
# `docs/2026-09-07-k2-pptx-odt-rtf-fixtures.md`. Each reader is checked the
|
||||
# same way as the two above: it finds the document's own strings, and it is
|
||||
# able to see a loss.
|
||||
|
||||
SLIDE_XML = """<?xml version="1.0"?>
|
||||
<p:sld xmlns:a="a" xmlns:p="p"><p:cSld><p:spTree>
|
||||
<p:sp><p:txBody><a:p><a:r><a:t>Kravspesifikasjon for tunnel</a:t></a:r></a:p></p:txBody></p:sp>
|
||||
<p:graphicFrame><a:tbl><a:tr>
|
||||
<a:tc><a:txBody><a:p><a:r><a:t>Navn tilbyder:</a:t></a:r></a:p></a:txBody></a:tc>
|
||||
<a:tc><a:txBody><a:p><a:r><a:t>Entreprenoer AS</a:t></a:r></a:p></a:txBody></a:tc>
|
||||
</a:tr></a:tbl></p:graphicFrame>
|
||||
</p:spTree></p:cSld></p:sld>
|
||||
"""
|
||||
|
||||
CONTENT_XML = """<?xml version="1.0"?>
|
||||
<office:document-content xmlns:office="o" xmlns:text="t" xmlns:table="tb">
|
||||
<office:body><office:text>
|
||||
<text:h text:outline-level="1">Kravspesifikasjon for tunnel</text:h>
|
||||
<table:table><table:table-row>
|
||||
<table:table-cell><text:p>Navn <text:span>tilbyder</text:span>:</text:p></table:table-cell>
|
||||
<table:table-cell><text:p>Entreprenoer AS</text:p></table:table-cell>
|
||||
</table:table-row></table:table>
|
||||
</office:text></office:body></office:document-content>
|
||||
"""
|
||||
|
||||
RTF_STREAM = (
|
||||
"{\\rtf1\\ansi\\deff0{\\fonttbl{\\f0\\froman Times New Roman;}}\n"
|
||||
"\\pard Kravspesifikasjon for tunnel\\par\n"
|
||||
"\\trowd\\cellx3000\\cellx6000\n"
|
||||
"\\pard\\intbl Navn tilbyder:\\cell \\pard\\intbl Entrepren\\u248 ?r AS\\cell \\row\n"
|
||||
"}"
|
||||
)
|
||||
|
||||
|
||||
def pptx(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "p.pptx"
|
||||
with zipfile.ZipFile(path, "w") as archive:
|
||||
archive.writestr("ppt/slides/slide1.xml", SLIDE_XML)
|
||||
return path
|
||||
|
||||
|
||||
def odt(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "o.odt"
|
||||
with zipfile.ZipFile(path, "w") as archive:
|
||||
archive.writestr("content.xml", CONTENT_XML)
|
||||
return path
|
||||
|
||||
|
||||
def rtf(tmp_path: Path) -> Path:
|
||||
path = tmp_path / "r.rtf"
|
||||
path.write_text(RTF_STREAM, encoding="ascii")
|
||||
return path
|
||||
|
||||
|
||||
def test_a_presentations_slide_text_is_read(tmp_path: Path) -> None:
|
||||
"""One string per `a:p`, table cells included -- a cell is a paragraph."""
|
||||
assert set(okf_fidelity.source_strings(pptx(tmp_path))) == {
|
||||
"kravspesifikasjon for tunnel",
|
||||
"navn tilbyder:",
|
||||
"entreprenoer as",
|
||||
}
|
||||
|
||||
|
||||
def test_an_odf_bodys_paragraphs_are_read(tmp_path: Path) -> None:
|
||||
"""Headings count, and a run split across a `text:span` is still one string."""
|
||||
assert set(okf_fidelity.source_strings(odt(tmp_path))) == {
|
||||
"kravspesifikasjon for tunnel",
|
||||
"navn tilbyder:",
|
||||
"entreprenoer as",
|
||||
}
|
||||
|
||||
|
||||
def test_an_rtf_streams_paragraphs_and_cells_are_read(tmp_path: Path) -> None:
|
||||
"""The control words are markup; the text between them is the document."""
|
||||
assert set(okf_fidelity.source_strings(rtf(tmp_path))) == {
|
||||
"kravspesifikasjon for tunnel",
|
||||
"navn tilbyder:",
|
||||
"entreprenør as",
|
||||
}
|
||||
|
||||
|
||||
@pytest.mark.parametrize("build", [pptx, odt, rtf])
|
||||
def test_each_new_reader_can_see_a_loss(build: Callable[[Path], Path], tmp_path: Path) -> None:
|
||||
"""The negative control, once per reader.
|
||||
|
||||
A reader that returns its strings and an instrument that always says yes
|
||||
are indistinguishable from a green number, so every reader added here owes
|
||||
the same proof as the first two.
|
||||
"""
|
||||
path = build(tmp_path)
|
||||
strings = list(okf_fidelity.source_strings(path))
|
||||
result = okf_fidelity.score(path, "\n".join(strings[:-1]))
|
||||
assert result.covered == len(strings) - 1 < result.total
|
||||
|
||||
|
||||
@pytest.mark.parametrize("build", [pptx, odt, rtf])
|
||||
def test_each_new_reader_pairs_a_label_only_when_a_value_sits_beside_it(
|
||||
build: Callable[[Path], Path], tmp_path: Path
|
||||
) -> None:
|
||||
path = build(tmp_path)
|
||||
label = next(item for item in okf_fidelity.source_strings(path) if item.endswith(":"))
|
||||
assert okf_fidelity.score(path, label).paired == 0
|
||||
assert okf_fidelity.score(path, f"{label} en verdi").paired == 1
|
||||
|
|
|
|||
|
|
@ -8,7 +8,8 @@ converter version, against the product path, or at all.
|
|||
Two questions, both of them the source document's own:
|
||||
|
||||
- **coverage** -- of the strings the file itself stores (a `docx` paragraph, an
|
||||
`xlsx` cell), how many appear in the converted text. Not character count: a
|
||||
`xlsx` cell, a `pptx` or `odt` paragraph, an `rtf` cell), how many appear in
|
||||
the converted text. Not character count: a
|
||||
converter emitting 2.7x more characters can carry FEWER source strings, which
|
||||
arm A measured directly.
|
||||
- **pairing** -- of the source rows carrying a label and a value, how many keep
|
||||
|
|
@ -42,9 +43,20 @@ from llm_ingestion_okf.extract import extract_text # noqa: E402
|
|||
MIN_STRING = 3
|
||||
|
||||
_WS = re.compile(r"\s+")
|
||||
_XML_TEXT = re.compile(r"<(?:w:t|t)(?:\s[^>]*)?>(.*?)</(?:w:t|t)>", re.DOTALL)
|
||||
_XML_TEXT = re.compile(r"<(?:w:t|a:t|t)(?:\s[^>]*)?>(.*?)</(?:w:t|a:t|t)>", re.DOTALL)
|
||||
_XML_PARA = re.compile(r"<w:p[\s>].*?</w:p>", re.DOTALL)
|
||||
_XML_SI = re.compile(r"<si>(.*?)</si>", re.DOTALL)
|
||||
#: A presentation's paragraph. A table cell holds one, so cells and body text
|
||||
#: are read by the same rule rather than by two that could disagree.
|
||||
_PPTX_PARA = re.compile(r"<a:p[\s>].*?</a:p>", re.DOTALL)
|
||||
#: A slide part. Sorted NUMERICALLY on the trailing index, because the
|
||||
#: lexicographic order of `slide2` and `slide10` is not the deck's order.
|
||||
_PPTX_SLIDE = re.compile(r"^ppt/slides/slide(\d+)\.xml$")
|
||||
#: An ODF paragraph or heading. A table cell holds a `text:p`, so the same rule
|
||||
#: covers cells; the inner markup (`text:span` and friends) is stripped rather
|
||||
#: than parsed, since the question is which STRING the file stores.
|
||||
_ODF_PARA = re.compile(r"<text:(p|h)(?:\s[^>]*)?>(.*?)</text:\1>", re.DOTALL)
|
||||
_TAGS = re.compile(r"<[^>]*>")
|
||||
_ENTITIES = (("&", "&"), ("<", "<"), (">", ">"), (""", '"'), ("'", "'"))
|
||||
|
||||
|
||||
|
|
@ -70,6 +82,112 @@ def _runs(fragment: str) -> str:
|
|||
return "".join(match.group(1) for match in _XML_TEXT.finditer(fragment))
|
||||
|
||||
|
||||
# --- the rtf stream ---------------------------------------------------------
|
||||
#
|
||||
# RTF is not a package, so there is no XML part to read the document's own
|
||||
# strings out of: the stream itself is the format. What follows reads exactly
|
||||
# the constructs the fixture set uses -- groups, destinations, control words,
|
||||
# the `\'hh` byte escape and the `\uN` unicode escape -- and it is stated as a
|
||||
# limit rather than sold as an RTF parser. That limit matters for what the
|
||||
# measurement can claim: for `pptx` and `odt` the fasit comes from a package
|
||||
# format neither we nor the converter defined, and for `rtf` it comes from a
|
||||
# reader written here against a stream written here. The `rtf` row is the
|
||||
# weakest of the three on that axis and the report says so.
|
||||
|
||||
#: Control words that END a paragraph or a cell. Everything the document is
|
||||
#: divided into for the purpose of "did this string survive".
|
||||
_RTF_BOUNDARIES = frozenset({"par", "cell", "row", "line", "sect", "nestcell", "nestrow"})
|
||||
|
||||
#: Groups whose contents are the file's own bookkeeping rather than its text.
|
||||
#: Reading them would count a font name as a document string.
|
||||
_RTF_DESTINATIONS = frozenset(
|
||||
{
|
||||
"fonttbl",
|
||||
"colortbl",
|
||||
"stylesheet",
|
||||
"info",
|
||||
"pict",
|
||||
"listtable",
|
||||
"generator",
|
||||
"filetbl",
|
||||
}
|
||||
)
|
||||
|
||||
#: A control word plus its optional numeric parameter. The trailing space is
|
||||
#: the word's DELIMITER and is consumed with it -- that single rule is what
|
||||
#: makes `\u248 ?` read as one character followed by a discardable `?`, and it
|
||||
#: is the rule the vendored converter applies differently (see the fixture
|
||||
#: README).
|
||||
_RTF_CONTROL = re.compile(r"\\([a-z]+)(-?[0-9]+)? ?")
|
||||
|
||||
|
||||
def rtf_paragraphs(raw: bytes) -> list[str]:
|
||||
"""The strings an RTF stream stores, one per paragraph or table cell."""
|
||||
text = raw.decode("cp1252", "replace")
|
||||
end = len(text)
|
||||
out: list[str] = []
|
||||
buf: list[str] = []
|
||||
position = 0
|
||||
depth = 0
|
||||
skip_depth: int | None = None
|
||||
pending_replacements = 0
|
||||
|
||||
while position < end:
|
||||
char = text[position]
|
||||
if char == "{":
|
||||
depth += 1
|
||||
position += 1
|
||||
continue
|
||||
if char == "}":
|
||||
if skip_depth is not None and depth <= skip_depth:
|
||||
skip_depth = None
|
||||
depth -= 1
|
||||
position += 1
|
||||
continue
|
||||
if char == "\\":
|
||||
control = _RTF_CONTROL.match(text, position)
|
||||
if control is not None:
|
||||
word, parameter = control.group(1), control.group(2)
|
||||
position = control.end()
|
||||
if skip_depth is not None:
|
||||
continue
|
||||
if word == "u" and parameter is not None:
|
||||
buf.append(chr(int(parameter)))
|
||||
pending_replacements = 1
|
||||
continue
|
||||
pending_replacements = 0
|
||||
if word in _RTF_BOUNDARIES:
|
||||
out.append("".join(buf))
|
||||
buf = []
|
||||
elif word in _RTF_DESTINATIONS:
|
||||
skip_depth = depth
|
||||
continue
|
||||
symbol = text[position + 1] if position + 1 < end else ""
|
||||
if symbol == "'":
|
||||
if skip_depth is None:
|
||||
buf.append(chr(int(text[position + 2 : position + 4] or "3f", 16)))
|
||||
position += 4
|
||||
continue
|
||||
if symbol == "*":
|
||||
skip_depth = depth
|
||||
position += 2
|
||||
continue
|
||||
if skip_depth is None:
|
||||
buf.append(symbol)
|
||||
position += 2
|
||||
continue
|
||||
position += 1
|
||||
if char in "\r\n" or skip_depth is not None:
|
||||
continue
|
||||
if pending_replacements:
|
||||
pending_replacements -= 1
|
||||
continue
|
||||
buf.append(char)
|
||||
|
||||
out.append("".join(buf))
|
||||
return out
|
||||
|
||||
|
||||
def source_strings(path: Path) -> tuple[str, ...]:
|
||||
"""The strings the FILE itself stores, deduplicated and in document order.
|
||||
|
||||
|
|
@ -78,6 +196,8 @@ def source_strings(path: Path) -> tuple[str, ...]:
|
|||
homework.
|
||||
"""
|
||||
suffix = path.suffix.lower()
|
||||
if suffix == ".rtf":
|
||||
return _dedupe(rtf_paragraphs(path.read_bytes()))
|
||||
with zipfile.ZipFile(path) as archive:
|
||||
names = set(archive.namelist())
|
||||
if suffix == ".docx":
|
||||
|
|
@ -88,9 +208,26 @@ def source_strings(path: Path) -> tuple[str, ...]:
|
|||
if "xl/sharedStrings.xml" in names:
|
||||
shared = archive.read("xl/sharedStrings.xml").decode("utf-8", "replace")
|
||||
found = [_runs(match.group(1)) for match in _XML_SI.finditer(shared)]
|
||||
elif suffix == ".pptx":
|
||||
found = []
|
||||
slides = [
|
||||
(int(match.group(1)), match.group(0))
|
||||
for match in (_PPTX_SLIDE.match(name) for name in names)
|
||||
if match is not None
|
||||
]
|
||||
for _, name in sorted(slides):
|
||||
slide = archive.read(name).decode("utf-8", "replace")
|
||||
found.extend(_runs(match.group(0)) for match in _PPTX_PARA.finditer(slide))
|
||||
elif suffix == ".odt":
|
||||
body = archive.read("content.xml").decode("utf-8", "replace")
|
||||
found = [_TAGS.sub("", match.group(2)) for match in _ODF_PARA.finditer(body)]
|
||||
else:
|
||||
raise ValueError(f"no source-string reader for {suffix!r}")
|
||||
|
||||
return _dedupe(found)
|
||||
|
||||
|
||||
def _dedupe(found: list[str]) -> tuple[str, ...]:
|
||||
seen: dict[str, None] = {}
|
||||
for item in found:
|
||||
value = normalise(item)
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue