docs(readme): a visible table for all 13 file types, pinned to the registry

The extractor registry reads 13 extensions. The README's opening line named
five of them, and the full list existed only in a hidden
`<!-- extract-formats: ... -->` comment, which no reader reads -- so the README
undersold what the code does and stated no evidence class anywhere a consumer
would look.

A `## Supported file types` table now carries one row per extension: reader,
dependency (core or the `[extract]` extra), the evidence class `_EVIDENCE`
records for the row, and one honest note. The three `constructed` office rows
carry their denominators (N = 1, N = 2, N = 1) in the table itself, so a row
that has met no document anyone wrote cannot read as a supported one; `.htm`
does not borrow `.html`'s 828-file class, because the code records none for it.
A `Not read today` section states the absences (`.doc`, `.epub`, `.eml`/`.msg`,
image files, source files, `.one`/`.vsd`) as facts, not as a queue.

Test first, red before the table existed: four assertions in
`tests/test_docs_promises.py` pin the table's row set to
`_CORE_EXTRACTORS | _OPTIONAL_EXTRACTORS`, each evidence cell to `_EVIDENCE`
(and to a fixed `stdlib, no corpus class` where the code records none), the
core/extra split to the registries, and the opening to the table.

No change to `extract.py` and no version bump: nothing about what is read
moved, only what the README says about it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 21:26:15 +02:00
commit 2d3c2707c7
3 changed files with 151 additions and 3 deletions

View file

@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## [Unreleased] ## [Unreleased]
### Changed
- **The README states every file type the extractor registry reads (K3-26).**
It read 13 extensions (`.md`, `.txt`, `.csv`, `.json`, `.html`, `.htm`,
`.xml`, `.pdf`, `.docx`, `.xlsx`, `.pptx`, `.odt`, `.rtf`) while the opening
line named five of them and the full list existed only in a hidden
`<!-- extract-formats: ... -->` comment, which no reader reads. A visible
`## Supported file types` table now carries one row per extension with its
reader, its dependency (core or the `[extract]` extra), the evidence class
the code records for it, and one honest note -- so a `constructed` row at
N = 1 cannot read as a supported one. A `Not read today` section states what
is absent (`.doc`, `.epub`, `.eml`/`.msg`, image files, source files,
`.one`/`.vsd`) as facts rather than as a queue. Four new assertions in
`tests/test_docs_promises.py` pin the table's row set, its evidence cells and
its core/extra split to the registry, and the opening to the table. No change
to `extract.py`: nothing about what is read moved, only what the README says
about it.
## [0.8.5] — 2026-09-12 ## [0.8.5] — 2026-09-12
### Added ### Added

View file

@ -1,8 +1,9 @@
# llm-ingestion-okf # llm-ingestion-okf
Turn a folder of documents (PDF, DOCX, XLSX, PPTX, MD) into a bundle a model can Turn a folder of documents into a bundle a model can answer from **with a
answer from **with a source on every claim** — offline, deterministic, no model source on every claim** — offline, deterministic, no model call anywhere in the
call anywhere in the run path. run path. Thirteen file types are read; [Supported file
types](#supported-file-types) lists each one with the evidence behind it.
## Install ## Install
@ -53,6 +54,56 @@ rows are **constructed** rather than measured — see
[Binary extraction](#binary-extraction). Phase 4 [Binary extraction](#binary-extraction). Phase 4
(the Node half) is planned (see `docs/plan/`). (the Node half) is planned (see `docs/plan/`).
## Supported file types
Thirteen extensions are read. The table below is pinned to the extraction
registry by `tests/test_docs_promises.py` — row for row, and cell for cell on
the evidence class the code records — so a type added without a row here turns
that test red.
| File type | Read by | Dependency | Evidence | Note |
|---|---|---|---|---|
| `.md` | `_extract_passthrough` | core | stdlib, no corpus class | The decoded bytes are the concept text, so `source_lines` are the original's own lines. |
| `.txt` | `_extract_passthrough` | core | stdlib, no corpus class | As `.md`. A document with no headings yields no segments, which is a failed build rather than a flat bundle. |
| `.csv` | `_extract_csv` | core | stdlib, no corpus class | Parsed with the stdlib reader and rendered as one markdown table; a file with no header row is refused. |
| `.json` | `_extract_json` | core | stdlib, no corpus class | Fenced verbatim. No structure is derived from the keys. |
| `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1``h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is 828 files — one product, one format, one publisher. |
| `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The 828-file class is recorded for `.html` alone, and this row does not borrow it. |
| `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`<standard>` root, or any `<sec>`) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `<!DOCTYPE` is refused unparsed. The denominator is one file, one publisher, one schema — 2 761 titled sections. |
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a 701-page process code whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
| `.docx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Five corpus documents. `source_lines` index the extracted text and not the original's paragraphs — the two counts agree on none of the five. |
| `.xlsx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Written as pipe tables, one source row per line; a sheet name becomes a heading, and a row is located by `source_sheet` and `source_rows`. |
| `.pptx` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 2 decks. 2 of 2 slide titles recovered on a deck that declares them, 0 of 2 on a deck that does not, where the converter writes `Slide 1` / `Slide 2` because it has no title to use. |
| `.odt` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document. 1 of 1 declared headings recovered, 1 concept, 0 characters in no segment. |
| `.rtf` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document, and the weakest row here. 0 declared headings: the container carries no heading style, so the author's title is bold text and the document lands as one concept — content preserved, structure zero. The `--bold-title` flag reads that bold line and is off by default. |
The three classes are the code's own and are not interchangeable. `measured`
means real documents someone wrote for their own purposes, counted against a
fasit written before the lookup. `constructed` means the row has been through
end to end on a hand-built document with a hand-written fasit and has met no
document anyone else wrote. `stdlib, no corpus class` means the code records no
class for the row at all. The office rows are set out in full under [Binary
extraction](#binary-extraction).
### Not read today
Facts about the registry as it stands, not a queue — nothing here is planned.
- `.doc` (Word 97) — the converter does not read it.
- `.epub` — the converter does read it, and the row is deliberately absent: it
would buy nothing today, the same half of the reason that keeps HTML out of
the converter.
- `.eml` and `.msg` — no reader. An email file arrives as an unregistered
extension.
- Image files — no reader, and OCR is a PDF page mode rather than an entry for
them.
- Source files (`.py`, `.ts`, and the rest) — no reader. The text rows are
`.md`, `.txt`, `.csv` and `.json`.
- `.one` (OneNote) and `.vsd` (Visio) — no reader.
An unregistered extension is a coded rejection (`extractor_unknown`), never a
silent skip.
## Install in detail ## Install in detail
Neither this package nor the guard it depends on is on a package index yet, so Neither this package nor the guard it depends on is on a package index yet, so

View file

@ -114,3 +114,82 @@ def test_the_readme_recursion_claim_matches_the_door() -> None:
gate=lambda body: GateDecision(sanitized_text=body, disposition="warn", reasons=()), gate=lambda body: GateDecision(sanitized_text=body, disposition="warn", reasons=()),
) )
assert [item.source_file for item in result.persisted] == ["sub/deep.md"] assert [item.source_file for item in result.persisted] == ["sub/deep.md"]
# --- the visible table, K3-26 ----------------------------------------------
#
# The comment marker above is machine-readable and invisible to a reader: the
# README's own prose named FIVE of the thirteen types the registry reads, and
# nothing went red, because the marker test only asks that the hidden list is
# complete. A reader does not read the marker. So the table a reader does see
# is pinned to the same registry, row for row, and to the evidence class the
# code records for each row.
_TABLE_HEADING = "## Supported file types"
# The evidence cell for a row `_EVIDENCE` does not carry. Those five are the
# stdlib rows: `_EVIDENCE` records a CORPUS class, and a row that has never
# been given one must not borrow `measured` from the row beside it.
_NO_CLASS = "stdlib, no corpus class"
def _table_rows() -> dict[str, list[str]]:
"""The table's data rows, keyed by suffix, with markup stripped per cell.
Backticks and asterisks are removed rather than matched, so the table can
be formatted freely and this test still reads what it says.
"""
text = README.read_text(encoding="utf-8")
assert _TABLE_HEADING in text, (
f"README.md carries no `{_TABLE_HEADING}` section; the format list is "
"then visible only in a hidden comment, which is what K3-26 fixed"
)
rows: dict[str, list[str]] = {}
for line in text.split(_TABLE_HEADING, 1)[1].splitlines():
stripped = line.strip()
if not stripped.startswith("|"):
if rows:
break
continue
cells = [re.sub(r"[`*]", "", cell).strip() for cell in stripped.strip("|").split("|")]
if cells and cells[0].startswith("."):
rows[cells[0]] = cells
return rows
def test_the_readme_table_names_every_type_the_registry_reads() -> None:
assert set(_table_rows()) == set(_CORE_EXTRACTORS) | set(_OPTIONAL_EXTRACTORS)
def test_the_readme_table_states_the_evidence_class_the_code_records() -> None:
rows = _table_rows()
assert rows, "no data rows found under the supported-file-types heading"
for suffix, cells in rows.items():
assert len(cells) >= 4, f"the {suffix} row has no evidence column: {cells}"
assert cells[3] == _EVIDENCE.get(suffix, _NO_CLASS), (
f"the {suffix} row says {cells[3]!r}; the code records "
f"{_EVIDENCE.get(suffix, _NO_CLASS)!r}"
)
def test_the_readme_table_separates_core_from_the_extract_extra() -> None:
"""Which rows need the optional extra is the first thing a consumer asks."""
for suffix, cells in _table_rows().items():
gated = "[extract]" in cells[2]
assert gated == (suffix in _OPTIONAL_EXTRACTORS), (
f"the {suffix} row's dependency cell reads {cells[2]!r}"
)
def test_the_readme_opening_does_not_name_five_of_thirteen() -> None:
"""The first thing a reader sees must not undersell what the code reads.
Either form passes: a pointer to the table, or the whole set spelled out.
A partial list -- the state before K3-26 -- passes neither.
"""
intro = README.read_text(encoding="utf-8").split("## Install", 1)[0]
if "#supported-file-types" in intro:
return
every = set(_CORE_EXTRACTORS) | set(_OPTIONAL_EXTRACTORS)
named = {s for s in every if re.search(rf"\b{s.lstrip('.')}\b", intro, re.IGNORECASE)}
assert named == every, f"the opening names {sorted(named)}, not all of {sorted(every)}"