open/llm-ingestion-okf
1
0
Fork 0
Code Issues Releases 1 Activity

docs(readme): a visible table for all 13 file types, pinned to the registry

Browse source 
The extractor registry reads 13 extensions. The README's opening line named
five of them, and the full list existed only in a hidden
`<!-- extract-formats: ... -->` comment, which no reader reads -- so the README
undersold what the code does and stated no evidence class anywhere a consumer
would look.

A `## Supported file types` table now carries one row per extension: reader,
dependency (core or the `[extract]` extra), the evidence class `_EVIDENCE`
records for the row, and one honest note. The three `constructed` office rows
carry their denominators (N = 1, N = 2, N = 1) in the table itself, so a row
that has met no document anyone wrote cannot read as a supported one; `.htm`
does not borrow `.html`'s 828-file class, because the code records none for it.
A `Not read today` section states the absences (`.doc`, `.epub`, `.eml`/`.msg`,
image files, source files, `.one`/`.vsd`) as facts, not as a queue.

Test first, red before the table existed: four assertions in
`tests/test_docs_promises.py` pin the table's row set to
`_CORE_EXTRACTORS | _OPTIONAL_EXTRACTORS`, each evidence cell to `_EVIDENCE`
(and to a fixed `stdlib, no corpus class` where the code records none), the
core/extra split to the registries, and the opening to the table.

No change to `extract.py` and no version bump: nothing about what is read
moved, only what the README says about it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
Kjell Tore Guttormsen 2026-09-12 21:26:15 +02:00
commit 2d3c2707c7
3 changed files with 151 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

57
README.md
View file

@ -1,8 +1,9 @@
# llm-ingestion-okf
Turn a folder of documents (PDF, DOCX, XLSX, PPTX, MD) into a bundle a model can
answer from **with a source on every claim** — offline, deterministic, no model
call anywhere in the run path.
Turn a folder of documents into a bundle a model can answer from **with a
source on every claim** — offline, deterministic, no model call anywhere in the
run path. Thirteen file types are read; [Supported file
types](#supported-file-types) lists each one with the evidence behind it.
## Install
@ -53,6 +54,56 @@ rows are **constructed** rather than measured — see
[Binary extraction](#binary-extraction). Phase 4
(the Node half) is planned (see `docs/plan/`).
## Supported file types
Thirteen extensions are read. The table below is pinned to the extraction
registry by `tests/test_docs_promises.py` — row for row, and cell for cell on
the evidence class the code records — so a type added without a row here turns
that test red.
| File type | Read by | Dependency | Evidence | Note |
|---|---|---|---|---|
| `.md` | `_extract_passthrough` | core | stdlib, no corpus class | The decoded bytes are the concept text, so `source_lines` are the original's own lines. |
| `.txt` | `_extract_passthrough` | core | stdlib, no corpus class | As `.md`. A document with no headings yields no segments, which is a failed build rather than a flat bundle. |
| `.csv` | `_extract_csv` | core | stdlib, no corpus class | Parsed with the stdlib reader and rendered as one markdown table; a file with no header row is refused. |
| `.json` | `_extract_json` | core | stdlib, no corpus class | Fenced verbatim. No structure is derived from the keys. |
| `.html` | `_extract_html` | core | measured | Block tags open their own lines and `h1`–`h6` carry the ATX marker for their own level. It does **not** go through the converter, although the converter reads HTML: that route would add CVE-2025-51591 (SSRF via an iframe in HTML input) and buy nothing. The denominator is 828 files — one product, one format, one publisher. |
| `.htm` | `_extract_html` | core | stdlib, no corpus class | The same reader as `.html`. The 828-file class is recorded for `.html` alone, and this row does not borrow it. |
| `.xml` | `_extract_xml` | core | measured | A NISO-STS document (`<standard>` root, or any `<sec>`) becomes one heading per titled section at the section's own nesting depth; any other XML keeps its text in document order and gets no invented structure. A `<!DOCTYPE` is refused unparsed. The denominator is one file, one publisher, one schema — 2 761 titled sections. |
| `.pdf` | `_extract_pdf` | `[extract]`: pdfplumber | measured | Eight corpus documents with a hand-counted fasit, plus a 701-page process code whose publisher also ships its structure. Prose only: drawn content has no text to recover, and every extraction warns. OCR lives here as a reading mode for a PDF page whose own text never arrived (`--ocr`, `OCR_CID_SHARE`), never as an entry for image files. |
| `.docx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Five corpus documents. `source_lines` index the extracted text and not the original's paragraphs — the two counts agree on none of the five. |
| `.xlsx` | `_extract_office` | `[extract]`: pypandoc-binary | measured | Written as pipe tables, one source row per line; a sheet name becomes a heading, and a row is located by `source_sheet` and `source_rows`. |
| `.pptx` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 2 decks. 2 of 2 slide titles recovered on a deck that declares them, 0 of 2 on a deck that does not, where the converter writes `Slide 1` / `Slide 2` because it has no title to use. |
| `.odt` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document. 1 of 1 declared headings recovered, 1 concept, 0 characters in no segment. |
| `.rtf` | `_extract_office` | `[extract]`: pypandoc-binary | constructed | N = 1 document, and the weakest row here. 0 declared headings: the container carries no heading style, so the author's title is bold text and the document lands as one concept — content preserved, structure zero. The `--bold-title` flag reads that bold line and is off by default. |
The three classes are the code's own and are not interchangeable. `measured`
means real documents someone wrote for their own purposes, counted against a
fasit written before the lookup. `constructed` means the row has been through
end to end on a hand-built document with a hand-written fasit and has met no
document anyone else wrote. `stdlib, no corpus class` means the code records no
class for the row at all. The office rows are set out in full under [Binary
extraction](#binary-extraction).
### Not read today
Facts about the registry as it stands, not a queue — nothing here is planned.
- `.doc` (Word 97) — the converter does not read it.
- `.epub` — the converter does read it, and the row is deliberately absent: it
would buy nothing today, the same half of the reason that keeps HTML out of
the converter.
- `.eml` and `.msg` — no reader. An email file arrives as an unregistered
extension.
- Image files — no reader, and OCR is a PDF page mode rather than an entry for
them.
- Source files (`.py`, `.ts`, and the rest) — no reader. The text rows are
`.md`, `.txt`, `.csv` and `.json`.
- `.one` (OneNote) and `.vsd` (Visio) — no reader.
An unregistered extension is a coded rejection (`extractor_unknown`), never a
silent skip.
## Install in detail
Neither this package nor the guard it depends on is on a package index yet, so