feat(extract): the file that IS the product now reads, as a core stdlib type
GREEN on the twelve tests from the two commits before it. `.xml` joins `_CORE_EXTRACTORS` beside `.html`, and 110 of 110 unreadable becomes a bundle. A CORE type, not an `[extract]` one, because the parser is stdlib: putting it behind the extra would make a pure-stdlib file type binary-dependent for no gain. And never through the converter, which the existing fence test now says in one more line: a file routed there is read by a second parser that never sees this reader's DTD refusal, so the hardening would be true of the code and false of the file. TWO PATHS, NAMED RATHER THAN GUESSED. STS when the root is `<standard>` or the document holds any `<sec>`; generic otherwise. Generic XML keeps its text in document order and gets NO structure -- an element name is never promoted to a heading, because an RSS feed is not a numbered standard. THE GRAMMAR IS MARKDOWN, the same markdown the office rows and round 11's HTML row reach the proposer through, and `propose.py` is untouched. `<sec>` with a `<title>` becomes one ATX line carrying `<label>` + space + `<title>` at its own nesting depth; `<sec>` with only a `<label>` becomes a body line with the label in front, the way `li` is treated in HTML -- 4 954 of R761's 7 715 are lettered points and one heading each would bury its own 2 761; `<table-wrap>` becomes its label plus ONE markdown table through this package's own `render_table`, separator line included, which is what makes it a block. TWO CHOICES THAT ARE MEASUREMENTS, not preferences: - Inline by ALLOWLIST, block by default -- the inverse of the HTML reader, because block-versus-inline is a property of HTML and XML has no such universal. Assuming block is safe (an extra break never removes text and a boundary needs a line that matches a grammar); assuming inline is not. The allowlist is load-bearing: that document carries 1 701 `<italic>` and 1 396 `<bold>` inside its prose. - The ATX ceiling is 6 and STS nesting reaches 7. The depth is CLIPPED, not dropped: 9 of the 2 761 titled sections sit at depth 7 and `#######` matches nothing, so dropping loses the section while clipping keeps the boundary and states the nesting one level too shallow. A DTD IS REFUSED UNPARSED, and that is a guarantee about this code rather than about the machine. Measured on this interpreter (3.14.0, pyexpat 2.7.3): an external SYSTEM entity is refused by the stdlib and never fetched, but the amplification limit that stops a billion-laughs comes from libexpat >= 2.4.0 and NOT from Python -- five levels still expand -- while `pyproject.toml` requires only `>=3.10` and no lockfile pins an interpreter. `XMLParser` exposes no `.parser` attribute on the C accelerator, so the handler route is not portable either. NO new dependency: `defusedxml` and `lxml` both occur 0 times in `uv.lock` and still do. pytest -q: 1566 passed, 1 skipped. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This commit is contained in:
parent
5bb3b045c2
commit
567a63d455
4 changed files with 276 additions and 4 deletions
11
README.md
11
README.md
|
|
@ -537,10 +537,17 @@ bundle:
|
|||
run path; output is reproducible byte-for-byte against golden fixtures.
|
||||
2. **Bundle inbox.** A drop directory where common file types are converted
|
||||
to OKF concept files. All file-type→text extraction lives in this library:
|
||||
`md`, `txt`, `csv`, `json`, and `html` are handled by the stdlib core;
|
||||
`md`, `txt`, `csv`, `json`, `html` and `xml` are handled by the stdlib core;
|
||||
`pdf` and the five office formats (`docx`, `xlsx`, `pptx`, `odt`, `rtf`)
|
||||
require the optional `[extract]` extra and are rejected fail-fast without
|
||||
it. Extracted text passes the security gate before anything is persisted.
|
||||
An `xml` file that declares NISO-STS structure (`<standard>` root, or any
|
||||
`<sec>`) becomes one heading per titled section, at the section's own
|
||||
nesting depth, with the section's `<label>` and `<title>` on one line; a
|
||||
`<sec>` carrying only a label is a body line and never a heading, and a
|
||||
`<table-wrap>` becomes one markdown table. Any other XML keeps its text in
|
||||
document order and gets no invented structure. XML carrying a
|
||||
`<!DOCTYPE` is refused unparsed.
|
||||
The drop directory is walked **recursively**, in sorted relative-path order:
|
||||
a file at any depth is ingested and records its path relative to the inbox
|
||||
root as its `source_file`, while dot-directories and a bundle directory
|
||||
|
|
@ -556,7 +563,7 @@ bundle:
|
|||
numbers index the extracted text and say so. Measurements:
|
||||
[`docs/2026-09-08-proveniens-k2.md`](docs/2026-09-08-proveniens-k2.md).
|
||||
|
||||
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
|
||||
<!-- extract-formats: .md, .txt, .csv, .json, .html, .htm, .xml, .pdf, .docx, .xlsx, .pptx, .odt, .rtf -->
|
||||
3. **External bundle import.** Import and merge of third-party OKF bundles:
|
||||
each concept is assessed via the security gate, and only concepts that
|
||||
pass are merged, materialized, and linked into the index.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue