First public release, published as one snapshot. What changed between versions is in CHANGELOG.md. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
160 lines
8.6 KiB
Markdown
160 lines
8.6 KiB
Markdown
# okf-wiki
|
|
Build an Open Knowledge Format wiki from a folder of documents in your own Claude Code session: every claim carries a verbatim quote and the place it comes from. The tools never call a model.
|
|
|
|
You have a folder of documents (Markdown, text, PDF, Word, Excel, PowerPoint) and want pages you can check: each
|
|
sentence names the document and the line, page, cell or slide it comes from, and the source opens at that place.
|
|
okf-wiki is a Claude Code plugin for one person or a small team who wants to look things up across such a folder and
|
|
check the answer.
|
|
|
|
*AI-generated: all code produced by Claude Code through dialog-driven development.*
|
|
|
|
## Install
|
|
|
|
Use the `https://` form: `claude plugin marketplace add` does not accept an `ssh://` URL.
|
|
|
|
```bash
|
|
claude plugin marketplace add https://git.fromaitochitta.com/open/okf-wiki.git
|
|
claude plugin install okf-wiki@okf-wiki
|
|
```
|
|
|
|
Then, in a Claude Code session, ask: `Build a wiki from the folder ~/Documents/reports`. The skill puts the frozen
|
|
copy, the work folder and the wiki beside that folder, not in it.
|
|
|
|
## Requirements
|
|
|
|
- Claude Code. The model work runs in your own session, so a build spends your own tokens (see [Cost](#cost)).
|
|
- Python 3 as `python3`, and PyYAML. Tested with Python 3.14 and PyYAML 6.0.3. Everything else is the standard library.
|
|
- PDF files need `pypdf==6.14.2`, that exact version (`python3 -m pip install pypdf==6.14.2`). Without it a folder
|
|
with a PDF is refused. Word, Excel and PowerPoint files need nothing extra.
|
|
- bash, for `bin/okf-wiki`. Tested on macOS.
|
|
|
|
## What it does
|
|
|
|
- Freezes a folder: every `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, `.xlsx` and `.pptx` file, in subfolders too.
|
|
Each file is kept as the text a reader of it sees, with the page, paragraph, sheet and cell, or slide of every
|
|
line. Hidden sheets, rows, slides and text and tracked deletions are left out and reported. A scanned PDF without a
|
|
text layer is skipped and named.
|
|
- Builds one concept page per idea, added to document by document. Every sentence has a footnote with a quote that
|
|
stands word for word in the source, at a stated place (`Report.pdf, page 2`).
|
|
- Checks the result with code: every quote at its place, the format, the links, pages nobody links to.
|
|
- Has an independent reader judge whether the quotes support the claims, and repairs the claims it rejects.
|
|
- Renders a static HTML view where every footnote opens the source at the quoted line.
|
|
- Serves read-only lookups over one or more collections as an MCP server. Plain file search keeps working beside it.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
F[Folder] --> S[Frozen copy] --> B[Blocks] --> C[Fragments*] --> P[Pages]
|
|
P --> A[Auditor]
|
|
P --> R[Reader*]
|
|
P --> V[HTML view]
|
|
P --> M[MCP lookups]
|
|
```
|
|
|
|
Steps marked `*` are done by subagents in your session. Every other step is plain Python with no model call.
|
|
|
|
## Non-goals
|
|
|
|
- No language model call from the shipped tools, and no API key. All model work happens in your Claude Code session.
|
|
- No search by meaning. The lookup server matches words; the model reads the pages it finds.
|
|
- No OCR. A PDF without a text layer is skipped.
|
|
- No marking of a claim as outdated by a later source. When sources disagree, both claims are kept, each with its quote.
|
|
- No format of its own. The pages follow Google's
|
|
[Open Knowledge Format](https://github.com/GoogleCloudPlatform/open-knowledge-format) (SPEC v0.2, read at commit
|
|
`ad30107`). Frontmatter keys that start with `x-` are this project's own additions, each with its measured reason
|
|
in [docs/x-fields.md](docs/x-fields.md).
|
|
- No hosted service, account or payment.
|
|
|
|
## Known limitations
|
|
|
|
- The reader judged 300 random claims of each built collection. It found 17 not supported (5.67 %) in a collection of
|
|
3,162 pages built from software documentation, and 24 (8.00 %) in a collection of 555 pages built from a folder of
|
|
PDF and Office files. The aim is under 3 %. A fresh build does not reach it.
|
|
- Repair brought the folder collection to 5 of 300 (1.67 %), measured 14.11.2026: every claim was read alone, the
|
|
rejected ones were repaired, and 96 claims that could not be proven from their quotes were removed. Most of those
|
|
were table rows that need more heading lines than a claim may cite, and bullets under a slide title.
|
|
- The language of a document is recorded only as Norwegian or English.
|
|
- Guards without a test of their own and limits of the measurement tools are listed in
|
|
[docs/known-limitations.md](docs/known-limitations.md).
|
|
|
|
## Cost
|
|
|
|
A build is paid for by your Claude Code session. Before the first pack you set a ceiling in tokens. After every pack
|
|
the skill checks a stop line, and the build stops before the next pack would go over the ceiling. Measured cost per
|
|
character of source text was 6.9 tokens on the documentation collection and 5.8 and 8.5 on two folder trials
|
|
([CHANGELOG](CHANGELOG.md), 1.1). A PDF of 387,767 characters was the largest measured to hold in one compiler run.
|
|
|
|
## The skill
|
|
|
|
`skills/okf-wiki/SKILL.md` is how your session builds the wiki, in packs of five sources:
|
|
|
|
1. Code splits the frozen copy into blocks.
|
|
2. A compiler subagent (`okf-wiki-compiler`) reads one pack's blocks and the list of concepts that exist, and writes
|
|
short fragments. Each has a verbatim quote from one source line and a provenance mark: `documented`, `inferred` or
|
|
`merged`. It names the main page of each single-topic document and a home (a main page that links to it) for every
|
|
other page.
|
|
3. Code checks every quote, drops a fragment that fails, assembles the pages and runs the auditor after every pack.
|
|
4. A linker subagent (`okf-wiki-linker`) proposes links by meaning. Code checks and places them and never makes a link
|
|
of its own.
|
|
5. A merger subagent (`okf-wiki-merger`) judges pairs of concepts built twice under two names; code merges them.
|
|
6. The reader (`okf-wiki-reader`) judges claims against the quotes they cite, up to ten per call. A claim it does not
|
|
support is repaired at its fragment by `okf-wiki-repairer` and read again.
|
|
|
|
Only one build runs at a time. Two at once would create the same concept twice.
|
|
|
|
To answer a question, the skill searches `index.md`, opens the main page of the area, follows its links, and opens the
|
|
source behind a footnote when the quote has to be checked.
|
|
|
|
## Running the tools yourself
|
|
|
|
Installed as a plugin, `okf-wiki MODULE ...` runs `python3 -m okfwiki.MODULE ...` from any folder in a Claude Code
|
|
session. From a clone, run `python3 -m okfwiki.MODULE` in the repository root, or call `bin/okf-wiki` by its path.
|
|
Every module takes `--help`. None of them calls a model.
|
|
|
|
```bash
|
|
okf-wiki sources folder DIR --collection NAME --out SOURCES # freeze a folder
|
|
okf-wiki sources check SOURCES # check the frozen copy
|
|
okf-wiki audit WIKI SOURCES # JSON findings; exit 0 clean, 1 findings, 2 could not run
|
|
okf-wiki view render WIKI SOURCES --out HTML # static HTML view
|
|
okf-wiki view check WIKI HTML # prints "footnotes resolved: X of N"
|
|
okf-wiki sources watch SOURCES --into DIR --bundle WIKI # documents new, changed or gone since the freeze, and their pages
|
|
```
|
|
|
|
The auditor reports four groups apart: `spec-11` (SPEC § 11 conformance), `links` (this project's link rules),
|
|
`citations` (every sentence has a footnote, every footnote a source, every quote stands at its place) and `format`.
|
|
|
|
## Lookups over MCP
|
|
|
|
Clone the repository, then register the server with the path to the clone:
|
|
|
|
```bash
|
|
git clone https://git.fromaitochitta.com/open/okf-wiki.git
|
|
claude mcp add okf-wiki -- /path/to/okf-wiki/bin/okf-wiki serve --bundle NAME WIKI SOURCES
|
|
```
|
|
|
|
Repeat `NAME WIKI SOURCES` after `--bundle` for more collections. Four tools follow the order the skill answers in:
|
|
`collections`, `search`, `page` and `source`. The server writes nothing.
|
|
|
|
## Verify it yourself
|
|
|
|
Nothing runs the tests automatically: the forge has no CI. From a clean clone, with PyYAML installed:
|
|
|
|
```bash
|
|
python3 -m unittest discover -s tests
|
|
python3 -m unittest discover -s gates/tests
|
|
python3 -m unittest discover -s evals/tests
|
|
```
|
|
|
|
Run a single test file with `-p 'test_X.py'`. The type check runs mypy through uvx, offline, so mypy and
|
|
types-PyYAML must already be in uvx's cache (run it once without `--offline` to fetch them):
|
|
`uvx --offline --with types-PyYAML mypy --strict okfwiki gates tests tools/*.py`.
|
|
|
|
`tests/v1_repo.sh` runs these with the owner's other repository checks. Several of them read local files that are
|
|
not in the repository (the test collections, the excluded-terms list) and print FAIL without them.
|
|
|
|
## Changelog
|
|
|
|
See [CHANGELOG.md](CHANGELOG.md).
|
|
|
|
## License
|
|
|
|
MIT. See [LICENSE](LICENSE).
|