okf-wiki/README.md
Kjell Tore Guttormsen 0fd86a23a3
okf-wiki 1.1.1
First public release, published as one snapshot. What changed between
versions is in CHANGELOG.md.

Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
2026-11-14 21:03:20 +01:00

160 lines
8.6 KiB
Markdown

# okf-wiki
Build an Open Knowledge Format wiki from a folder of documents in your own Claude Code session: every claim carries a verbatim quote and the place it comes from. The tools never call a model.
You have a folder of documents (Markdown, text, PDF, Word, Excel, PowerPoint) and want pages you can check: each
sentence names the document and the line, page, cell or slide it comes from, and the source opens at that place.
okf-wiki is a Claude Code plugin for one person or a small team who wants to look things up across such a folder and
check the answer.
*AI-generated: all code produced by Claude Code through dialog-driven development.*
## Install
Use the `https://` form: `claude plugin marketplace add` does not accept an `ssh://` URL.
```bash
claude plugin marketplace add https://git.fromaitochitta.com/open/okf-wiki.git
claude plugin install okf-wiki@okf-wiki
```
Then, in a Claude Code session, ask: `Build a wiki from the folder ~/Documents/reports`. The skill puts the frozen
copy, the work folder and the wiki beside that folder, not in it.
## Requirements
- Claude Code. The model work runs in your own session, so a build spends your own tokens (see [Cost](#cost)).
- Python 3 as `python3`, and PyYAML. Tested with Python 3.14 and PyYAML 6.0.3. Everything else is the standard library.
- PDF files need `pypdf==6.14.2`, that exact version (`python3 -m pip install pypdf==6.14.2`). Without it a folder
with a PDF is refused. Word, Excel and PowerPoint files need nothing extra.
- bash, for `bin/okf-wiki`. Tested on macOS.
## What it does
- Freezes a folder: every `.md`, `.markdown`, `.txt`, `.pdf`, `.docx`, `.xlsx` and `.pptx` file, in subfolders too.
Each file is kept as the text a reader of it sees, with the page, paragraph, sheet and cell, or slide of every
line. Hidden sheets, rows, slides and text and tracked deletions are left out and reported. A scanned PDF without a
text layer is skipped and named.
- Builds one concept page per idea, added to document by document. Every sentence has a footnote with a quote that
stands word for word in the source, at a stated place (`Report.pdf, page 2`).
- Checks the result with code: every quote at its place, the format, the links, pages nobody links to.
- Has an independent reader judge whether the quotes support the claims, and repairs the claims it rejects.
- Renders a static HTML view where every footnote opens the source at the quoted line.
- Serves read-only lookups over one or more collections as an MCP server. Plain file search keeps working beside it.
```mermaid
flowchart LR
F[Folder] --> S[Frozen copy] --> B[Blocks] --> C[Fragments*] --> P[Pages]
P --> A[Auditor]
P --> R[Reader*]
P --> V[HTML view]
P --> M[MCP lookups]
```
Steps marked `*` are done by subagents in your session. Every other step is plain Python with no model call.
## Non-goals
- No language model call from the shipped tools, and no API key. All model work happens in your Claude Code session.
- No search by meaning. The lookup server matches words; the model reads the pages it finds.
- No OCR. A PDF without a text layer is skipped.
- No marking of a claim as outdated by a later source. When sources disagree, both claims are kept, each with its quote.
- No format of its own. The pages follow Google's
[Open Knowledge Format](https://github.com/GoogleCloudPlatform/open-knowledge-format) (SPEC v0.2, read at commit
`ad30107`). Frontmatter keys that start with `x-` are this project's own additions, each with its measured reason
in [docs/x-fields.md](docs/x-fields.md).
- No hosted service, account or payment.
## Known limitations
- The reader judged 300 random claims of each built collection. It found 17 not supported (5.67 %) in a collection of
3,162 pages built from software documentation, and 24 (8.00 %) in a collection of 555 pages built from a folder of
PDF and Office files. The aim is under 3 %. A fresh build does not reach it.
- Repair brought the folder collection to 5 of 300 (1.67 %), measured 14.11.2026: every claim was read alone, the
rejected ones were repaired, and 96 claims that could not be proven from their quotes were removed. Most of those
were table rows that need more heading lines than a claim may cite, and bullets under a slide title.
- The language of a document is recorded only as Norwegian or English.
- Guards without a test of their own and limits of the measurement tools are listed in
[docs/known-limitations.md](docs/known-limitations.md).
## Cost
A build is paid for by your Claude Code session. Before the first pack you set a ceiling in tokens. After every pack
the skill checks a stop line, and the build stops before the next pack would go over the ceiling. Measured cost per
character of source text was 6.9 tokens on the documentation collection and 5.8 and 8.5 on two folder trials
([CHANGELOG](CHANGELOG.md), 1.1). A PDF of 387,767 characters was the largest measured to hold in one compiler run.
## The skill
`skills/okf-wiki/SKILL.md` is how your session builds the wiki, in packs of five sources:
1. Code splits the frozen copy into blocks.
2. A compiler subagent (`okf-wiki-compiler`) reads one pack's blocks and the list of concepts that exist, and writes
short fragments. Each has a verbatim quote from one source line and a provenance mark: `documented`, `inferred` or
`merged`. It names the main page of each single-topic document and a home (a main page that links to it) for every
other page.
3. Code checks every quote, drops a fragment that fails, assembles the pages and runs the auditor after every pack.
4. A linker subagent (`okf-wiki-linker`) proposes links by meaning. Code checks and places them and never makes a link
of its own.
5. A merger subagent (`okf-wiki-merger`) judges pairs of concepts built twice under two names; code merges them.
6. The reader (`okf-wiki-reader`) judges claims against the quotes they cite, up to ten per call. A claim it does not
support is repaired at its fragment by `okf-wiki-repairer` and read again.
Only one build runs at a time. Two at once would create the same concept twice.
To answer a question, the skill searches `index.md`, opens the main page of the area, follows its links, and opens the
source behind a footnote when the quote has to be checked.
## Running the tools yourself
Installed as a plugin, `okf-wiki MODULE ...` runs `python3 -m okfwiki.MODULE ...` from any folder in a Claude Code
session. From a clone, run `python3 -m okfwiki.MODULE` in the repository root, or call `bin/okf-wiki` by its path.
Every module takes `--help`. None of them calls a model.
```bash
okf-wiki sources folder DIR --collection NAME --out SOURCES # freeze a folder
okf-wiki sources check SOURCES # check the frozen copy
okf-wiki audit WIKI SOURCES # JSON findings; exit 0 clean, 1 findings, 2 could not run
okf-wiki view render WIKI SOURCES --out HTML # static HTML view
okf-wiki view check WIKI HTML # prints "footnotes resolved: X of N"
okf-wiki sources watch SOURCES --into DIR --bundle WIKI # documents new, changed or gone since the freeze, and their pages
```
The auditor reports four groups apart: `spec-11` (SPEC § 11 conformance), `links` (this project's link rules),
`citations` (every sentence has a footnote, every footnote a source, every quote stands at its place) and `format`.
## Lookups over MCP
Clone the repository, then register the server with the path to the clone:
```bash
git clone https://git.fromaitochitta.com/open/okf-wiki.git
claude mcp add okf-wiki -- /path/to/okf-wiki/bin/okf-wiki serve --bundle NAME WIKI SOURCES
```
Repeat `NAME WIKI SOURCES` after `--bundle` for more collections. Four tools follow the order the skill answers in:
`collections`, `search`, `page` and `source`. The server writes nothing.
## Verify it yourself
Nothing runs the tests automatically: the forge has no CI. From a clean clone, with PyYAML installed:
```bash
python3 -m unittest discover -s tests
python3 -m unittest discover -s gates/tests
python3 -m unittest discover -s evals/tests
```
Run a single test file with `-p 'test_X.py'`. The type check runs mypy through uvx, offline, so mypy and
types-PyYAML must already be in uvx's cache (run it once without `--offline` to fetch them):
`uvx --offline --with types-PyYAML mypy --strict okfwiki gates tests tools/*.py`.
`tests/v1_repo.sh` runs these with the owner's other repository checks. Several of them read local files that are
not in the repository (the test collections, the excluded-terms list) and print FAIL without them.
## Changelog
See [CHANGELOG.md](CHANGELOG.md).
## License
MIT. See [LICENSE](LICENSE).