Kjell Tore Guttormsen
9436427520
`IndexPolicy` grew five judging fields and `IndexPolicy.violations`, closing the
gap a test has been pinning since `848e395`. `STRICT_V1.index` is now the proving
consumer's shape rather than DEFAULT's.
The design was settled by a conflict rather than by a preference. The convention
owner reported that an index is an authored count of a directory's children and
that a validator enumerating the directory has implemented the wrong contract.
Checked against the proving consumer before adopting it, the two turn out to be
directly opposed: gate BUNDLE_INDEX_COMPLETE (validate.py:1081-1120, ERROR) builds
its expected set by enumerating the directory and demands an exact bidirectional
match, and bundle.py:498-567 writes every index from a tree walk. Neither is
incoherent inside its own spec, so authored-versus-derived is a policy field in
both directions and a library invariant in neither.
Consequences encoded rather than documented: nothing here enumerates a directory
— the caller supplies the listing, `violations` refuses one when the profile's
index is authored and refuses to run without one when it is derived, so code
written to the wrong contract fails at the call instead of passing every test one
would think to write. Root and nested indexes are asymmetric (confirmed in both
consumers, different key sets). A per-entry description is template-level, so
`render_link` refuses both a missing description and an unwanted one.
DEFAULT keeps upstream's root-only index and judges nothing, for the same measured
reason it carries no required frontmatter key set: upstream binds `index.md` to
the bundle root alone, so a judging default would condemn conforming bundles.
25 new tests, 493 green. C1 re-proven: `git diff --stat examples/` empty.
142 lines
6.9 KiB
Markdown
142 lines
6.9 KiB
Markdown
# llm-ingestion-okf
|
|
|
|
## Context
|
|
|
|
Shared OKF (Open Knowledge Format) ingestion library. Three entry doors,
|
|
one boundary rule:
|
|
|
|
- **Door A — spec-based ingestion:** implements the normative
|
|
`ingest-spec.md` owned by `portfolio-optimiser-commons` (manifest →
|
|
`file`/`sql`/`http` connector → deterministic materialization of
|
|
`ingest-{id}.md` → index generation; zero model calls). This repo
|
|
IMPLEMENTS the spec; commons keeps authorship. Spec changes the library
|
|
needs go via commons, never edited locally. The library ships the §11
|
|
golden fixtures (byte-exact) for the three door-A source types
|
|
(`ingest-golden-{file,sql,http}/`, shipped in `9dd86b1`).
|
|
- **Door B — bundle inbox:** converts dropped files to OKF concepts. All
|
|
file-type→text extraction lives HERE (the guard is text-only). v1 core:
|
|
`md`, `txt`, `csv`, `json`, `html` (stdlib). `pdf`/`docx`/`xlsx` only via
|
|
the optional `[extract]` extra; without it those types are rejected
|
|
fail-fast.
|
|
- **Door C — external bundle import:** third-party OKF bundles are assessed
|
|
per concept via the guard's `okf.import_bundle`; only concepts clearing the
|
|
guard's non-blocking floor are merged/indexed here. Two invariants, both
|
|
load-bearing: a merged concept is written **verbatim** (this library's
|
|
line-oriented frontmatter parser cannot round-trip the block lists the
|
|
guard's parser accepts, so stamping an external concept would destroy sender
|
|
data and persist bytes the guard never screened), and ownership is therefore
|
|
proven by **content identity** — an occupied target name is re-used only
|
|
when the bytes there are already identical, never overwritten otherwise.
|
|
|
|
**Boundary rule (non-negotiable, zero overlap):** `llm-ingestion-guard`
|
|
(pinned `>=0.2,<0.3`) answers "is this content safe to persist?" —
|
|
scan/sanitize/quarantine/fail-secure/provenance-stamp. This library is
|
|
plumbing: connect source → materialize deterministic OKF bundle → generate
|
|
index. Never reimplement security; call the guard at persist gates
|
|
(`prepare_input`/`screen_output`, `okf.import_bundle`). When in doubt which
|
|
side of the boundary something belongs on: ask the operator.
|
|
|
|
**Implementation baseline:** the stricter behaviors from
|
|
`portfolio-optimiser` (streaming row caps, utf-8-sig, in-memory staging with
|
|
pre-mutation collision gate, validated `ingested_at`, typed `IngestError`)
|
|
are the library baseline. First consumer: `portfolio-optimiser-claude`.
|
|
|
|
### Roadmap (committed phases — all known OKF surfaces get covered)
|
|
|
|
1. **Phase 1 — Door A (Python).** ingest-spec implementation + the §11
|
|
golden fixtures. Consumers: `portfolio-optimiser-claude` first, then
|
|
`portfolio-optimiser`.
|
|
2. **Phase 2 — Doors B/C (Python).** Bundle inbox and external-bundle
|
|
import, guard-gated.
|
|
3. **Phase 3 — Configurable bundle contract.** Types, layers, frontmatter
|
|
sets, index shape, and reserved-file policy become config instead of
|
|
constants; proving consumer is `claude-code-llm-wiki` (`strict-v1`
|
|
profile). Two consumers hold opposite postures on whether an index is
|
|
authored or directory-derived, so neither is a library invariant and
|
|
nothing here enumerates a directory unless the profile says derived.
|
|
4. **Phase 4 — Node half (`node/`).** Zero-dependency Node/ESM package
|
|
(importable *and* CLI-invokable, vendorable per plugin — matching the
|
|
marketplace precedent) for the second-brain world: bundle check, index
|
|
generation, inbox split/frontmatter/write, and doc conversion
|
|
(docx/pdf/eml/html → md). Covers okr, linkedin-studio, ms-ai-architect,
|
|
and the marketplace catalog.
|
|
|
|
The two halves share the OKF contract and fixture suite, **not code**.
|
|
|
|
Phase 4 preconditions (coordination, not unilateral moves):
|
|
- Lifts okr's reference implementations (`okf-check.mjs`, `okf-index.mjs`,
|
|
innboks libs) in agreement with okr and the marketplace catalog; the
|
|
catalog remains the convention owner and re-pins its shared gate here.
|
|
- linkedin-studio's `ingest/published/` provenance-record grammar stays
|
|
plugin-local by design (different lifecycle) — do not normalize it.
|
|
- The Node-side persist gate remains security territory: guard-as-contract
|
|
(per okr's adoption doc) until a Node guard exists in the security repo.
|
|
No security reimplementation here, in either runtime.
|
|
|
|
### Non-goals (all phases)
|
|
|
|
- Verdict/feedback machinery (method-spec) — stays in consumer repos.
|
|
- Embedding/RAG/retrieval layers.
|
|
- Security functionality — always the guard's domain.
|
|
|
|
## Stack
|
|
|
|
Python 3.10+. Package `llm_ingestion_okf` (src layout, hatchling).
|
|
**Exactly one runtime dependency, ever:** `llm-ingestion-guard>=0.2,<0.3`
|
|
(itself zero-dep), landed with the Door B/C persist gates. Everything else is
|
|
stdlib, and a packaging test enforces it. Only `guard_adapter.py` imports the
|
|
guard; importing the package does not. Install channel until the package
|
|
index exists (a direct reference is a channel, not the pin):
|
|
`pip install "llm-ingestion-guard @ git+https://git.fromaitochitta.com/open/llm-ingestion-pipeline-security.git@v0.2.0"`.
|
|
Binary extraction parsers live behind the `[extract]` extra only.
|
|
|
|
Phase 4 adds a `node/` half: Node/ESM with zero npm dependencies
|
|
(`node:` builtins only), both importable and CLI-invokable, consumed by
|
|
vendoring per plugin rather than npm publishing. The halves share contract
|
|
and fixtures, never code.
|
|
|
|
## Conventions
|
|
|
|
- Type hints everywhere; `mypy --strict` target.
|
|
- Determinism is bit-exact: `ingested_at` is an explicit required argument
|
|
(no wall-clock defaults); LF-only output; golden fixtures compared
|
|
byte-for-byte.
|
|
- No model calls anywhere in the run path.
|
|
- Credentials only as env-var *references* resolved at runtime; never in
|
|
manifests, logs, or frontmatter.
|
|
- Network access requires an explicit per-run opt-in flag; refuse fail-fast
|
|
otherwise.
|
|
- Conventional Commits: `type(scope): description`.
|
|
- English for all code, docs, and commit messages (public repo).
|
|
|
|
## Commands
|
|
|
|
- Test: `pytest`
|
|
- Lint: `ruff check .` + `ruff format --check .`
|
|
- Type check: `mypy --strict src/`
|
|
|
|
## Workflow
|
|
|
|
- TDD: no production code without a failing test first.
|
|
- This repo is published PUBLICLY (`open/` namespace on Forgejo). `STATE.md`
|
|
and `docs/oppstartsprompt.md` are LOCAL-ONLY (gitignored) — never commit
|
|
session state or internal briefs. No secrets, sober English prose, no
|
|
marketing language.
|
|
- After `git commit`: push to Forgejo (`git push origin`) immediately.
|
|
Never GitHub.
|
|
|
|
## Communication patterns
|
|
|
|
### Linking to local files
|
|
|
|
When pointing to local files in responses, always use markdown link syntax
|
|
with a descriptive name:
|
|
|
|
- Use `[Human-friendly name](file:///absolute/path)` — never bare
|
|
`file:///...` URLs or autolinks `<file://...>`.
|
|
- Always use absolute paths. Never `~/` or relative paths.
|
|
- For multiple files, render as a bullet list of named markdown links.
|
|
|
|
Why: bare `file://` URLs only render the first as clickable across multiple
|
|
lines. Named markdown links make each entry independently clickable and look
|
|
cleaner.
|