Commit graph

73 commits

Author SHA1 Message Date
910a12455a docs(limitations): correct the percent-escape framing -- slugger class, not language
A consumer corrected a claim we shipped this morning. We wrote that both
zero-measuring corpora were English, and used that to explain the third corpus's
10 Norwegian hits as a sampling bias. One of those two is a Norwegian repo with
Norwegian content, so the stated explanation was wrong.

Their measurement separates two axes we had conflated. For third-party LINK
corpora -- URLs collected from other people's sites -- language does predict, and
the Norwegian legal/government sources are where the escapes were. For GENERATED
paths the predictor is slugger class: a whitelist slugger ([^a-z0-9]+ -> "-")
cannot emit an escape in any language because it discards the character before
anything encodes it, while encodeURIComponent produces them systematically as soon
as titles are non-ASCII. Their own slugger returns a structural zero on Norwegian
input, not a statistical one.

So non-ASCII language is a confounder for encode-vs-whitelist, and slugger class is
the thing that is actually testable at a consumer -- a one-line code read rather
than a corpus census. Reframed accordingly, with the earlier conflation named so
the correction is visible rather than silently rewritten.
2026-07-27 20:20:45 +02:00
f391b2cee3 docs(readme): sync two stale counts to ground truth
The test badge still said 577 after two commits that added 65 tests, and the
coverage-matrix claim said 126 where CORE_CASES now holds 134 payloads. Both
verified by running rather than by reading: pytest reports 642, and
len(coverage.CORE_CASES) is 134.
2026-07-27 09:12:15 +02:00
2d5a9cd0c7 docs(okf): the mapping class is inexpressible, and T2 binds import not emission
A consumer measured our frontmatter gate against 0.2.0 and asked us to confirm
against a newer guard, correctly noting that a divergence would be a version
difference rather than a contradiction. Measured at 0.3.1: no divergence. Both
their results reproduce exactly.

Two things are sharper than what we documented yesterday:

ALL THREE ROUTES TO A MAPPING FAIL, each on a different rule -- flow {k: v} on the
disallowed value-start indicator, block on the nested-mapping check, dotted keys on
the key pattern. So the mapping CLASS has no expressible form; it is not a choice
between two shapes where one is better. That matters because OKF v0.2's `generated`
IS a mapping (`by` is required when it is present), so it cannot be expressed at
all. Pinned by a test asserting the three failures stay distinct.

T2 RUNS ON DOOR C ONLY. parse_frontmatter is referenced nowhere in the door A/B
persist path, so the same frontmatter that FAIL_SECUREs through import_bundle
passes screen_output unremarked. The grammar bounds what a consumer can RECEIVE,
never what a producer can EMIT -- which is the question a consumer's emitter design
was blocked on.

Also corrects one of our own rows: we reported "sources block list" as a single
FAIL_SECURE case. The parser distinguishes three shapes -- flat scalars parse
correctly, one key per item misparses silently to a string, two keys per item hard-
rejects. A one-element `verified:` block list passes today.

631 -> 642 passed. README item count synced.
2026-07-27 09:11:22 +02:00
684ce3a45f docs(url-shape): make the rule reconstructable, and record what three corpora measured
Three consumers reconstructed is_ordinary_url from prose we sent in coordination
messages and each produced a different wrong number on a real corpus: one omitted
the base64 20-char floor and fired on path words like /blog/; one omitted the
opaque-token condition entirely and undercounted; one computed Shannon entropy
over whole filenames instead of tokens and concluded the 4.4 floor over-blocks
ordinary documents. Same cause each time -- our prose described the rules without
their tokenizer.

docs/URL-SHAPE.md states the algorithm in order, spells out the separator class
and all three length floors, and lists the three reconstruction errors as worked
counter-examples. Its example table is parsed and asserted against the real
predicate by tests/test_url_shape_doc.py, so the reference cannot drift from the
code -- all 18 rows verified load-bearing.

LIMITATIONS.md brought current with the field measurements:

- Percent-escape is no longer zero. Two English corpora measured 0; a 389-file
  Norwegian/Microsoft corpus found 10, all Norwegian (%C3%B8, %C3%A5 are just
  o-slash and a-ring). It is a non-ASCII-language tax, and both zero-measuring
  corpora being English was a sampling bias invisible from inside.
- The query over-block now has THREE disjoint benign populations: utm_* tracking,
  content identity (?v=, ?channel_id=), and Microsoft Learn's ?view= version
  selector. No parameter-level remedy covers any two, which moves this from a
  conclusion to a settled constraint on 0.4.0.
- Legitimate CDN asset ids trip the hex branch permanently; the branch is otherwise
  precise (no other FP in 2401 distinct URLs) and stays.
- Raw HTML with a relative URL attribute is HIGH though it reaches no external
  host, and end tags are counted.
- OKF frontmatter: a one-key block-sequence item is silently misparsed to a string
  where two keys hard-reject, so a pointer can ride past the resource allowlist.
  Consequence: a conformant OKF v0.2 concept cannot traverse door C at all, since
  both backward-breaking migration targets are nested. Fail-secure, but a
  compatibility wall that needs a deliberate parse-safety decision.
- A persist gate cannot cover execution risk, and that boundary is unowned.

New behaviour claims are pinned by tests so a closed concession fails and forces
this doc to be updated. 593 -> 631 passed.
2026-07-27 08:56:24 +02:00
956c835d38 docs(limitations): bound two URL-shape false positives with field measurements
Two consumers measured the 0.3.1 URL-shape rule against real corpora on the
same day. Record what they found, and pin the dispositions both of them
inferred wrongly.

Percent-escape FP: zero occurrences across both corpora (0/347 external URLs
in a 527-document vendor-docs corpus, 0/81 in a capture store). This bounds
the shape rather than closing it -- a consumer that slugs filenames from
titles produces %20 systematically, and that corpus is still unmeasured.

Query FP: the over-block that actually occurs. The two corpora hit disjoint
benign populations -- 16/16 publisher-authored utm_* tracking versus 35/35
content identity (?v=, ?channel_id=) where the parameter IS the resource.
An allowlist keyed on tracking-parameter names resolves the first entirely
and the second not at all, so no parameter-level remedy covers both. That
is input to the 0.4.0 axis-separation scope, not a fix here.

Both consumers reported dispositions they had inferred rather than run, and
both were wrong: a query-carrying link is MEDIUM, so it is held or warned,
never hard-failed. tests/test_wiring.py now pins that, plus the active-tag
gate -- counting raw HTML tags overcounts what the gate flags, since
formatting markup is inert and only name/on*=/URL-attr tags are active.

593 passed.
2026-07-25 20:56:14 +02:00
55ce6265d4 docs(plan): state the gate requirement precisely -- fixture set, not tagged artifact
okf asked outright whether the v1.0 gate requires 'the tagged artifact resolves
0.3.1' or 'the fixture set passes against 0.3.1'. It is the second, and the
answer is locked here so a later session cannot quietly upgrade the ask into a
release request against their operator.

They refused to widen their published range to run the measurement, and the
refusal is better than the ask it answers: widening a published range in order
to generate the evidence that would justify widening it is circular, and a red
result would then have meant unwinding a release to fix a test setup. The
measurement has to be able to say no. Their scratch-venv method -- outside the
project environment, --no-deps, guard installed from the v0.3.1 tag, resolved
version asserted from importlib.metadata -- satisfies the gate exactly.
2026-07-25 20:02:41 +02:00
6b705736cd docs(plan): record two consumer notification promises as release constraints
Both were given to named consumers who are building on current behaviour, and
both were about to live only in STATE.md -- which is gitignored, so a promise
kept there does not survive a machine. Breaking either silently is a release
defect, not a preference.

1. Any change to how PRESET_USER_UPLOAD grades ordinary links and remote images
   requires notice to linkedin-studio before it ships; they pin v0.3.1 at wiring.
   Applies to 0.4.0: the axis separation must change disposition, not grading.
2. Closing the relative-target asymmetry requires notice to llm-ingestion-okf;
   it is a Door C property rather than a guard gap, but closing it changes what
   arrives at their persist gate.

Also records a mechanism we had not modelled: okf reports percent-escapes
reaching filenames through a slugger, so %20 is manufactured systematically from
document titles rather than being incidental as it is in the documentation URLs
the FP was calibrated against.
2026-07-25 20:00:25 +02:00
d39033676a docs(plan): the v1.0 gate cannot be opened by a green result alone
okf pins llm-ingestion-guard >=0.2,<0.3 in [project].dependencies and pins the
uv source to tag v0.2.0 -- verified at their v0.4.0 tag and at their HEAD. The
cap excludes 0.3.1, so the committed Door B+C fixture run cannot reach the
version the gate names. It does not fail either: uv resolves v0.2.0, the
constraint holds, and the fixtures come back green because v0.2.0 never had the
regression. Green from an unmodified tree is the one outcome that proves nothing
and reads as proof.

The gate now requires the result to name the resolved guard version, and records
that transitive arrival does not count as an integration -- okf v0.4.0 makes the
guard a mandatory runtime dependency, so a consumer can acquire us without
choosing us, and today lands on v0.2.0 without the 0.3.0 hardening or the 0.3.1
fix. Silent under-defend, not a break.
2026-07-25 19:57:41 +02:00
10a37d104e docs(plan): retarget the Session G integration gate at v0.3.1 2026-07-25 15:45:45 +02:00
207b2c1679 docs(plan): record Session H as landed — 0.3.1 tagged with measurements 2026-07-25 15:45:14 +02:00
6e9b8168e3 fix(calibration): grade active content on URL shape, not construct type
v0.3.0 made the untrusted upload path unusable: measured on both doors, an
ordinary remote image fail_secure'd and an ordinary link/autolink/refdef
quarantined, so only documents without external references persisted.

Two independent defects compounded; neither fix works alone:

1. `markdown-image: HIGH` fired on any external image. The exfil primitive is a
   URL that moves bytes outward, not an image. `is_ordinary_url` now grades on
   shape - http(s)/protocol-relative, no query, no userinfo, no percent-escape,
   no opaque host label or path segment -> LOW; anything data-carrying keeps the
   carrier's severity. raw-html and data: URIs stay HIGH unconditionally.
   Opacity reuses entropy's primitives; floors calibrated against real doc URLs
   (worst legit token H=4.08, exfil segments 4.36-4.54) and frozen in
   calibration.

2. The quarantine_default floor fired on ANY finding, a premise that broke when
   every ordinary link became a finding. It now fires at MEDIUM+ - a no-op for
   every detector that shipped before 0.3.0 (no LOW/INFO exists), which is what
   makes this a patch rather than a minor.

The corpus blind spot that let this pass 522 green tests is closed: the FP
corpus carries realistic markdown and is asserted on the OUTPUT gate under
PRESET_USER_UPLOAD, with a counter-corpus of exfil-shaped URLs that must still
block. Beaconing and short opaque segments are conceded in LIMITATIONS and
asserted by the coverage matrix rather than papered over.

No new public API; no new preset (0.4.0 work); allow_reserved default unchanged.
2026-07-25 15:36:02 +02:00
da7421e6c8 docs(plan): scope Session H — 0.3.1 regression fix for active-content calibration
v0.3.0 makes the untrusted upload path refuse ordinary documents. Measured on
both doors under PRESET_USER_UPLOAD: one ordinary markdown image reaches
fail_secure, one ordinary link/autolink/refdef reaches quarantine_review, and
only a document with no external references persists.

Root cause is three independent defects that compound. Severity tracks
construct type rather than URL shape, so every external image is HIGH when the
exfiltration primitive is actually a URL that carries data outward. The
quarantine floor fires on any finding at all, a premise that broke once every
link became a finding. And the false-positive corpus could not have caught
either: it holds no markdown links or images, asserts only under
PRESET_TRUSTED_SOURCE where everything warns anyway, and runs _scan_input, so
the output gate where active_content lives is never exercised by it.

Both code changes are required together -- verified that fixing severity alone
still quarantines via the floor, and fixing the floor alone still fail-secures
on a HIGH image.

Records the version reasoning too: 0.3.1 is honest as a patch because the
lexicon carries no LOW/INFO patterns (40 high, 22 medium, 21 critical) and no
other detector emits LOW, so raising the floor to MEDIUM+ is a no-op for every
finding that existed before 0.3.0. The new middle preset stays out of this
release; it is additive API and belongs in 0.4.0.
2026-07-25 15:11:12 +02:00
7025759789 docs(plan): record the revised v1.0 gate and the 0.3.0 cut ahead of Session G
PLAN-v1.md said consumer integration was NOT part of the v1.0 sequence, and
Session G listed A-F as its only dependency -- both of which are now false.
The gate change lived only in STATE.md, which is local-only and overwritten
each session, so the plan was the wrong side of the truth.

Records: 1.0.0 now gates on the first real integration coming back green
(llm-ingestion-okf step 4 against v0.3.0), not on our own suite; the reasoning
(522 self-authored tests prove the code matches the design, not that the
design survives contact -- our first external contact found both a
tag-vs-main divergence and three undocumented behaviour changes); and that
one validated `integrated` outweighs five `planned` declarations, so we do not
wait on repos that may never arrive.

Also fixes Session G's now-stale mechanics: its CHANGELOG entry can no longer
"list A-F" because A/A2/B already shipped under [0.3.0]; the version-sync file
list gains the README status line and install pin; and verification gains an
anonymous clean-venv install check against the new tag, since README must
never advertise a command we have not run.
2026-07-25 12:27:01 +02:00
467b9e3e13 chore(release): 0.3.0 — a gate-loosening change earns a minor, not a patch
Version-sync across pyproject, __init__, README badge/status/install pin and
CHANGELOG, and cut [Unreleased] to [0.3.0].

Why minor: 0772daf made allow_reserved default to True on okf.import_bundle,
so a consumer who upgrades without touching their code gets a LOOSENED gate --
a reserved index.md/log.md in a received bundle is now scanned and may become
mergeable, where v0.2.0 rejected it unconditionally. Under 0.x a >=0.2,<0.3
pin absorbs a 0.2.1 silently but stops at 0.3.0, which is the signal such a
change should send. Found by llm-ingestion-okf, our first real downstream
consumer, against main.

Development Status stays 3 - Alpha and the README keeps "the public API may
still change": this ships the hardened surface (Sessions A/A2/B) to consumers
who are still pinned at v0.2.0 and therefore have none of it. It is not the
v1.0 freeze -- that stays gated on the first real integration coming back
green, which is exactly what 0.3.0 makes possible.

Verified: 522 passed; coverage matrix exit 0; version string present in all
four files; no dangling 0.2.0 outside CHANGELOG history.
2026-07-25 12:24:36 +02:00
80f741e9cc docs(changelog): record the three post-v0.2.0 behaviour changes; next cut is 0.3.0
A downstream consumer (llm-ingestion-okf) found that main had diverged from
v0.2.0 on okf.import_bundle: 0772daf adds allow_reserved defaulting to True,
so a reserved index.md/log.md in a received bundle is now scanned rather than
unconditionally rejected. Their tests pin the v0.2.0 reject, so a 0.2.1 cut
from main would have changed downstream behaviour silently.

Checking that turned up a wider hole: [Unreleased] documented only the
coverage matrix and the docs work, while three commits since the tag change
what an unchanged caller observes -- 0772daf (loosens), 4d53765 (active
content now reaches decide(), HIGH/MEDIUM with compound escalation) and
f4e89d2 (base64-decoded plaintext now hits secret-egress). None were listed,
so CHANGELOG could not answer "is this bump safe?".

Records all three under Changed, and states that the next release is 0.3.0 --
under 0.x a >=0.2,<0.3 pin absorbs a 0.2.1 silently but stops at 0.3.0, which
is exactly the signal a gate-loosening change should send.
2026-07-25 07:15:57 +02:00
99c57629f8 docs(readme): document the real install channel; drop false PyPI claim
The Install section promised `pip install llm-ingestion-guard`, but the
distribution is not on PyPI — a downstream consumer following the README
would fail. Replace it with the channel that actually works, verified in a
throwaway venv against the public mirror:

    pip install "llm-ingestion-guard @ git+https://.../@v0.2.0"

Anonymous HTTPS read on the open/ mirror is confirmed, so consumer CI needs
no deploy key. Also states the two things a consumer must know up front:
a git URL is a PEP 508 direct reference (exact tag, not a >=0.2,<0.3 range;
range pinning arrives with a Forgejo PyPI registry at the first patch release
or second consumer), and vendoring is unsupported because it severs the
security patch channel.

Answers the install-channel question raised by llm-ingestion-okf via coord.
2026-07-25 06:26:25 +02:00
27cd7f3f25 chore(gitignore): ignore *.local.sh (local-only operator scripts) 2026-07-15 19:47:34 +02:00
3dda1f68fa docs(readme): add concrete 'What it protects against' catalogue; split limitations
Answers the gap that the README said what it does NOT stop (a long limitations
section) but never plainly listed what it DOES. Add a 'What it protects against'
section high up: attack classes grouped by OWASP anchor (LLM01 injection + 83
lexicon classes + carriers, LLM02 egress, LLM05 EchoLeak, LLM06 agency, LLM10
fail-secure, OKF T1-T6, container front-end), each driven by a live coverage-matrix
payload. Move the full honest-limitations list to docs/LIMITATIONS.md; README keeps
a high-impact summary + link. Net: protection and limits read in balance, 261 -> 216
lines. Coverage 126/126 and 522 tests unchanged; every class listed is real.
2026-07-15 19:09:04 +02:00
d73cfc882f docs(readme): lead with write-time trust-boundary; feature shipped OKF adapter
Rewrite the README value proposition to make the necessity land on mechanism,
not adjectives: write-time ingestion is the trust boundary query-time guardrails
structurally cannot see; an OKF/LLM-wiki has no format-level authenticity, so the
ingestion pipeline IS the trust boundary. Add a first-class 'OKF / LLM-wiki
support (shipped)' section for import_bundle mode-b (per-concept gates). Fix stale
test badge (357 -> 522) and drop the brittle module count. Tighten prose; retain
all honest-limitations items (a shipped control). Every symbol/preset/command
verified against v0.2 code.
2026-07-15 18:58:21 +02:00
a1f3fe1983 docs(adoption): reusable consumer adoption brief for OKF second-brain repos
Self-contained brief a consumer repo can plan an inclusion from: what the
guard is (write-time, not query-time), the two bookends + 8-step contract,
the shipped OKF adapter (import_bundle mode-b, per-concept gates), how to
verify (coverage matrix -> 126 classes), how to depend (stdlib-only core),
and a planning checklist for WHEN/WHERE to wire it (untrusted boundary, not
first-party onboarding). Every claim verified against v0.2 code.
2026-07-15 12:35:09 +02:00
f5eae9a16e test(coverage): runnable threat-coverage matrix (real-case validation gate)
Add a single declarative manifest proving, in one place, every vulnerability
class the guard stops — and the documented gaps it does not. This is the
real-case validation gate ahead of any v1.0 freeze (v1.0 stays parked until
verified on real cases).

- src/llm_ingestion_guard/coverage.py: stdlib-only manifest (CORE_CASES) +
  narrated runner. `python -m llm_ingestion_guard.coverage` prints
  class -> OWASP -> expected -> observed -> verdict; exit 0 iff every caught
  class is caught and every documented gap holds. 126 caught classes + 4 gaps.
  Lexicon cases are generated from load_lexicon() via a payload dict, so a
  pattern with no payload fails loudly at import (self-verifying).
- tests/test_coverage_matrix.py: asserts total recall, that every documented
  gap still holds, and completeness (every lexicon id + every OWASP anchor has
  a case). Adds the full 25-pattern LLM02 secret-egress set (fixtures assembled
  from split tokens so no secret shape sits in source) and the container-layer
  front-end classes (CSV formula-injection, zip-slip, zip-bomb, symlink).
- README + CHANGELOG: point to the runnable matrix.

+165 tests (357 -> 522). No core dependency added.
2026-07-15 11:20:22 +02:00
66f3cbf4f5 docs(pdf): concede .pdf as a deliberate design boundary (Session F1)
Reframe .pdf from a 'known gap'/TODO to a deliberate concession across the
honest-limits and OKF-upload docs. A top-level .pdf drop is already refused as
an unsupported format (inbox_frontend.py raises on the else branch), not
half-scanned; adding a PDF parser + reportlab (solely to author white-on-white
test fixtures) is disproportionate for a dev-scoped showcase, and the OCR /
font-render stego carriers a PDF would smuggle are out of scope regardless.

- README honest-limits: .pdf = concession, not TODO; only the numeric CSV FP
  remains a known gap.
- docs/PLAN.md: upload table .pdf row marked 'conceded'; honest-scope paragraph
  names .pdf; assertions tightened to 'every accepted format'.

Closes the last format gap before v1.0 freeze (Session G). No code touched;
357 tests green.
2026-07-15 10:15:39 +02:00
1625f3893b docs: version-sync + SECURITY/CONTRIBUTING + honest-limits (Session E)
- README: tests badge 275->357; status v0.1->v0.2 (repo is 0.2.0; the v1.0
  bump belongs to the Session G freeze, not this docs pass); add three
  honest-limits — lone-HIGH-in-trusted-prose->WARN, vacuous quarantine-floor,
  Cyrillic/Latin homoglyph-mix false positive.
- docs/BRIEF.md: drop "No code yet" pre-implementation framing -> implemented v0.2.
- docs/OKF-INGESTION-BRIEF.md 4: correct cross-link control language —
  absolute https / references/ targets are spec-permitted, not rejected.
- Add SECURITY.md (private Forgejo disclosure) + CONTRIBUTING.md (stdlib-only
  core, Iron-Law TDD, no trailers, Forgejo-only invariants).
2026-07-15 10:08:24 +02:00
ee402e4ea8 refactor(calibration): consolidate tunable thresholds into calibration.py
Session D: move every calibration constant (entropy floors 5.4/128, 5.1/64,
4.7/40 + shape floors; MAX_SCAN_CHARS; rot13-min; cognitive-load lengths
2000/2500; disposition ranks; active-content severities) into one documented
calibration.py, so a parallel Node/TS port can mirror exactly the same numbers.

Pure refactor, zero behavior change: calibration is a leaf module (imports only
report.Severity) that entropy/lexicon/disposition/active_content now source
their thresholds from. MAX_SCAN_CHARS is re-exported from lexicon so output.py
and existing callers are unaffected. The 347 pre-existing tests pass unmodified;
new test_calibration.py freezes the values and asserts each detector actually
reads its threshold from calibration (identity-checked, not a dead copy).
2026-07-15 09:44:53 +02:00
4a9cfd2bbe docs: reframe novelty claim to composite write-time contract (review MAJOR #3)
Replace the unverified/absolute novelty statement with the defensible
four-part-contract form, verified against a focused adversarial PyPI+GitHub
survey (2026-07-15):

- BRIEF §11: 'assumed, not verified' -> verified survey with sources. Names
  aig-guardian (real, query-time; blurs only the minimal-dep-library
  differentiator), GuardLLM (nearest neighbour, runtime hardening, no
  scan-before-persist / capability isolation / fail-secure), and ipi-scanner
  (orphaned placeholder repo, recorded for honesty not as prior art).
- README: differentiator moved from 'library vs hosted/model' to the full
  four-part write-time contract.
- PLAN §27-31: drop the unverifiable 'the first' superlative.

Also promotes the v1.0 session plan (PLAN-v1.md) and the cross-model review
(review-2026-07.md) into docs/ on the open/ mirror, referenced by PLAN.md's
re-sequencing addendum.
2026-07-15 09:22:38 +02:00
f4e89d2885 feat(egress): decode-rescan feeds base64 plaintext to secret-egress (review MINOR)
Output gate step 3 now runs scan_secret_egress over every decoded base64
blob's plaintext, not only scan_lexicon. A base64-wrapped credential that
formerly vanished (decode fed the lexicon, which has no secret patterns)
now surfaces as decoded:egress:* carrying the blob offset. Evidence stays
length-only, so the decoded finding never leaks the secret value.

Hex-wrapped secrets remain a documented honest-limit (entropy exposes
decoded plaintext for base64 only). README honest-limits + CLAUDE.md
Kontekst updated; 3 tests added (347 passed, was 344).
2026-07-15 07:11:29 +02:00
0772dafb70 feat(okf): scan reserved index.md/log.md in mode-b import, not path-reject (review MAJOR #2)
A received OKF bundle MAY legitimately carry index.md (directory listing, read
first under progressive disclosure) and log.md (update history) at any level
(spec §3.1/§6/§7). import_bundle previously hard-rejected those basenames in the
T4 path gate, so a conformant third-party bundle was over-blocked in full
(FAIL_SECURE) — and because the reject fired before scan_concept, index.md's
body (the highest-priority injection surface) was never scanned.

import_bundle now defaults allow_reserved=True: reserved basenames are scanned
as structural files (path-safety checks — traversal / absolute / backslash / .md
— still apply). The shadow-reject (an *upload* masquerading as index.md) is
preserved: the front-end passes allow_reserved=False so a materialized upload
landing on a reserved basename is still refused. That front-end opt-in was
required to keep the shadow-reject once the default flipped (not in the plan's
Filer set; traced from the code).

- okf.py: validate_concept_path/_validate_concept/import_bundle gain the
  keyword; validate_concept_path default stays False (strict standalone).
- tests: +3 (legit index/log admit; injection in index.md body caught;
  okf_version frontmatter admits). Per-concept-iteration test switched to a
  traversal vector; mode-b showcase's index.md surface reframed from
  reserved-name-reject to index.md-body-scan.
- README honest-limits + CLAUDE.md context note the mode-b/upload distinction.

Suite: 341 -> 344 passed. Core invariant intact (dependencies=[]).
2026-07-15 06:43:50 +02:00
4d53765c63 feat(guard): active-content detector wired into the output gate (review MAJOR #1)
Close the EchoLeak wiring hole (CVE-2025-32711 class): markdown images/
links, reference definitions, autolinks, raw active HTML and data: URIs
now surface as report-only findings (active:*, OWASP LLM05) in
scan_output step 6, so screen_output and okf.import_bundle dispose of
them instead of admitting them with findings=[].

- new active_content.py: canonical home of the shared pattern table +
  scan_active_content; neutralize refactored to import it (mutating API
  and behavior unchanged, all neutralize tests pass as-is)
- images/links flagged only for absolute/protocol-relative URLs:
  relative in-bundle links are legitimate wiki/OKF mechanism (principle 5)
- evidence carries defanged URLs only (hxxps://evil[.]example)
- EchoLeak vectors planted in both showcases; detach proofs cover them
- README export list + checklist step 6, CLAUDE.md context line updated

Suite: 321 -> 341 passed. Core invariant intact (dependencies=[]).
2026-07-15 06:11:33 +02:00
31166d0af0 docs(readme): drop .xlsx from known gaps (covered in 2h) 2026-07-07 07:41:00 +02:00
ca26e117ea feat(inbox): .xlsx extraction — formula gate, hidden sheets, cell comments (stage 2h) 2026-07-07 07:41:00 +02:00
a46a96db0a docs(readme): drop docx-tables/pptx-groups from known gaps (covered in 2g) 2026-07-07 07:32:38 +02:00
6f32e70c6d feat(inbox): docx table cells + grouped pptx shapes (stage 2g)
Two structural regions the office extractors missed, no new dependency:
- .docx table cells (they live outside doc.paragraphs);
- grouped .pptx shapes (add_group_shape moves a shape inside the group, so the
  extractor recurses through MSO_SHAPE_TYPE.GROUP to reach it).

Each proven by an injection-in-region REJECT plus a clean-region ADMIT. Tests
310 -> 314.
2026-07-07 07:32:10 +02:00
abcdfa3663 docs(readme): honest-limits for the two-stage upload inbox (stage 2f)
Concede the binary layer that survives text extraction as out-of-scope (a
shipped control): VBA/macros, OLE/embedded objects, OCR-needed image text,
font/render stego, encrypted files. Point to the two-stage OKF inbox showcase as
the worked example, and list the known gaps (.xlsx/.pdf deferred, .docx tables,
grouped .pptx shapes, numeric CSV -/+ FP).
2026-07-06 11:18:07 +02:00
d8a95e465f feat(inbox): .pptx extraction — speaker notes, off-slide box, alt-text (stage 2e)
The pptx payload hides where an audience watching the slides does not look. The
extractor surfaces all three regions into the concept text so the stage-2 scan
catches the injection:

- speaker notes (slide.notes_slide.notes_text_frame.text);
- off-slide (off-canvas) text boxes (shape.text_frame.text, position-agnostic);
- image/shape alt-text (cNvPr@descr, read off the XML — python-pptx 1.0.2 has no
  stable public accessor across shape types).

Detach-proof: same visible slide without notes ADMITs. python-pptx imported
lazily (dev/showcase-scoped, not a core dep). Tests 305 -> 310.
2026-07-06 11:16:59 +02:00
26a231d6c4 feat(inbox): .docx extraction — hidden runs, comments, core metadata (stage 2d)
The docx payload hides where a human reviewing the file in Word does not look.
The extractor surfaces all three regions into the concept text so the stage-2
scan catches the injection:

- hidden/vanish runs (w:vanish) — still runs, so paragraph.text includes them;
- review comments (doc.comments[].text);
- core metadata properties (subject/keywords/comments/title/category/author).

Detach-proof: the same visible body WITHOUT the hidden run ADMITs, so it is the
extractor surfacing the hidden region that caught it, not merely 'a docx'.
python-docx is imported lazily (dev/showcase-scoped, not a core dep). Tests
300 -> 305.
2026-07-06 11:13:52 +02:00
02d59efeb2 feat(inbox): .csv formula injection + folder walk (stage 2c)
- .csv: cells leading with =/+/-/@ (leading whitespace stripped first) are the
  CSV-injection/DDE vector the guard cannot recognize, so the front-end refuses
  them; the raw cell text is still materialized so a prompt-injection phrase in a
  cell is caught by the stage-2 scan (T1). Detach-proof: plain-cell version of
  the same file ADMITs. Numeric -/+ leads are the accepted FP (honest-limits).
- folder: walked member-by-member with relative paths preserved, so a reserved
  basename member (index.md) lands on the guard's T4 gate; symlinks refused.

Refactor: per-file dispatch shared by top-level drops and folder walk (strict
raises on unsupported top-level suffix, folder skips). Tests 293 -> 300.
2026-07-06 11:10:55 +02:00
52aa40b17a feat(inbox): .zip container threats — zip-slip, zip-bomb, symlink (stage 2b)
The front-end reads zip entries in memory (never extracts to disk), so it owns
the container caps while a traversal entry maps onto the guard's path gate:

- zip-slip: a '../../evil.md' entry materializes onto a traversal concept path
  (preserved verbatim, not normalized) -> stage-2 T4 -> FAIL_SECURE -> REJECT.
- zip-bomb: per-entry + per-archive uncompressed-size caps (OWASP LLM10) refuse
  an oversize entry before its bytes are read; a bounded read defends a lying
  header. Detach-proof: a generous cap admits the same archive, so the cap is
  load-bearing.
- symlink entry: refused at the front-end (no legitimate concept meaning).

Caps are kwargs on extract_inbox/receive (small in tests, generous by default).
Tests 288 -> 293.
2026-07-06 11:07:46 +02:00
24e57ca10b feat(inbox): two-stage upload front-end — text formats (stage 2a)
Realistic-upload showcase (PLAN §247), first slice. A stage-1 front-end reads
dropped files and materializes them into an OKF bundle {concept_path: text} +
provenance; receive() wires extract -> import_bundle -> verdict. This slice
covers .txt/.md (stdlib only); .zip/.csv/folder/.docx/.pptx follow.

- Front-end lives in tests/ (showcase/dev-scoped), core stays stdlib-only:
  dependencies=[] untouched; python-docx/python-pptx added to the [dev] extra
  (used from stage 2d/2e), never a public [extract] extra (not v1 per PLAN).
- .txt injection -> guard T1 -> REJECT; clean .txt -> ADMIT; a dropped .md keeps
  its frontmatter so a dangerous value -> T2 REJECT; an index.* upload
  materializes onto the reserved uploads/index.md -> T4 REJECT.
- Detach-proof: neuter extraction to an empty bundle -> the poisoned upload
  ADMITs, proving the verdict depends on extraction carrying the payload.

Tests 282 -> 288.
2026-07-06 11:04:23 +02:00
0061c42f0b test(okf): OKF inbox showcase — mode-b receive/quarantine gate, end-to-end
The OKF analogue of tests/test_showcase.py: one received external bundle plants
one attack per OKF surface (T1 body + frontmatter-description, T3 non-https
resource, T4 path-traversal + reserved-name, T2 dangerous frontmatter value,
T5a dangerous-scheme link, §7.2 dangling link, homoglyph-obfuscated body) run
through the public okf surface as an upload-inbox consumer would compose it.
Aggregate fails secure -> REJECT; a clean bundle admits (WARN); the log marks
rejected concepts. Detach-proof: an always-admit gate catches none of the
surfaces, so the assertions have teeth. Stream-2 centrepiece (PLAN §212);
Stage-1 realistic upload extraction (§247) is the next increment.

Tests 277 -> 282.
2026-07-06 10:49:01 +02:00
896ab4034a docs(plan): showcase covers realistic upload formats — two-stage extract→guard, dev-scoped parsers, honest binary-layer limits 2026-07-06 10:18:43 +02:00
ba7fc68c05 docs(plan): re-sequence v0.2+ streams — defer consumer integration, add OKF inbox showcase as next build
Ground-truth pass over the named flagship consumer (portfolio-optimiser's
'OKF-upload-inbox') found it does not exist as a seam: both optimiser siblings
are frozen at release, carry their own OKF layer, receive no external bundles,
and take no dependency on this guard. Defer consumer integration (stream 2)
until the guard is mature; mature it here first, Node-port-friendly. Next
concrete build is the in-repo OKF inbox showcase (mode-b import_bundle as a
receive/quarantine gate), spec'd under 'The OKF inbox showcase'.
2026-07-06 10:12:50 +02:00
542ac92349 release: v0.2.0 — OKF adapter (stream 1); version sync pyproject/__version__/badge/CHANGELOG v0.2.0 2026-07-06 09:45:53 +02:00
07e0b2153a feat(okf): wire adapter into public API — import_bundle carries link graph, package exposes okf namespace (TDD, +2) 2026-07-06 09:44:41 +02:00
b80c896fd9 docs(changelog): release v0.1.0 + Unreleased OKF adapter (stream 1); test badge 214->275 2026-07-06 09:37:38 +02:00
30aa0a42a1 feat(okf): in-import cross-link graph — extract/resolve/reject links, dangling-link signal (T5a/A, TDD, +10) 2026-07-06 09:35:59 +02:00
320a40244f feat(okf): bundle-import iterator (mode b) — per-concept validate+stamp, aggregate disposition, log.md (T7, TDD, +8) 2026-07-06 09:32:50 +02:00
eac3c91b89 feat(okf): provenance stamping — origin/channel -> trust/disposition per concept, log.md entries (T6, TDD, +5) 2026-07-06 07:53:14 +02:00
f9a89938b4 feat(okf): resource-URL https allowlist reject-gate (T3, TDD, +10) 2026-07-06 07:43:37 +02:00
ec121f3259 feat(okf): path/reserved-name validation gate — traversal + index.md/log.md shadow (T4, TDD, +10) 2026-07-06 07:40:56 +02:00
f8bc5db547 feat(okf): whole-concept scan surface — frontmatter values + resource + body (T1, TDD, +6) 2026-07-06 07:39:18 +02:00