Commit graph

48 commits

Author SHA1 Message Date
f4e89d2885 feat(egress): decode-rescan feeds base64 plaintext to secret-egress (review MINOR)
Output gate step 3 now runs scan_secret_egress over every decoded base64
blob's plaintext, not only scan_lexicon. A base64-wrapped credential that
formerly vanished (decode fed the lexicon, which has no secret patterns)
now surfaces as decoded:egress:* carrying the blob offset. Evidence stays
length-only, so the decoded finding never leaks the secret value.

Hex-wrapped secrets remain a documented honest-limit (entropy exposes
decoded plaintext for base64 only). README honest-limits + CLAUDE.md
Kontekst updated; 3 tests added (347 passed, was 344).
2026-07-15 07:11:29 +02:00
0772dafb70 feat(okf): scan reserved index.md/log.md in mode-b import, not path-reject (review MAJOR #2)
A received OKF bundle MAY legitimately carry index.md (directory listing, read
first under progressive disclosure) and log.md (update history) at any level
(spec §3.1/§6/§7). import_bundle previously hard-rejected those basenames in the
T4 path gate, so a conformant third-party bundle was over-blocked in full
(FAIL_SECURE) — and because the reject fired before scan_concept, index.md's
body (the highest-priority injection surface) was never scanned.

import_bundle now defaults allow_reserved=True: reserved basenames are scanned
as structural files (path-safety checks — traversal / absolute / backslash / .md
— still apply). The shadow-reject (an *upload* masquerading as index.md) is
preserved: the front-end passes allow_reserved=False so a materialized upload
landing on a reserved basename is still refused. That front-end opt-in was
required to keep the shadow-reject once the default flipped (not in the plan's
Filer set; traced from the code).

- okf.py: validate_concept_path/_validate_concept/import_bundle gain the
  keyword; validate_concept_path default stays False (strict standalone).
- tests: +3 (legit index/log admit; injection in index.md body caught;
  okf_version frontmatter admits). Per-concept-iteration test switched to a
  traversal vector; mode-b showcase's index.md surface reframed from
  reserved-name-reject to index.md-body-scan.
- README honest-limits + CLAUDE.md context note the mode-b/upload distinction.

Suite: 341 -> 344 passed. Core invariant intact (dependencies=[]).
2026-07-15 06:43:50 +02:00
4d53765c63 feat(guard): active-content detector wired into the output gate (review MAJOR #1)
Close the EchoLeak wiring hole (CVE-2025-32711 class): markdown images/
links, reference definitions, autolinks, raw active HTML and data: URIs
now surface as report-only findings (active:*, OWASP LLM05) in
scan_output step 6, so screen_output and okf.import_bundle dispose of
them instead of admitting them with findings=[].

- new active_content.py: canonical home of the shared pattern table +
  scan_active_content; neutralize refactored to import it (mutating API
  and behavior unchanged, all neutralize tests pass as-is)
- images/links flagged only for absolute/protocol-relative URLs:
  relative in-bundle links are legitimate wiki/OKF mechanism (principle 5)
- evidence carries defanged URLs only (hxxps://evil[.]example)
- EchoLeak vectors planted in both showcases; detach proofs cover them
- README export list + checklist step 6, CLAUDE.md context line updated

Suite: 321 -> 341 passed. Core invariant intact (dependencies=[]).
2026-07-15 06:11:33 +02:00
31166d0af0 docs(readme): drop .xlsx from known gaps (covered in 2h) 2026-07-07 07:41:00 +02:00
ca26e117ea feat(inbox): .xlsx extraction — formula gate, hidden sheets, cell comments (stage 2h) 2026-07-07 07:41:00 +02:00
a46a96db0a docs(readme): drop docx-tables/pptx-groups from known gaps (covered in 2g) 2026-07-07 07:32:38 +02:00
6f32e70c6d feat(inbox): docx table cells + grouped pptx shapes (stage 2g)
Two structural regions the office extractors missed, no new dependency:
- .docx table cells (they live outside doc.paragraphs);
- grouped .pptx shapes (add_group_shape moves a shape inside the group, so the
  extractor recurses through MSO_SHAPE_TYPE.GROUP to reach it).

Each proven by an injection-in-region REJECT plus a clean-region ADMIT. Tests
310 -> 314.
2026-07-07 07:32:10 +02:00
abcdfa3663 docs(readme): honest-limits for the two-stage upload inbox (stage 2f)
Concede the binary layer that survives text extraction as out-of-scope (a
shipped control): VBA/macros, OLE/embedded objects, OCR-needed image text,
font/render stego, encrypted files. Point to the two-stage OKF inbox showcase as
the worked example, and list the known gaps (.xlsx/.pdf deferred, .docx tables,
grouped .pptx shapes, numeric CSV -/+ FP).
2026-07-06 11:18:07 +02:00
d8a95e465f feat(inbox): .pptx extraction — speaker notes, off-slide box, alt-text (stage 2e)
The pptx payload hides where an audience watching the slides does not look. The
extractor surfaces all three regions into the concept text so the stage-2 scan
catches the injection:

- speaker notes (slide.notes_slide.notes_text_frame.text);
- off-slide (off-canvas) text boxes (shape.text_frame.text, position-agnostic);
- image/shape alt-text (cNvPr@descr, read off the XML — python-pptx 1.0.2 has no
  stable public accessor across shape types).

Detach-proof: same visible slide without notes ADMITs. python-pptx imported
lazily (dev/showcase-scoped, not a core dep). Tests 305 -> 310.
2026-07-06 11:16:59 +02:00
26a231d6c4 feat(inbox): .docx extraction — hidden runs, comments, core metadata (stage 2d)
The docx payload hides where a human reviewing the file in Word does not look.
The extractor surfaces all three regions into the concept text so the stage-2
scan catches the injection:

- hidden/vanish runs (w:vanish) — still runs, so paragraph.text includes them;
- review comments (doc.comments[].text);
- core metadata properties (subject/keywords/comments/title/category/author).

Detach-proof: the same visible body WITHOUT the hidden run ADMITs, so it is the
extractor surfacing the hidden region that caught it, not merely 'a docx'.
python-docx is imported lazily (dev/showcase-scoped, not a core dep). Tests
300 -> 305.
2026-07-06 11:13:52 +02:00
02d59efeb2 feat(inbox): .csv formula injection + folder walk (stage 2c)
- .csv: cells leading with =/+/-/@ (leading whitespace stripped first) are the
  CSV-injection/DDE vector the guard cannot recognize, so the front-end refuses
  them; the raw cell text is still materialized so a prompt-injection phrase in a
  cell is caught by the stage-2 scan (T1). Detach-proof: plain-cell version of
  the same file ADMITs. Numeric -/+ leads are the accepted FP (honest-limits).
- folder: walked member-by-member with relative paths preserved, so a reserved
  basename member (index.md) lands on the guard's T4 gate; symlinks refused.

Refactor: per-file dispatch shared by top-level drops and folder walk (strict
raises on unsupported top-level suffix, folder skips). Tests 293 -> 300.
2026-07-06 11:10:55 +02:00
52aa40b17a feat(inbox): .zip container threats — zip-slip, zip-bomb, symlink (stage 2b)
The front-end reads zip entries in memory (never extracts to disk), so it owns
the container caps while a traversal entry maps onto the guard's path gate:

- zip-slip: a '../../evil.md' entry materializes onto a traversal concept path
  (preserved verbatim, not normalized) -> stage-2 T4 -> FAIL_SECURE -> REJECT.
- zip-bomb: per-entry + per-archive uncompressed-size caps (OWASP LLM10) refuse
  an oversize entry before its bytes are read; a bounded read defends a lying
  header. Detach-proof: a generous cap admits the same archive, so the cap is
  load-bearing.
- symlink entry: refused at the front-end (no legitimate concept meaning).

Caps are kwargs on extract_inbox/receive (small in tests, generous by default).
Tests 288 -> 293.
2026-07-06 11:07:46 +02:00
24e57ca10b feat(inbox): two-stage upload front-end — text formats (stage 2a)
Realistic-upload showcase (PLAN §247), first slice. A stage-1 front-end reads
dropped files and materializes them into an OKF bundle {concept_path: text} +
provenance; receive() wires extract -> import_bundle -> verdict. This slice
covers .txt/.md (stdlib only); .zip/.csv/folder/.docx/.pptx follow.

- Front-end lives in tests/ (showcase/dev-scoped), core stays stdlib-only:
  dependencies=[] untouched; python-docx/python-pptx added to the [dev] extra
  (used from stage 2d/2e), never a public [extract] extra (not v1 per PLAN).
- .txt injection -> guard T1 -> REJECT; clean .txt -> ADMIT; a dropped .md keeps
  its frontmatter so a dangerous value -> T2 REJECT; an index.* upload
  materializes onto the reserved uploads/index.md -> T4 REJECT.
- Detach-proof: neuter extraction to an empty bundle -> the poisoned upload
  ADMITs, proving the verdict depends on extraction carrying the payload.

Tests 282 -> 288.
2026-07-06 11:04:23 +02:00
0061c42f0b test(okf): OKF inbox showcase — mode-b receive/quarantine gate, end-to-end
The OKF analogue of tests/test_showcase.py: one received external bundle plants
one attack per OKF surface (T1 body + frontmatter-description, T3 non-https
resource, T4 path-traversal + reserved-name, T2 dangerous frontmatter value,
T5a dangerous-scheme link, §7.2 dangling link, homoglyph-obfuscated body) run
through the public okf surface as an upload-inbox consumer would compose it.
Aggregate fails secure -> REJECT; a clean bundle admits (WARN); the log marks
rejected concepts. Detach-proof: an always-admit gate catches none of the
surfaces, so the assertions have teeth. Stream-2 centrepiece (PLAN §212);
Stage-1 realistic upload extraction (§247) is the next increment.

Tests 277 -> 282.
2026-07-06 10:49:01 +02:00
896ab4034a docs(plan): showcase covers realistic upload formats — two-stage extract→guard, dev-scoped parsers, honest binary-layer limits 2026-07-06 10:18:43 +02:00
ba7fc68c05 docs(plan): re-sequence v0.2+ streams — defer consumer integration, add OKF inbox showcase as next build
Ground-truth pass over the named flagship consumer (portfolio-optimiser's
'OKF-upload-inbox') found it does not exist as a seam: both optimiser siblings
are frozen at release, carry their own OKF layer, receive no external bundles,
and take no dependency on this guard. Defer consumer integration (stream 2)
until the guard is mature; mature it here first, Node-port-friendly. Next
concrete build is the in-repo OKF inbox showcase (mode-b import_bundle as a
receive/quarantine gate), spec'd under 'The OKF inbox showcase'.
2026-07-06 10:12:50 +02:00
542ac92349 release: v0.2.0 — OKF adapter (stream 1); version sync pyproject/__version__/badge/CHANGELOG v0.2.0 2026-07-06 09:45:53 +02:00
07e0b2153a feat(okf): wire adapter into public API — import_bundle carries link graph, package exposes okf namespace (TDD, +2) 2026-07-06 09:44:41 +02:00
b80c896fd9 docs(changelog): release v0.1.0 + Unreleased OKF adapter (stream 1); test badge 214->275 2026-07-06 09:37:38 +02:00
30aa0a42a1 feat(okf): in-import cross-link graph — extract/resolve/reject links, dangling-link signal (T5a/A, TDD, +10) 2026-07-06 09:35:59 +02:00
320a40244f feat(okf): bundle-import iterator (mode b) — per-concept validate+stamp, aggregate disposition, log.md (T7, TDD, +8) 2026-07-06 09:32:50 +02:00
eac3c91b89 feat(okf): provenance stamping — origin/channel -> trust/disposition per concept, log.md entries (T6, TDD, +5) 2026-07-06 07:53:14 +02:00
f9a89938b4 feat(okf): resource-URL https allowlist reject-gate (T3, TDD, +10) 2026-07-06 07:43:37 +02:00
ec121f3259 feat(okf): path/reserved-name validation gate — traversal + index.md/log.md shadow (T4, TDD, +10) 2026-07-06 07:40:56 +02:00
f8bc5db547 feat(okf): whole-concept scan surface — frontmatter values + resource + body (T1, TDD, +6) 2026-07-06 07:39:18 +02:00
22e65dcec5 feat(okf): strict reject-by-default frontmatter parser (T2, TDD, +12) 2026-07-06 07:36:14 +02:00
525eb194f5 docs(readme): surface OKF residual risks §7.2 dormant-link + §7.4 own-security-content (T8) 2026-07-06 07:32:20 +02:00
43368e9684 docs(plan): lock v0.2+ stream sequencing (OKF->consumers->Node) + dependency rationale 2026-07-06 07:30:09 +02:00
df30c7bcce docs(readme): correct test badge 201->214 before v0.1.0 tag v0.1.0 2026-07-06 07:12:52 +02:00
5cd9b451fb docs(okf): capture OKF-ingestion security brief as v0.2 future work
Google's Open Knowledge Format (OKF v0.1) is the LLM-wiki pattern this
library guards. Capture a security brief for OKF ingestion as tracked
future work -- NOT shipped scope. The v0.1 core is format-agnostic text;
OKF surfaces beyond the body (YAML frontmatter, resource URLs, cross-link
graph, file path / reserved names, format-level authenticity) are uncovered.

- Coverage claims verified against the code at 5397ba1 (brief section 9): no
  YAML parse in src, neutralize defangs but has no reject-gate, no path/graph
  logic, disposition machinery exists.
- OKF confirmed real at the format level (Google Cloud, 2026-06-12; spec
  GoogleCloudPlatform/knowledge-catalog).
- Design move: an OKF adapter ON TOP of the format-agnostic core; the core
  stays text->findings.
- Section 8 lists 8 v0.2 tasks (T1-T8); T8 (surface two OKF residual risks in
  README honest-limits) can land independently.
2026-07-06 07:07:09 +02:00
5397ba15a1 fix(security): harden 5 adversarial-review findings (M1/M2/M3 + m4/m6) via TDD
Pre-release hardening from an independent adversarial review; each fixed
test-first (failing test -> fix -> green). 214 tests pass.

- entropy (M1): decode-and-rescan now runs BEFORE false-positive suppression,
  so an SRI/media-prefixed injection blob is still decoded and lexicon-rescanned.
  Suppression gates only the entropy finding, never the decode.
- output/disposition (M3): the invisible-carrier invariant now holds on the
  persist gate. scan_output flags zero-width/BIDI presence and disposition
  treats those + lexicon:unicode-tags-present as any-tier carriers, so a carrier
  in model output fails secure even under a trusted policy.
- contract (M2): assert_credential_allowlist catches a bare <PROVIDER>_KEY
  (e.g. STRIPE_KEY) that the old regex silently missed (fail-open). Deliberately
  broad: also flags PARTITION_KEY/SORT_KEY as loud, allowlistable FPs -- fail-loud
  beats fail-silent for an isolation control.
- disposition (m6): guard runs decide inside its guarded block -> total
  fail-closed even on a malformed report.
- output (m4): egress placeholder suppression anchors word markers (example,
  todo, ...) to a word boundary, closing a fail-open where a real secret merely
  containing such a word was suppressed.

Docs: CHANGELOG Security subsection; README honest-limit for lexicon dedup (m5,
documented tradeoff, not fixed).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-05 10:45:05 +02:00
86726ed109 docs(readme): add house-style shields.io badges (version/status/python/tests/license)
Mirror the badge convention from the ms-ai-architect plugin README — shields.io
STATIC badges (img.shields.io/badge/), which work on Forgejo where dynamic
build-status endpoints do not. Type-adapted for a library rather than a plugin:
the Platform slot becomes python-3.10+, the content-count slot becomes
tests-201-passing, plus an honest status-alpha badge.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-05 07:05:16 +02:00
a44cf3c7b7 test(showcase): end-to-end multi-vuln pipeline + adversarial/FP corpora (recall)
Module 11's final deliverable (BRIEF §9, PLAN §132), built last. Exercises the
full public toolkit as a consumer would compose it — the _ingest helper doubles
as the README worked example.

- test_showcase: ONE poisoned artifact planting a vulnerability per detection
  channel — 5 invisible carriers (zero-width/BIDI/unicode-tag/html-comment/
  data-uri), injection across raw + homoglyph + rot13 + whole-string-base64
  (decode-and-rescan) channels with DISTINCT lexicon ids so dedup won't merge
  them, plus a credential the mock transform leaks into its output (LLM02). All
  10 planted labels caught; disposition FAIL_SECURE; egress evidence never
  carries the secret value; a clean document + clean transform WARNs (no FP).
- test_corpus: adversarial recall == 100% over the planted classes (a silent
  drop is a regression), and a false-positive corpus of attack-resembling legit
  content (injection vocabulary, secret-shaped placeholders/varrefs, high-entropy
  checksums) that disposes WARN under a trusted source — most producing NO
  finding (the suppression rules hold). One MEDIUM homoglyph case proves hard-fail
  is an explicit opt-in: same finding WARNs trusted, QUARANTINEs under upload.

Secret fixtures assembled from fragments (gitleaks-safe); every payload verified
against the real detectors before assertion. 12 new tests; 201 green total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 23:34:51 +02:00
0af8f68cae feat(wiring): §6 bookends (prepare_input/screen_output) + full public API + README v0.1 (TDD)
Module 11 (final) — the top-level wiring. The library never makes the model call
(no SDK imported by the core), so the public surface is the toolkit plus two
library-side bookends around the caller's tool-less transform (Form 1, chosen
with the operator over an export-only toolkit and a full orchestrator — the
bookends fit existing pipelines with least friction, encode the two halves the
library can stand for, and impose no control flow):

- prepare_input(text, source=INPUT) -> PreparedInput(fenced, nonce, report):
  §6 steps 1-2, sanitize THEN fence (carrier can never smuggle a forged
  delimiter). Merged report carries both steps' findings; renders no disposition.
- screen_output(text, policy, *, provenance, transform_failed) -> DispositionResult:
  §6 steps 6-7, scan_output under guard() so a scanner error fails CLOSED
  (FAIL_SECURE, never a silent persist). transform_failed routes the compound
  forced-fallback rule.
- __all__ exports the full framework-agnostic surface: detectors, result types,
  disposition machinery + presets, contract asserters, the grounding seam.

Docs: README refreshed from the stale "brief / pre-implementation" line to a v0.1
alpha status with a Form-1 quickstart, the §6 adopt-this checklist, and an honest
-limitations section (structural unsolvability at the text layer; semantic
poisoning invisible to lexicon+entropy; text-only, no multimodal). CHANGELOG
seeded; CLAUDE.md remote/status lines corrected (remote IS set, no longer
brief-stage).

10 wiring tests (public surface, prepare_input compose, screen_output fail-closed
+ compound). 189 green (showcase + corpora follow).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 23:34:28 +02:00
1e63643157 feat(grounding): SourceGroundingCheck protocol + pass-through default — the semantic-poisoning seam (TDD) [skip-docs]
Module 10 of the build order — interface only in the stdlib core (PLAN §67/§94,
BRIEF §7/§10). lexicon and entropy see surface signals; they are structurally
blind to semantic poisoning — a factually false or subtly biased claim, in clean
prose, carrying no suspicious token. Catching that needs grounding against a
source of truth (embedding classifier, retrieval check, LLM judge) — a model
call, and models never live in this core.

So this module ships exactly two things and no detector logic:
- SourceGroundingCheck: the text -> Report protocol a [judge] implementation
  must satisfy to plug in. runtime_checkable for a coarse callable-vs-not
  isinstance gate; a conforming impl may be a plain function or a stateful
  callable holding a retriever/client — the grounding source of truth is the
  impl's concern, never a seam parameter. Its findings flow through disposition
  like any other detector's, with zero grounding-specific plumbing.
- no_grounding_check (bound as DEFAULT_GROUNDING_CHECK): the pass-through
  default returning an empty Report. The honest scope note made structural —
  semantic poisoning is not solved at the text layer, and the core does not
  pretend to.

Design choice, tested: the default returns NO finding (not an INFO "unchecked"
marker). An INFO finding would flip every artifact to found and trip the
quarantine_default floor in disposition, quarantining every upload for a check
that never ran. The seam stays silent; the honesty lives in the contract.

8 new tests (pass-through empties, silence on semantic poison, protocol
conformance incl. a stateful judge stub, and a composition test proving a
plugged-in check's HIGH finding fails secure through disposition unchanged);
169 green total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 22:44:41 +02:00
9b75bf0bb7 feat(contract): tool-less + credential-allowlist + env-scoping quarantine asserters (TDD) [skip-docs]
Module 9 of the build order — the differentiator. Where the other modules
detect and report, these are pure functions that ENFORCE an invariant and
raise: the write-time analogue of runtime least-privilege. They harden the
call a pipeline makes around its quarantined transform; the library makes no
model call itself.

Three asserts, matching the reusable-contract checklist (BRIEF §6 steps 3-4):
- assert_tool_less(request): the model request carries no tool surface —
  tools/functions/tool_choice/function_call/mcp_servers, populated. Covers
  both Anthropic and OpenAI request shapes; empty/None is genuinely tool-less.
- assert_credential_allowlist(env, allowed): the process env holds no
  credential beyond this stage's allowlist (subset = least-privilege; the
  enrichment stage sees only the model key, never the publish credential).
- scoped_env(env, allowed): the isolation primitive — strips off-allowlist
  credentials, keeps PATH etc.; the assert then passes by construction.

Generalized from claude-code-llm-wiki tools/wiki_ingest/enrich.py
assert_quarantine (a pipeline-specific gate) into framework-agnostic pieces.
Two deliberate safety choices inherited from the reference: credential
detection is name-based (env VALUES are never read — value-scanning is the
output module's job); and a raised ContractViolation names only the offending
KEY, never a value, so it is safe to route to an alert channel (minimal-alert,
BRIEF §6 step 8). CREDENTIAL_NAME_RE ported verbatim from the proven reference.

23 new tests; 161 green total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 20:44:20 +02:00
409b847c42 feat(disposition): source-trust + provenance gate — WARN|QUARANTINE|FAIL_SECURE, compound + fail-closed (TDD) [skip-docs] 2026-07-04 20:34:32 +02:00
19981623f5 feat(output): report-only OUTPUT gate — compose + secret egress (TDD) [skip-docs]
Module 7 of the build order: the last gate before model output is persisted.
Composes scan_lexicon + scan_entropy over the emitted text, feeds each base64
blob entropy decoded back through the lexicon (decode-and-rescan → decoded:*
findings with blob offset), and adds the LLM02 secret/credential egress layer
(cloud/provider keys, PEM headers, DB conn-strings, JWT, labelled
password/secret/api-key assignments) with placeholder/varref FP-suppression.

Report-only (never mutates; neutralize is the separate opt-in mutator). Evidence
carries only a description + match length — never the secret value. Self-safe:
single input-size cap, linear egress patterns. PEM patterns use -{5} form so the
module itself never trips a secret scanner. 26 new tests; 108 green total.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 20:18:46 +02:00
78c9f2f7f1 feat(neutralize): opt-in pure defang of active-content output (TDD) [skip-docs]
Build-order step 6. Close the EchoLeak class (CVE-2025-32711): active content in
persisted model OUTPUT that a downstream renderer auto-fetches or makes clickable,
exfiltrating data zero-click. These carriers are neither injection strings nor
high-entropy, so lexicon + entropy miss them — a distinct control (OWASP LLM05,
Improper Output Handling).

Defang, don't delete. URLs in active-content position are rewritten to a
non-resolvable but auditable form (https://evil.com -> hxxps://evil[.]com;
data:/javascript: colon neutralized to [:]); raw active HTML is escaped so a
renderer shows inert literal text. Visible information survives review; only the
machine-actionable affordance dies. Dot-defang is idempotent (never [[.]]).

Six classes, each a Finding: markdown-image (HIGH, the zero-click primitive),
inline-link (MEDIUM), reference-link definition (MEDIUM, the documented
image-filter bypass), angle-bracket autolink (MEDIUM), raw active HTML (HIGH,
inherently-active tag OR event/URL attribute — benign <b>/<em> left untouched),
standalone data: URI (HIGH). Processing order prevents double-counting.

Opt-in and separate: calling neutralize() IS the opt-in to mutate; the report-only
gate stays pure (design principles 3 & 4). Byte-identical on clean output, mirror
of the sanitizer invariant. Scope conceded in the docstring: a targeted defanger,
not a full HTML sanitizer.

17 tests: byte-identity + FP guards (lone <>[], metadata:, benign HTML), image
defang + non-resolvability, secret-exfil URL, inline/reference/autolink, raw-html
escape + script neutralization, data: URI, no double-count, counts, source.

[skip-docs]: README positioning + honest-limitations remains the deliberate
build-order step-11 deliverable (steps 1-5 likewise left README frozen).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HyRCQMocjZ6SmSQ6JidJ2k
2026-07-04 18:55:12 +02:00
5fb7e0c9fa feat(fence): randomized unspoofable delimiter + attacker marker-strip (TDD) [skip-docs]
Build-order step 5. Wrap untrusted content in a quarantine fence a downstream
trusted prompt can rely on: everything between the delimiters is data, never
instructions.

Two load-bearing properties:
- Per-call cryptographic nonce (secrets.token_hex, 128-bit) in the delimiter, so
  an attacker embedded in the payload cannot forge the matching closing marker to
  break out — the nonce is unpredictable and fresh every call.
- Marker-strip FIRST: fabricated fence markers already in the payload are removed
  (any/no nonce, case-insensitive, ReDoS-safe negated-class regex) and flagged as
  fence:marker-injection (HIGH, LLM01) before wrapping — defense in depth against
  a lucky guess of the static skeleton, and it surfaces the attempt.

Pure text -> (fenced_text, report, nonce); only mutation is the marker-strip.
Nonce exposed so the caller can reference the fence in the trusted prompt.
10 tests: wrap/preserve, per-call randomness, nonce length, breakout containment,
marker-strip (+ case-insensitive), prose-word FP guard, source, empty input.

[skip-docs]: README positioning + honest-limitations is a deliberate build-order
step-11 deliverable (steps 1-4 likewise left README frozen). README status line
("pre-implementation") is stale and flagged for the step-11 refresh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 18:23:35 +02:00
a55404460a docs(plan): add end-to-end showcase pipeline test as the final deliverable
One realistic content sample carrying many vulnerabilities at once, run through
a mock ingestion pipeline (sanitize -> lexicon+entropy+decode-rescan -> output
gate -> disposition); assert every planted vuln is caught and disposition fails
secure. Doubles as the README worked example. Added to build-order step 11 and
the test strategy; tracked as task #11. Inspiration: llm-security/examples/*.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 17:23:17 +02:00
f397cd94e1 feat(lexicon): JSON injection lexicon + variant-set scan (TDD) [skip-docs]
Build order step 4 — the load-bearing port from the llm-security seed
(injection-patterns.mjs + string-utils.mjs), stdlib-only.

- injection_lexicon.json: 83 patterns (CRITICAL/HIGH/HYBRID/MEDIUM) as the
  single source of truth (regex + id + severity + owasp + desc), compiled once
  by a thin loader. Decoupled from the engine for a future TS port.
- scan_lexicon(text, source, max_scan_chars) -> Report: matches every pattern
  against a deduped variant set (raw / normalized / homoglyph-folded / rot13),
  plus unicode-tag presence signal and the cognitive-load trap.
- normalize_for_scan chain ported: unicode-tags -> bidi -> HTML-entities ->
  unicode/hex/URL escapes -> whole-string base64 (reuses entropy.try_decode_base64)
  -> collapse letter-spacing; plus fold_homoglyphs / rot13.
- Self-safety (OWASP LLM10): input-size cap (scan prefix + flag oversize) and
  ReDoS-safe port — the two nested-.*? sub-agent patterns bounded to
  (?:\S+\s+){0,N}?; verified true positives still fire.
- Non-Latin data (homoglyph map, BIDI block) built from explicit code points;
  JSON non-ASCII kept as \uXXXX escapes.

24 tests; 55 green total.

[skip-docs]: README positioning + honest-limitations is a deliberate build-order
step-11 deliverable (steps 1-3 likewise left README frozen). README status line
("pre-implementation") is stale and flagged for the step-11 refresh.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 17:20:21 +02:00
e5e91df369 feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD)
Build order step 3. Pure text -> findings detector ported from the
llm-security entropy-scanner seed:

- Length-calibrated Shannon-entropy classification (CRITICAL 5.4/128,
  HIGH 5.1/64, MEDIUM 4.7/40).
- Shape floor: base64-like (len>100) / hex (len>64) reach at least MEDIUM
  even when entropy alone does not trigger — the only path that catches hex
  (16-symbol alphabet caps H at 4.0 < 4.7).
- Decode-and-rescan (must-have): base64 blobs that decode to printable text
  are exposed on EntropyResult.decoded for a later lexicon rescan.
- FP suppression scoped to the text-relevant subset (base64 media data-URI
  prefixes + SRI sha*- prefix); source-code-specific seed rules omitted.

Exposes ported primitives shannon_entropy / is_base64_like / is_hex_blob /
try_decode_base64. 16 new tests; full suite 31 green. Stdlib-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 09:43:24 +02:00
a9c4ccd8c7 feat: scaffold package + report and sanitize modules (TDD)
Build order steps 1-2 of docs/PLAN.md:
- pyproject.toml (llm-ingestion-guard, stdlib-only core, extras [ml]/[judge]/[dev]), LICENSE (MIT)
- report: Finding/Report/Severity/Source shared type (pure data)
- sanitize: carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
  data: URI) with the byte-identical / removes-only invariant
- docs/PLAN.md: v1 implementation plan (positioning A, gap-expanded scope,
  llm-security reuse map)

15 tests passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 09:24:20 +02:00
39991bd251 chore: repo scaffolding via /repo-init
Add .gitignore (Python stack + obligatory lines; /STATE.md kept local-only
until a private remote exists), CHANGELOG.md, and the Communication patterns
section in CLAUDE.md. STATE.md is gitignored — must never reach the public
open/ mirror.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
2026-07-04 08:39:23 +02:00
d983aa3c95 docs(brief): fold in ms-ai-architect Layer B learnings (first output-side impl)
Based on the working, tested Layer B ingestion-gate in ms-ai-architect (2026-07-04):
- A: correct consumer 2 — it fetches authored Learn docs + code samples via the
  microsoft-learn MCP, NOT the open Q&A forum / MSDN / Stack Overflow (that claim
  was unverified/overstated); low-trust surface is intra-document. Repo/name
  reconciled: the consumer is ms-ai-architect, no separate "MS AI Security plugin".
- B: §4.7 — trust tiers WITHIN a document (code sample / localized string vs
  authored prose), not only across sources; carriers + critical block in any tier.
- C: §12 — consume the lexicon as pure imported functions, not the llm-security
  scan CLI (which under-covers Markdown prose + base64-in-code-block, verified).
- D: §4.6 — fail-secure extends to scanner-unavailable: un-scannable ⇒ BLOCK.
Adds ms-ai-architect as the second reference implementation (output side).
2026-07-04 08:32:38 +02:00
41473d9db2 docs(brief): add MS AI Security plugin as second consumer; source-trust disposition
Second target consumer named: MS AI Security plugin, an ingestion
pipeline over Microsoft Learn content that includes user-generated Q&A
(learn.microsoft.com/answers, plus ingested MSDN/Stack Overflow). This
is the high-untrust case where the contract is load-bearing, not
hygiene. New design principle 4.7: disposition scales with source trust
(pinned changelog = WARN; open UGC = quarantine/hard-fail on high
severity). Day-1 rationale: architectural controls are cheap to design
in, expensive to retrofit (consumer 1 is proving that at its A13).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPAmFyEVWbwvmSNVdXTu4d
2026-07-04 06:26:55 +02:00
b505f70485 docs: design brief for reusable LLM ingestion-pipeline security library
Brief-stage repo. docs/BRIEF.md defines a minimal, framework-agnostic
library that packages the write-time ingestion contract (sanitize ->
fence -> tool-less quarantined transform -> per-stage capability
isolation -> scan output before persist -> fail-secure) as reusable
code. Positioned honestly against query-time guardrails (LLM Guard,
NeMo, Rebuff, Vigil) with a prior-art verification log. Reference
implementation: claude-code-llm-wiki Stage B.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HPAmFyEVWbwvmSNVdXTu4d
2026-07-04 06:18:32 +02:00