llm-ingestion-pipeline-secu.../CHANGELOG.md

96 lines
5.4 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [0.2.0] — 2026-07-06
### Added — OKF adapter (stream 1)
An OKF (Google Open Knowledge Format v0.1) adapter *on top of* the
format-agnostic core (`llm_ingestion_guard.okf`). The core stays `text ->
findings`; the adapter knows OKF structure and routes scannable regions into the
existing machinery. All TDD (failing test first), +61 tests. Verified against the
OKF `SPEC.md` (2026-07-06). See `docs/OKF-INGESTION-BRIEF.md` §8.
- `parse_frontmatter` — strict, reject-by-default frontmatter loader; refuses
anchors, aliases, explicit tags, merge keys, block scalars and flow collections
by construction, so YAML anchor/alias DoS and `!!python/object` coercion cannot
occur (not a general YAML parser, by design). (T2)
- `scan_concept` — whole-concept scan surface: frontmatter values (incl.
`description`, read first under progressive disclosure), `resource` and body all
go through `scan_output`. (T1)
- `validate_concept_path` — path / reserved-name gate: rejects `..` traversal,
absolute paths and `index.md` / `log.md` shadowing; returns the concept-ID. (T4)
- `validate_resource_url``resource` https allowlist: rejects non-https before
commit (reject, not defang — the format imposes no scheme constraint itself). (T3)
- `stamp_concept` / `format_log_entry` — provenance stamping: origin × channel →
trust × disposition per concept, emitted as `log.md` lines. Trust follows the
origin, never the insertion channel. (T6)
- `import_bundle` — received-bundle iterator (mode b): validates each concept
(path, frontmatter, resource, scan, stamp) independently; one bad concept is
rejected fail-secure while the rest are still checked; the aggregate disposition
is the most severe. (T7)
- `link_graph` / `resolve_link` / `extract_link_targets` — in-import cross-link
graph: resolves `.md` links (bundle-absolute or relative) to concept-IDs, flags
dangling links (the §7.2 dormant-injection signal) and rejects dangerous-scheme
or bundle-escaping targets. (T5a)
### Deferred
- Cross-run persisted link graph (T5b) — catching a link planted in one run whose
poisoned target is written in a *later* run (§7.2) needs durable graph state
whose storage/ownership depends on the consuming pipeline. Deferred to the
consumer-wiring stream; cross-run dormant links remain a documented residual
risk (README honest-limitations).
## [0.1.0] — 2026-07-06 (alpha)
The stdlib-only core, built test-first (TDD) per `docs/PLAN.md`. Tagged `v0.1.0`.
### Added
- `report` — shared `Finding` / `Report` / `Severity` / `Source` types.
- `sanitize` — carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
`data:`); byte-identical on clean input.
- `entropy` — Shannon / base64-like / hex-blob detection; base64 decode-and-rescan.
- `lexicon` — JSON pattern data + loader; raw/normalized/homoglyph/rot13 variants;
ReDoS-bounded, size-capped.
- `fence` — randomized per-call spotlight delimiter; attacker marker-strip.
- `neutralize` — opt-in defang of active-content output (byte-identical when clean).
- `output` — compose lexicon + entropy + decode-rescan over emitted text; secret
egress patterns (OWASP LLM02); report-only, never mutates.
- `disposition` — WARN | QUARANTINE_REVIEW | FAIL_SECURE under a source-trust
policy; compound-signal escalation; fail-**closed** when the scanner errors.
- `contract` — write-time asserters that raise: `assert_tool_less`,
`assert_credential_allowlist`, `scoped_env`.
- `grounding` — the `SourceGroundingCheck` seam for semantic poisoning (interface
only; `[judge]` implementation plugs in behind an extra).
- Top-level wiring — the `prepare_input` / `screen_output` §6 bookends plus the
full public surface; end-to-end showcase and adversarial + false-positive corpora.
### Security
Pre-release hardening from an independent adversarial review (all TDD, failing
test first):
- `entropy` — decode-and-rescan now runs **before** false-positive suppression,
so an injection blob prefixed with an SRI/media marker (to dodge the entropy
finding) is still decoded and rescanned by the lexicon. Suppression gates only
the entropy finding, never the decode.
- `output` — the invisible-carrier invariant now holds on the persist gate:
`scan_output` flags zero-width / BIDI presence (`output:zero-width-present`,
`output:bidi-present`) and `disposition` treats those plus
`lexicon:unicode-tags-present` as any-tier carriers, so a carrier in model
output fails secure even under a trusted policy.
- `contract``assert_credential_allowlist` catches a bare `<PROVIDER>_KEY`
(e.g. `STRIPE_KEY`); the previous regex silently missed it (fail-open). The
rule is deliberately broad (also flags `PARTITION_KEY`/`SORT_KEY` as loud,
allowlistable false positives) — fail-loud beats fail-silent for isolation.
- `disposition``guard` runs `decide` inside its guarded block, so a malformed
report can no longer escape the fail-closed guarantee.
- `output` — secret-egress placeholder suppression anchors word markers
(`example`, `todo`, …) to a word boundary, so a real secret that merely
*contains* such a word is no longer suppressed (fail-open egress miss closed).