open/llm-ingestion-pipeline-security
1
0
Fork 0
Code Issues Pull requests Releases Activity
6 commits 1 branch 2 tags 992 KiB
Commit graph

2 commits

Author SHA1 Message Date
Kjell Tore Guttormsen
e5e91df369 feat(entropy): high-entropy/encoded-blob detection with decode-and-rescan (TDD) 
Build order step 3. Pure text -> findings detector ported from the
llm-security entropy-scanner seed:

- Length-calibrated Shannon-entropy classification (CRITICAL 5.4/128,
  HIGH 5.1/64, MEDIUM 4.7/40).
- Shape floor: base64-like (len>100) / hex (len>64) reach at least MEDIUM
  even when entropy alone does not trigger — the only path that catches hex
  (16-symbol alphabet caps H at 4.0 < 4.7).
- Decode-and-rescan (must-have): base64 blobs that decode to printable text
  are exposed on EntropyResult.decoded for a later lexicon rescan.
- FP suppression scoped to the text-relevant subset (base64 media data-URI
  prefixes + SRI sha*- prefix); source-code-specific seed rules omitted.

Exposes ported primitives shannon_entropy / is_base64_like / is_hex_blob /
try_decode_base64. 16 new tests; full suite 31 green. Stdlib-only.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
 2026-07-04 09:43:24 +02:00
Kjell Tore Guttormsen
a9c4ccd8c7 feat: scaffold package + report and sanitize modules (TDD) 
Build order steps 1-2 of docs/PLAN.md:
- pyproject.toml (llm-ingestion-guard, stdlib-only core, extras [ml]/[judge]/[dev]), LICENSE (MIT)
- report: Finding/Report/Severity/Source shared type (pure data)
- sanitize: carrier stripping (zero-width, BIDI, Unicode-tag, HTML comment,
  data: URI) with the byte-identical / removes-only invariant
- docs/PLAN.md: v1 implementation plan (positioning A, gap-expanded scope,
  llm-security reuse map)

15 tests passing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01K8GmKRCdsPjWYAKWsNgeQS
 2026-07-04 09:24:20 +02:00